Tuesday, November 24, 2009

CAT: restore the faith

Eight days into the 'mother of all exams' reports of hardware problems, software errors, poor administration, repeated questions) and other testing issues are still coming in.

It is time the directors of IIMs acknowledge this experiment has failed and call for a re-test - in old fashioned paper and pencil format. A high stakes exam like CAT is an act of faith. When that faith in its fair delivery and character of merit is at doubt, it no longer serves its purpose.

Just like when it physically leaked in Nov 2003; and had to be re-administered.

But the issue goes beyond this year. At some level we've all accepted that computer based testing is better, more efficient than traditional methods. Is that really the case?

The issue is not just hardware or software - but more fundamental. This exercise is a 'CAT and mouse' game - and not just because the exam went online this time.

On the one hand you have a couple of hundred IIM professors, a handful of whom would sit down, rack their brains and come up with the CAT paper. On the other hand, you had hundreds of 'experts' preparing mock CATs, analysing past papers, predicting future patterns. And many of these experts are IIM grads themselves.

When it came to setting one paper a year, the CAT could somehow pull it off. Remain distant, difficult, unpredictable - the Mount Everest of all exams. Putting it online - with a poorly set question bank - has taken it down to the level of a Sahyadri.

This destroys everything the CAT has stood for - all these years.

Now one can dispute whether CAT actually does select people with the best managerial potential in the first place. An article posted on fairtest.org notes that in 1985 Harvard Business School (HBS) decided to eliminate the GMAT from their admissions process.

John Lynch, the Admissions Director at the time, gave several compelling reasons. In a blind test, Harvard found that admissions decisions made with and without the GMAT were essentially the same. Success at Harvard depended on intangibles such as motivation, interpersonal skills, perseverance and hard work – all factors not measured by GMAT.

Looking at undergraduate grade-point average (UGPA), ethics, leadership, community activities, prior work experience and the interview made GMAT scores "superfluous".


However, 11 years later HBS reinstated use of the test. The point is, as long as you are using a GMAT or CAT, let there be no doubts about the administration and standards of that test.

Which brings me back to the question of computer based testing. In 2008 there was a GMAT Cheating Scandal involving Scoretop, where 'live' questions were posted on a members-only website. This article published at that time reveals some interesting facts:

1) The item pool is periodically refreshed, but the same questions are reused for at least several weeks.

2) The testing industry was well aware of the vulnerability of computer adaptive tests to what it calls “pre-disclosure.” Before the 1993 introduction of the computer adaptive Graduate Record Exam (GRE), two researchers at the Educational Testing Service (ETS) wrote about their “fear [that examinees] will remember questions and reveal them to their friends or to a coaching school” and that “a group of examinees [might] memoriz[e] subsets of the pool and combin[e] their knowledge.

3) To expose the problem, staff from the Stanley Kaplan Education Center took the computerized GRE, compiled a list of items they had memorized, and presented it to ETS officials. ETS, which then administered the GMAT as well as the GRE, responded by suing Kaplan for copyright violations, even though the questions were never made public (see http://www.fairtest.org/ets-and-test-cheating).

4) After this incident, test-makers said they began using much larger item pools and changing them more frequently, but there is no proof for this claim. In 2006, ETS lost the GMAT testing contract after a series of administrative and scoring errors. The test is now run by the global conglomerate Pearson.


Under 'Indian conditions' where stakes are so high for both students and the coaching classes, I think it will be far more difficult to maintain the integrity of the question bank!

In conclusion, computerised testing can work - but requires herculean effort and partnership between the IIMs and the testing agency. It's not an exercise which IIMs can simply sub-contract - like housekeeping!

Brand IIM is like the venerable banyan tree, and the CAT forms the mighty roots of that tree. Destroy those roots and the very tree will start withering...

The IIMs must reclaim the CAT immediately, or they will lose the ground beneath their feet.

For more on the issues related to effectiveness and fairness of SAT, GRE and GMAT see articles on www.fairtest.org

22 comments:

  1. Surely a nice article and genuine call to retake test with paper based type.

    That is the only way to reinstate faith on the system.

    Number of students those are appearing for CAT was one of the main issue raised by IIM's again and again. I wonder , won't it be better to have one screening test - For example GMAT itself to qualify less number of students for CAT ? [ Or if IIM have allergy for GMAT ,some other criteria to screen and limit numbers for CAT ? ] Though not sure the practical aspect of this suggestion - something is require to be done to reduce number for CAT.

    ReplyDelete
  2. Oh you bet. We all are wishing for a re-test here. The very act that it is placed across a ten day bracket is something initself a detrimental thing. Let alone the creepy mass cheating news that are coming in, even the way some people were able to have an extra edge in terms of the way to attempt the whole test.

    Everything changed drastically as we progressed towards the end dates. And thus an overall percentile would make heads roll.

    The test paper tested only one thing - speed. The aptitude part never came into consideration.

    People aspiring for niche institute like MICA, say, will have a tough time even getting a call this time round. For, these people [ me included] are not known for their active left brain...and hence they could not attempt a dracula in there and hence.... stand a chance to lose out to people who dont even want to apply for such niche institute. WOW...that's almost amazing...an alsoran when there IS no other..lol

    Let's see what comes out of it. A straight poll will give out a majority to a retest.

    These are not prates. When something is been spoken about in so many channels and streams and places,..there has got to be something wrong.

    Hoping for a remedy!

    ReplyDelete
  3. While the idea of a computer-adaptive CAT should definitely be questioned, I think I have an interesting perspective in having never given the CAT, but having dealt with Prometric for my GRE and TOEFL.

    The inherent problem with computer-based tests is the limited number of candidates that can take the test at any one time, making the use of a diverse question bank imperative. However, the way this is done is also important. This year, one of the many grouses of candidates, has been the utter confusion and lack of transparency in the system.

    Let me tell you that while the GRE is executed very smoothly in India by Prometric, the TOEFL is not. When I first heard that the contract for the CAT had been given to Prometric, I wondered which of the two tests above it would be like. Unfortunately, it turned out like the TOEFL.

    My point is that this is not some failure unique to the IIMs or the CAT. Prometric has a track record of rescheduled tests, glitches during the test and absurd scoring with the TOEFL. Just go to any TOEFL discussion board and see the candidates' gripes. It's just that the scale of the CAT has magnified these shortcomings. Prometric's weaknesses are neither new nor unique at all!

    ReplyDelete
  4. U touched it 2 d core, wid awesome insights..........
    Gave my xam 1st day 1st slot, feelin totally betrayed.......Over n above wat dis has done is questioned d credibility of d IIM's selection process even mayb even dere r some glitches n all...........but seriously it's HIGH TYM dey rise up2 d situation n conduct a re-exam even a single %ile can make our lives hell n d 12 months preparation equates 2 a big VOID..........

    N tell u 1 thing :-
    Putting it online - with a poorly set question bank - has taken it down to the level of MAUNA KEA instead of Sahyadri, dey r quite tall.
    CHEERS !!!!!!!

    ReplyDelete
  5. Exactly my thoughts

    ReplyDelete
  6. Why have they made such a big mess of it??? BITS moved to an all online test years ago --- and it made no headlines.

    ReplyDelete
  7. IIM,Plz announce the Re-Test in paper based format soon.You can continue this online format next year with little extra caution.

    ReplyDelete
  8. Touche!

    Although I didnt face any problems with computer equipment, the conducting of the test itself left a LOT to be desired!! And though my problems might seem small compared to what the others faced, but they are still important enough to totally alter the direction of my life!

    I had taken the CAT on 30th at COE, Pune. The Prometric staff were absolutely inconsiderate of the fact that this was such an exam! They were talking quite ordinarily amongst themselves, even joking laughing amongst themselves! Very frequently AFTER THE EXAM STARTED, there were announcements of the kind "Where is the sharpener??! Who has the sharpener? You should keep it on the centre table!" (there were 3 sharpeners for around 20 people!) or even "PLEASE DONT CLICK THE END BUTTON! YOUR TEST WILL END IF YOU DO THAT!" (I understand that its a important point, but since they had already mentioned it around 10 times before the exam started, what was the need to do it again and again in the middle of the exam?!! And WHO designs a s/w without any confirmation popups on clicking "End"??! )

    In CAT, a single 5 second concentration lapse can cost you 5%ile! And all such disturbances definitely caused me at least 4 - 5 concentration lapses!

    Moreover, WHY did they change the software IN BETWEEN THE EXAM DATES??? My brother took the exam 5 days later, and surprise surprise!! This software HAD confirmation popups!!! And other changes to make it better! So all those announcements not to click "END" (not to mention those examinees whose exam got ended because of mistakenly clicking END once) were not needed. Not only is this unfair, it is downright wrong to put the people who took in on the first couple of days under such psychological pressure, whereas not doing the same for the rest of the people!

    Thank you for bringing these issues to light! Please bring these issues to the notice of people who matter. I have already complained to Prometric about their staff, but I dont know how far that will be heard.

    One more point. By the 3rd/4th day, it was evident that the question bank consisted of MANY questions from previous years papers. I saw people mugging up previous years CAT papers. Since when did CAT measure muggability? Ridiculous.

    ReplyDelete
  9. Hi Mam,

    Thank you for this article.
    I hope IIMs realize the gravity of the situation at least now, when their own alumnus has presented a holistic view of the whole situation with relevant examples from present and distant past.

    Thanking you once again.
    Ashwin Kumar D P

    ReplyDelete
  10. rashmi mam

    whats ur take on this
    i appeared for CAT on 5th dec (8th being the last ) and ,there are rumours going around ,that those who have taken the test in the last few days /slots ,will have a diff scheme of marking (maybe a negative or even a progressive negative marking ) bcoz according to them we were aware of many questions ,in fact i never wanted a late slot ,it was just that by the time i got the brochure ,all earlier slots in my state capital(jaipur) were full ..so i had to go for watvr available ...is this differential marking scheme thing rumour possible

    ReplyDelete
  11. I Agree.More so for CAT, as here even a single mark can make the difference between being in the IIM shortlist and not being in it. This is unlike GRE / GMAT where such inaccuracies have less impact as the score there is used more to eliminate rather than select a candidate

    ReplyDelete
  12. Nice, you have covered everything in detail.

    ReplyDelete
  13. haah!!its like u hav strike right on d chords....well im not going to take d test this year...but given to feedback of the xaminees nd yes offcourse d mayhem created by PROMETRIC i just only hope nd pray...get us d pp format back...unles nd until u got d coins placed in the right purse....i mean given d kind of glory nd proud d CAT nd its xam pursue...screwing wid this really hampers the engima of indian youth....obviously yaar..for some people out dere this is a stake of lyf nd death(including me too!!!)...well..changes r good..but for better purpose....nd given thingh to RESTORE THE FAITH OF CAT..people out dere hav to think a lot nd do a lot to get d things on d track back!!!

    ReplyDelete
  14. From the sensational and blatantly erroneous reports in various news papers and forums, your analysis is a drastic change which is a respite.

    I appeared for CBT CAT this year on Dec 1. Like thousands of others, I also tried to make sense of what the coaching institutes, forums and print media had to say about the new avatar of entrance exam conducted by IIMs and Prometric.

    Even if I keep aside the cynicism towards these reports, there are several fallacies which I observed in them.

    First of all, a very simple observation. Why the hell are we concerned about the students IIMs are taking? It is they who have made this formidable reputation, and it is they themselves who are the preservers of that acclaim. IIMs don't have neither the time nor the energy to pay attention to our emotional attachment to CAT and how we think they should take students. Our concerns are, hence, very much misplaced. Further, the IIMs aare responsible for making the questions. I am sure they know that questions are being repeated. When IIMs have asked students not to disclose questions, how is it so that people are criticizing IIMs for questions? It is illegal as nobody can discuss or disclose the content of the paper.

    One observation of yours regarding the predictability of CBT CAT is agreeable. Even I am of this opinion, that the first thing a candidate checked when he opened the CAT paper was the number of questions. Apart from the takers of first test, candidates of no other test must have faced this issue. That might lead us to conclude that it helps those people who can only work in their comfort zones. But after giving the exam and after hearing from several colleagues of mine, I just didn't feel that this is the case. Every paper is unique in the respect that the level of DI, QA and RC/VA is varying. The number of questions asked in sub-categories of these is also changing, though not that drastically as was the case till last year. And, without sounding very judgmental, I would like to say, that not even the managers of Harvards and Whartons are strong enough to deal with unpredictability, what difference does it make if some of the people are less capable of dealing with it.

    Another point is that of temperament. Those who are complaining of things like change in date, test not opening, review button placed close to exit button and what not, simply don't deserve to be managers from such prestigious institutes as IIMs. Some of these guys are going to shoulder the responsibility of whole companies/organizations and if they can't deal with such trivial issues then it is better they don't consider a degree in management. Managers are not supposed to crib, they are supposed to solve problems.

    And lastly, why do I sense nostalgia in the older generation for paper-and-pen CAT? There is just no explanation for it. IIMs are very much of the students they are looking for, and they have every right to modify the entrance process they want. Instead of criticizing their bold decision to change the entrance exam, we should be supporting them in their endeavour to successfully execute this process. Giving chance to 2.4 lakh people, certainly they have devised strategy to tackle problem of 10 day bracket. Though, it remains to be seen how they go about it in future, let's not deride them for no reason, and that too before the whole thing is out.

    ReplyDelete
  15. Send breathtaking gifts to USA and make a beguiling focal point to any celebration. Add unique beauty and style to the occasions of your special ones in USA with our exotic range of gifts.

    Please visit:
    www.expressgiftbasketsusa.com

    ReplyDelete
  16. CAT FAIL
    IIMs FAIL

    remember the basic thing

    unlike foreign universities where the professors who are brilliant individually and together are beyond compare the same is not true for almost any Indian university in totality.

    IIMs mainstay are its students and not its faculty and directors.

    and why isn't gmat itself being used!?!!? i don't get it.

    ReplyDelete
  17. CAT has to move online and should be available all year. This will in the long run make life easy for CAT aspirants. Transition is pain and some people do suffer. Also, Management includes outsourcing what is outsourcable and managing in-house what is core to the organization.

    ReplyDelete
  18. While there is definitely a cause to feel concerned about the glitches in the computerised testing, we have to understand that since the IIMs are considered the pinnacle of education achievement simply because the salary an IIM grad commands, the number of students appearing for CAT will keep on going up.

    In such a scenario, it will become well nigh impossible to use the manual paper testing process. How do you test and evaluate more than a million candidates in a month by a couple of hundred evaluaters?

    The only way out is that the professors of IIMs start thinking differently about CAT.

    GMAT/TOEFL system offers one possible solution.

    How about a candidate taking a CAT exam anytime throughout the year, with his/her score being valid for a certain number of years. Then he/she can choose to apply for any year in any IIM/another school that uses CAT scores independently and attach the CAT score with application.

    I am pretty sure we have enough IT brains in the country (among them a fair number of alumnus), who can come up with a solution where a prospective candidate can apply online for CAT, get a user id and password, log in and take the exam.

    I am pretty sure the same system can be made fail safe. After all we are banking and buying online.

    ReplyDelete
  19. Rashmi, I wrote a detailed reply to your post on my own blog: In defence of the computerized CAT. I'd love to know your opinion on this.

    ReplyDelete
  20. Finally..., finally I see the article I was expecting on youth curry. IIMs have taught us a wrong management lesson. We students were hoping that, IIMs will surely take a right call. But they have showed their back to the problem, disowned the responsibility and justified their blunder. I doubt whether they ever would admit it.
    But, I have a request to you. Next time you reprint STAY HUNGRY STAY FOOLISH, please remove the forward written by Mr. Barua. he just doesn't deserve to figure in there. I have torn it from my personal copy. He is acting like "Dhritarashtra of Mahabharata". He knows everything. But still not ready to accept the truth....Why should such people preach ethics and write books on scams of others..?

    ReplyDelete
  21. why do we see so few blogs from you now a days

    ReplyDelete

Disqus for Youth Curry - Insight on Indian Youth