CBT CAT 09 - Evaluation and Ranking

Update: I got a re-assuring reply from Professor Pankaj Chandra, Director IIM B, saying - "Sorry for the inconvenience caused to you. We will address the issues that you have raised in your email." Thank you Sir, it means the world to me for you taking out time to reply to my email. Hope everything is sorted out and resolved pretty soon so that everyone gets a fair chance to start their MBA from the IIMs !!

Also check out this TOI article.
End of Update

Following is the email which I've written to the directors of the prestigious IIMs regarding information for evaluation and ranking of scores obtained by students in the CBT CAT 09. Decided to post it in the blog sphere as well so as to know if there are others who share a similar concern.

Disclaimer : I'm not whining against CBT CAT 09 nor asking for a re-test / paper based test to be conducted again. Also those who have scored awesomely in this years CAT might be against the views expressed in this post and will definitely tell me to "adapt" to the changes and that this is the system and that we have to live with it. Okay, thanks in advance. And congrats to all of you for cracking CAT. Please spare me the lecture.

Respected Sir(s)

I, the undersigned, aspire to start my management studies in one of the prestigious IIMs from next year. I have appeared for CAT 09 on 5th Dec from Hyderabad and I have no complaints with respect to the actual test conduction process on a personal level since everything went smooth for me. There were no technical glitches or inconvenience of any type. I guess the situation has been improved drastically from what it was at the onset and I thank the quick measures taken for the same.

However, keeping asides all the other issues we have seen bogging CBT CAT 09 so far, I always thought that the greatest challenge of this year's exam was to arrive at 20 papers with questions having more or less similar difficulty level. This is based on the presumption that, the evaluation of candidates is going to remain constant as earlier years - i.e. - the "percentile" marking / grading system - where you rank students based on the "score" they obtain.

However, me and a very large number of students who have appeared for CAT 09 feel otherwise (that the difficulty level was different and waving in different slots) due to the following reasons:

1) Students who gave CAT in the earlier two days, though had to under go the trauma of delayed server responses and virus attacks, have reported that the level of questions was very easy and most of them managed to score 50+ out of 60 which was so unlike a CAT exam. Unless, it was intended that the CAT exam be brought down to a lower level, I feel this has really dented the image of CAT as an exam where scoring 50% of the total marks was considered as a daunting task earlier.

2) Students who gave CAT in the middle days reported that the questions were getting repeated from the earlier slots and also were being taken "as_it_is" from previous years CAT papers. This, I feel, was a big blow to those who took the tests earlier because they would never even have got wind of some of the difficult questions since they were actually attempting it for the first time. It might be unnecessary to point it out Sir, but even 3-4 questions solved in a minute just because of "prior knowledge" ensure that you make a quantum leap by saving time and dedicating it to other ques. Is this fair to all others who instead got unique questions ?

3) After the break out of all the above issues along with numerous others, it seems that the CAT has been brought back to being CAT on the last few days. Suddenly, the level of difficulty of questions in some sections has been steeped up making it really difficult to cross even the 45 mark !! I will admit, I might not be the brightest of the CAT aspirants, but then I am definitely not among the dumbest. Now we are seeing almost a no-show of repeat questions and no previous CAT questions. This is good from an exam point of view, but how do you justify the same happening for some students earlier, which clearly gives them advantage over others. As you know, a difference of a couple of questions make it a big deal of difference due to the current "percentile" based marking / grading.

The issue which I want to address through this letter is that how are we going to grade students appearing for CAT 09 based on questions with varying difficulty level ? Is it really fair to compare someone who got 50 correct in a easy paper (By easy I mean and include any / all of the following : easier questions lifted from database, questions repeated from earlier slots, questions taken from earlier slots, questions passed on by friends / teachers / forums etc) to someone who got 40 correct in a paper which did not have the above mentioned features ? Unless of course, there is going to be some sort of normalization based on the difficulty level, some sort of relative grading, I believe many are going to suffer from gross injustice and would only be left to regret at not being offered a fair shot at the IIMs.

From IIMs stand point I have heard that the argument being offered is that such things happen with other exams like GMAT as well and no body complains there. However, I would like to point out some key differences between the two, which I know you are definitely aware of :

- GMAT can definitely be taken more than once in a year in a testing window which is very large as compared to that of CAT 09. The point being, one can improve upon his / her score if he / she feels she got a bad deal. CAT 09, on the other hand can be given only once, and you just have to live with it and give it next year only.

- While GMAT also sometime may show up with repeat questions and different difficulty level questions, we can't forget its a "Computer Adaptive Test" - where the difficulty level depends on how many questions you answer correctly. The database is repeatedly updated to avoid repetitions. Even the scoring algorithm makes sure you can't skip to the next question until you answer the current one. Most importantly, the "result" is generated at the end of the test itself.

In the case of CAT which is a "Computer Based Test" - and most importantly - where the results are based on "%tile" and not "%tage", it really boils down to how many students you outscore / outperform. In such an evaluation scenario, how is it justified to have some students face a difficult paper and some an easy one (the demarcation of the above being done in Point 3) and on top of it, its just unbelievable to imagine that our futures will now be controlled via a random algorithm which is modified "midway" between the exam after glitches being pointed out by earlier CAT takers !!

I believe that there are many things which have gone the undesired way in CBT CAT 09. But the most serious issue according to me which needs to be addressed is how the "ranking" of students is now going to be done? Do we just have to live with the earlier ranking scheme based on %tiles and cut offs which will be same for everyone or are the IIMs going to take pain to make a definitive analysis of the different slots and come out with something which is just for all ?

By the way let me just repeat, I am not asking for a re-test / paper based test as some might perceive but I'm only asking for the right to know how will the grading and ranking be done and would it be fair to everyone given the above things which have hampered this year's CAT.

I don't know if this letter will even be "opened / read / thought upon" by the busy IIM directors since you must be receiving thousands of such emails / letters by now and one more student crying out aloud will just be like a drop in an ocean. They might even consider me treating CAT as sour grapes. But at least I will be in peace thinking that I did raise my voice / concern when I had the time and means and would avoid the regret in case I don't complete my MBA from an IIM.

I sincerely hope that I would get a fitting reply from some of the authorities responsible for CAT 09.

Thanks,
Suyash Joshi

Please feel free to comment and provide your feedback about the above "rant" by clicking here. Also read the disclaimer at the top of the post before posting your opinion on me / any views in the post.