The Case Against The GMAT

Kevin Frey, managing director of Rotman's full-time MBA program

Kevin Frey, managing director of Rotman’s full-time MBA program

Frey deployed advanced data analysis to get to the more critical variables that directly forecast whether an applicant will have a job three months after graduating from Rotman’s MBA program. “What shakes out are a set of five variables that are significantly predictive of employment,” he says—and not one of them is the GMAT. “It’s an exciting finding because the whole industry pivots around the GMAT and it was at best marginally significant. We don’t have hard statistical confidence the GMAT actually matters at all and if it does, it doesn’t matter very much. It had a very small effect size.”

ADMISSION INTERVIEW SCORES & AWA SCORES WERE STRONGLY CORRELATED WITH EMPLOYABILITY

What does matter? Here are the five most significant warning signs that an applicant is most likely to be an employment risk, in order of importance:

  • Citizenship from a region of the world for which Frey declines to identify.
  • Years of work experience.
  • Admissions interview scores for candidates.
  • Analytical Writing Assessment (AWA) test.
  • Undergraduate grade point average.
  • Then, finally, came an applicant’s GMAT score—barely.

“Some statisticians would not even include the GMAT score because it was only marginally significant,” adds Frey. “It’s barely in the score.”

Based on the study, Rotman then devised an “employability risk score” for every person who applied to its MBA program in the 2014-2015 admissions cycle. A perfect score is zero. A score of four or more signals that the candidate is a high risk of not getting a job within three months of graduation. “Like a credit card company that assesses the likelihood that an applicant will be able to service their debts, we wanted to cut through all the noise and identify the variables that could predict the probability that prospective students would be unemployed post-graduation,” explains Frey.

In contrast to the GMAT, the AWA score (which ranges from zero to six) and the scores admission officials gave applicants for their face-to-face interviews were highly predictive of a candidate’s ability to land a job and to gain a solid starting salary. “They are far more important than a GMAT score,” says Frey. “If an employability risk score of four makes you high risk then having an AWA score under a certain number (which Frey will not disclose) would get you two points on the employment risk score and having a GMAT below a certain score would get you only one point.” Translation: Both AWA scores and admission interviews are at least twice as predictive as the GMAT for employability.

‘A BIG, BIG RISK FACTOR: TEN OR MORE YEARS OF WORK EXPERIENCE’

Candidates with ten or more years of work experience were more likely to be unemployed three months after graduation. “It’s a big, big risk factor,” says Frey. “That person is three times more likely to be unemployed post grad than our control group which had three to five years of work experience. It was highly predictive.”

Frey believes the predictive nature of the admission interview scores was “completely logical because to get a job you have to do well in an interview. If you get ranked well on the admissions interview, chances are you probably will do well on a job interview later.” At Rotman, all interviews are done not by alumni or second year students but by admissions staff.

More surprising was the predictive ability of the AWA. “The AWA had a substantial effect size, yet we had never looked at it before. But it is highly predictive of your ability to get a job.”

References, admission essays, academic and industry background, among other variables, had no significant impact on employability, says Frey. “What you majored in didn’t matter. Neither does gender.”

How has it changed admission decisions at Rotman. Rather than routinely shovel scholarship money to high GMAT candidates, the admissions committee has a more reasoned approach. “We want students who will have an impact in the world,” adds Frey. “The people we want to give scholarships to are those that organizations will want to hire. So we have reversed engineered this. We don’t want to give scholarship dollars to students who will likely underperform in their careers. GMATs are becoming more and more irrelevant to the scholarship discussion. A high AWA will definitely help your case for a scholarship now. And yet only a year ago we would not have even considered the AWA.”

Acting Admissions Director Gauthier believes that the GMAT assumes so much importance in MBA admissions “because it’s easy. When you look at thousands of files, you have to sort on something. So it has become a sorting mechanism that some find helpful.”

  • AshamedToBeCanadian

    This is AWFUL!! Rotman is basically going on the record to reveal one of the worst kept secrets of MBA admissions…that your post-MBA employability is THE most important factor.

    Is this what the MBA has become? Just a “training program” for people who already have what it takes to get a “better” job (read: higher salary), and we don’t want to talk to you otherwise? Sounds just like the headhunters that offer no added value and will only talk to you when you already fit all the job requirements.

    What happened to education and universities being transformational experiences? And about helping willing individuals start their leadership journey, which typical takes way longer than 1-2 years. Too idealistic, I guess *sigh*

    It’s now all about the rankings, immediate short-term KPIs (salary increases, placement rates)…further proof that the MBA is destined to die.

    Roger Martin has got to be pissed after everything he did during his tenure to turn the school’s reputation around.

  • CPO_C_Ryback

    Whoa, Hoss.

    The GMAT is a empirically-rigorous, objective method to gauge how well a student will do in their first year.

    A low GMAT score — there is a very good chance, that student will take a severe mental-pounding in the first year. It will take a super-human effort to pass all those courses.

    As for HBS — please. Most of their applicants have been in the top 1% academically, all their lives. They could probably also go medical or law, and get in.

  • Michael Cohan

    John,

    An excellent article per usual. I love the “Moneyball” analogy. I also welcome Rotman’s study on which admissions criteria predict “short-term employability.” It would be
    interesting to see further analysis in this area like correlation to long-term career trajectory or the quality of employment. One explanation as to why older graduates in business school are risk factors is that job searches for older candidates in general take longer because these candidates are more senior and selective. But these older graduates might add some other dimension to “the overall quality of a class”.

    Another observation is that with “Moneyball”, the Oakland Athletics found different statistics to be better indicators at producing runs. It is a given that producing runs is essential to “winning in baseball”. But to what extent is short-term employability “winning at business school”, “a proxy for the overall quality of a class” or as Ricardo said “success”? I would be curious to see what metrics HBS used to measure “success”. For those of us that can remember, HBS used to ask MBA applicants the essay question, “How do you define success?” Just as applicants had different answers, so too would business schools.

    But pulling myself from the trees to the forest, I have observed in my clients over the past eleven years that the best ones usually score above a certain level on the GMAT (normalized by region), but at a certain point a higher GMAT score hardly makes a difference.

  • JohnAByrne

    If EMBA tuition is four times the cost of the part-time program at the same school I would definitely go part-time, especially if you are going to pay full freight. The advantages of an EMBA over a part-time are significant: You go through the program as a cohort so you really get to know and depend upon your classmates. Much of the learning in any MBA program is from your fellow students. EMBA students also tend to be far more accomplished and along in their careers so they are naturally better to network with. An EMBA schedule–as grueling as it can sometimes be for someone balancing family and job–is still more workable than evening part-time programs that fewer people see to completion. One suggestion: Talk to the EMBA folks at the school and tell them you are concerned about the costs. Perhaps they will cut you a deal. There is a good bit of scholarship money out there and a good bit of discounting. You never know. It may not be worth four times the part-time program, but it could be worth two times.

  • Janice Lao

    Hi John. Would you kindly please elaborate more as to why EMBA would be a better choice as opposed to a part-time MBA program for example? What if tuition for EMBA costs 4x as much as tuition for the part-time MBA program at the same school? Would you still recommend EMBA in this case?

  • Dr_Ads

    Actually most accept either GMAT or GRE, but having good scores in both increases your odds of getting in.

  • Jaycee

    I agree that there are a lot of issues with this study. First of all, who says the GMAT has been an indicator of employability? It doesn’t measure interviewing or job searching skills- I always thought it was a supposed measure of “intelligence” and whether a student could keep up with the rigor of MBA academics. Second, there is serious OVB in many of their conclusions. For example, someone with 10 yrs of experience not getting a job within 3 months of graduation: I don’t think that is because they are less employable. Maybe for some, but I know some older MBAs who are more willing to hold out for the right job than a 25 yr old who will take the first offer they get in order to get their foot in the door. This is sloppy analysis at best.

  • fermen

    that was before the gmat got to be a very profitable business.

  • Ricardo Betti

    John, I’ve been in admissions consulting for 25 years, and I remember that HBS dropped the GMAT requirement for many years (in the late ’90s). The reason: they did a thorough research similar to Rotman’s, and got the exact same results, “no correlation with success”. I had many clients admitted there without GMAT, and they suceeded very well in their careers afterwords.

  • Alex

    Would Rotman do this study if they didn’t have such a mediocre GMAT average?

    This is such a set up. Nice try but this isn’t going to change any other school/employer’s perception of the GMAT other than Rotman itself. And over time, its Canadian rivals are going to be eating even more GMAT points away from the school.

  • John,

    Great article.

    Did they mention the GRE at all in their research? Rotman does accept the GRE.

    For the last several years Rotman has waived the GMAT for CFA holders. Was that discussed at all?

    The correlation between the AWA and interview makes some sense. They both reflect communications skills. But traditionally the AWA has been influential on the margins. I’ve never heard of it being used as a major predictor of success or as decisive in admissions decisions.

    It’s also ironic that single interviews are being seen in many fields as less credible or reliable indicators of employee or student success. I believe Google has abandoned the individual interview in its hiring process. Medical schools are moving increasingly to a multiple-mini-interview format to reduce bias and get a better sense of fit with med school applicants. I guess Rotman has its video interview as well as the personal one so it isn’t really relying on one interview.

  • Brazilian

    Completely agree! Correlation is not causation!!

  • JohnAByrne

    I completely agree. Those with 10-plus years are generally better off in EMBA programs.

  • wedeman

    some of them are sponsored. .better take it full time if ur company sponsor u.

  • gerfd

    well..all top european schools ask for gmat for the mba.

  • Lisa Grassfield

    I’m curious about the reported liability associated with individuals with >10 years of work experience. If a 10+ year professional suspends employment to enter a full time MBA program, as an admissions representative, I’d have some concerns. In fact, I’d advise against it. Better to pursue an Executive MBA format to maximize the network of peers. Return on educational investment is high for EMBA graduates.

  • TechInsane

    Sorry but this has been known for a long time, exams are not the best way to measure cognitive intelligence (IQ), and we are not even talking about Emotional Intelligence and other forms of intelligence. I am glad I ignored taking the GMAT for my MBA, just a waste of money in my book. I hope this data and others make people who spend months on months stressed out and concerned about their future do that no more.

    Finally, European universities do not use GMAT, if one has an undergraduate degree that is all they need to study for a masters which is for a year and not two, hence why the cost of education is cheaper.

    Students should go into universities and come out enlightened and not riddled with debt, education is one source of why a country has a competitive edge against others. I wish the people in positions of power would look into this with input from businesses.

  • Matt

    I’m sensing a ton of spurious correlation and OVB. This is not a good selling piece for top candidates looking at Rotman, perhaps they should’ve thought that out.

  • Hello

    Um… Someone should tell them that the AWA is likely correlated with the “region of the world”, the most significant variable that they found. The AWA is only significant because internationals have a much more difficult time finding work. This is a misuse of statistics. Wow.