Nine Biggest Mistakes In MBA Rankings

rankings image

Rankings of academic institutions are inherently tough.  It’s difficult to put metrics around a nuanced educational environment.  Thinking of business schools, how does one quantify such things as intellectual vigor, creative thought, ambition, leadership, interpersonal skills, stellar teaching, innovative curricula, networking prowess, and success in the workplace?  That being said, here are some of the most common ranking mistakes that we see from the school side, which should at least give us pause:

Imprecise Questions:  Media outlets need to take the time to educate themselves on the appropriate terms and lingo for higher education, admissions, salary earnings, and other data. Failure to do so leads to howling discrepancies.

Examples:  An admitted student is not the same as an enrolled student; What is meant by “full-time” vs “part-time” when speaking of faculty (are we talking % time hired or tenure-track status)?;  Are emeritus professors included in the faculty count (how about adjuncts, joint-appointments, faculty from other departments who routinely teach B-School classes?); Different schools use different methods to calculate standard (SAT, GMAT) exam scores; Is the school following MBA Career Services & Employer Alliance guidelines for reporting salary data?

Failure to Show Scores and Clustering:  Ranks are inherently misleading because you don’t know by how much one school outranks another. A couple of the major MBA rankings now include index scores, but many others do not. The same issue occurs within ranking subcategories, where you get a rank instead of actual data.  One doesn’t know if there is essentially a tie between schools, or if a yawning abyss separates them.

Failure to Adjust Salary for Cost of Living or for Industry Sectors.  This is perhaps less a “mistake” and more a “critique” because there are strong biases at play that are likely to be overlooked. Among MBA rankings, only Forbes attempts to adjust salary for cost of living and only Financial Times adjusts for variation between industry sectors. Since roughly 45% of all ranking weight averaged among the five main MBA rankings (USN, BW, Forbes, Financial Times, and Economist) relies on salary and placement–by far the most heavily weighted area–failure to make these adjustments creates a strong hidden bias. The cost of living in NYC and San Francisco is not the same as Denver and Minneapolis. Bicoastal salaries are often 5% to 10% higher than those not on the coasts. And regarding industry sectors, consulting and finance tend to pay (sometimes considerably) more than marketing, general management or non-profit sector jobs. In short, schools more on the coasts, in large metro areas, or those heavily focused on consulting and finance will have a large advantage. Foreign schools also have made huge inroads into global rankings based on higher salaries (than the U.S.).

Failure to Separate Apples from Oranges:  As Malcolm Gladwell pointed out, you can rank all things on a single dimension (miles per gallon, GMAT score, percentage employed at graduation, etc.), but if you want to attempt a holistic or comprehensive ranking, you need to limit the scope to a class, type, or specialty.  We rank SUVS, trucks, and sports cars separately for a reason. Rankings often combine one-year MBA programs with two-year ones, 800-student programs with 40-student ones, foreign programs with domestic ones, programs with senior managers as students with programs with 21-year-olds as students. While it’s tempting to do so, because they offer the same basic degree, the results quickly become meaningless.

  • Renault

    Who cares?

    Nobody’s going to business school to learn anything.

  • JP

    I think this article needs to be read by everyone who visits this website. Especially those who are involved in the Booth/Wharton ranking war. I said it in a previous post and this article supports my assertion: B-School rankings are done to bolster sales and interest in the various publications and manufacture ranking changes each time in order to maintain buyer interest and relevance. I think they are helpful to a degree and can even be used to encourage schools to improve, but should be taken with more than a pinch of salt.

  • Orange1

    Oh for the days my dad talked about, when rankings were just about non-existent and the word “brand” (thank you Donald Trump) was not used. But seriously, are there really only a handful of schools that can teach this stuff? Can BW, USN, and FT sit in every class at Harvard or UCLA or Michigan or Stern or Tuck and say, “a ha, this place is so much better and the people are smarter?”

  • Ryan S.

    Ironically, the schools I want to go to are in the exact opposite order that they place in the rankings. Having had an awful experience in undergraduate school, fit (location, culture, curriculum) cannot be mentioned enough.

    Slight exaggeration for effect here: I hated Boston, with an unrivaled passion, visited it and it was a total dump. Even if I had the grades to get into say, a Harvard or MIT, I’d far rather go to a school which was a better fit, where I knew I could thrive.

    Those who look at rankings and automatically identify the highest ones that they could get into as the ones where they will apply are very shortsighted.

  • Thank you, Mr. Turner, for succinctly itemizing the problems with B-schools rankings.

    Your second item highlights what I believe to be the greatest among these flaws: a lack of statistical analysis of the data. For example, as BusinessWeek does not release the raw scores of the schools it ranks we can only speculate if the difference between any of, say, the top 5 (or 10) is significant.

    At the end of the day, however, statistics doesn’t sell magazines and concluding that no significant difference exists in the scores of the top 5 schools won’t have magazines flying off the shelves. I have heard it said informally that the MBA rankings edition of BW is always their best selling. I don’t know if that’s true, but its safe to assume that BW doesn’t rank b-schools as a public service. (Of course, that applies to any publication in the university/school/program ranking space.)