Rankings of academic institutions are inherently tough. It’s difficult to put metrics around a nuanced educational environment. Thinking of business schools, how does one quantify such things as intellectual vigor, creative thought, ambition, leadership, interpersonal skills, stellar teaching, innovative curricula, networking prowess, and success in the workplace? That being said, here are some of the most common ranking mistakes that we see from the school side, which should at least give us pause:
Imprecise Questions: Media outlets need to take the time to educate themselves on the appropriate terms and lingo for higher education, admissions, salary earnings, and other data. Failure to do so leads to howling discrepancies.
Examples: An admitted student is not the same as an enrolled student; What is meant by “full-time” vs “part-time” when speaking of faculty (are we talking % time hired or tenure-track status)?; Are emeritus professors included in the faculty count (how about adjuncts, joint-appointments, faculty from other departments who routinely teach B-School classes?); Different schools use different methods to calculate standard (SAT, GMAT) exam scores; Is the school following MBA Career Services & Employer Alliance guidelines for reporting salary data?
Failure to Show Scores and Clustering: Ranks are inherently misleading because you don’t know by how much one school outranks another. A couple of the major MBA rankings now include index scores, but many others do not. The same issue occurs within ranking subcategories, where you get a rank instead of actual data. One doesn’t know if there is essentially a tie between schools, or if a yawning abyss separates them.
Failure to Adjust Salary for Cost of Living or for Industry Sectors. This is perhaps less a “mistake” and more a “critique” because there are strong biases at play that are likely to be overlooked. Among MBA rankings, only Forbes attempts to adjust salary for cost of living and only Financial Times adjusts for variation between industry sectors. Since roughly 45% of all ranking weight averaged among the five main MBA rankings (USN, BW, Forbes, Financial Times, and Economist) relies on salary and placement–by far the most heavily weighted area–failure to make these adjustments creates a strong hidden bias. The cost of living in NYC and San Francisco is not the same as Denver and Minneapolis. Bicoastal salaries are often 5% to 10% higher than those not on the coasts. And regarding industry sectors, consulting and finance tend to pay (sometimes considerably) more than marketing, general management or non-profit sector jobs. In short, schools more on the coasts, in large metro areas, or those heavily focused on consulting and finance will have a large advantage. Foreign schools also have made huge inroads into global rankings based on higher salaries (than the U.S.).
Failure to Separate Apples from Oranges: As Malcolm Gladwell pointed out, you can rank all things on a single dimension (miles per gallon, GMAT score, percentage employed at graduation, etc.), but if you want to attempt a holistic or comprehensive ranking, you need to limit the scope to a class, type, or specialty. We rank SUVS, trucks, and sports cars separately for a reason. Rankings often combine one-year MBA programs with two-year ones, 800-student programs with 40-student ones, foreign programs with domestic ones, programs with senior managers as students with programs with 21-year-olds as students. While it’s tempting to do so, because they offer the same basic degree, the results quickly become meaningless.