When MBA Rankings Lag The Facts

by Jeff Schmitt on

ranking

You can’t put a price on a good reputation. That’s true in business…and even more so in academia. Take business schools, for example. Prospective employers and partners often make assumptions based on where you graduated. List an MBA from a top tier and doors magically open. Fellow alums give you the secret handshake. Your superiors bother to learn your name.  And your peers listen more intently when you speak (at least at first).

A business school’s reputation carries that kind of weight. Question is, how much of that reputation is based on quantitative data over carefully-calibrated branding?

U.S. News and World Report recently attempted to answer that question for undergraduate programs. As the proverbial “gold standard” for postsecondary rankings, U.S. News conducted its second annual analysis of schools whose rank doesn’t match its reputation. Using a methodology developed by Bob Morse, the director of research at U.S. News, undergraduate programs were broken into two categories: “overperformers” and “underperformers.”

WHICH SCHOOLS UNDERPERFORM THEIR RANKINGS?

Here’s how it worked: U.S. News bases its undergraduate rankings on factors like “admissions selectivity, financial and faculty resources, graduation and retention rates, alumni giving, and graduation rate performance.” Each of these indicators is given a particular weight in a formula used to calculate a school’s overall rank. However, U.S. News also factors in “peer assessments,” where administrators and tenured faculty rate other schools using a 5 point scale, with 1 being “marginal” and 5 being “outstanding.”

After ranking schools from highest to lowest on peer assessment, U.S. News subtracts a school’s overall rank from its peer assessment rank. A positive number indicates that a school is an “over performer,” while a negative number reflects an “underperformer.” In Morse’s words, an overperforming school has a “reputation among its academic peers [that] has not kept pace with what it has achieved in the underlying academic indicators. This could be because academic reputation is a lagging indicator – it can take time for a school’s academic peers to understand the real progress of a university.” Conversely, an “underperforming” school’s ranking may be artificially inflated based on the subjective peer assessments of the academics who were polled.

Impressed with this formula, we wondered what it might reveal about MBA programs. In the 2014 U.S. News Business School Rankings, peer assessment scores were derived from “business school deans and directors of accredited master’s programs in business.” According to U.S. News, 43% of the deans and directors surveyed responded, with these peer assessments accounting for 25% of each school’s weighted ranking. As a result, a fourth of a business program’s ranking is based on the opinions of experts who may not necessarily be basing their ratings on empirical data.

A SCHOOL WITH A RANK HIGHER THAN ITS REPUTATION IS BEING UNDERSOLD

Of course, to agree with the premise you have to buy into the notion that the remaining part of U.S. News’ methodology accurately measures the relative rank of the full-time MBA program it purports to measure. Besides the peer assessment portion, the rest of the rank is pretty much based on quality factors: average GMAT and GPA, acceptance rate for applicants, average starting salary and bonus, percentage of class with jobs at graduation and three months later, and an opinion poll of corporate recruiters.

If a school’s core stats award it a rank above the rank given to the school by rival deans and MBA directors, it suggests the school hasn’t done a very good job of tooting its own horn and letting others know that its base quality exceeds its reputation. If a school’s stats show that it is below a rank accorded to the school by peers, it suggests the school has a reputation that is higher than deserved. The school should be working hard to improve its underlying quality measures.

(See following pages for full analysis and tables)

1 2 3 4 Next
  • Enough

    Why should the views of professors in the schools themselves be used to determine whether a school is over/under its ranking? Though its accepted that the rankings are flawed to begin with and exist partially to sell magazines, tenured professors are wrapped way up too much in their own and their peers’ research capability to be able to assess objectively which schools are really having impact in the business profession. Business research and many of the professors who do it increasingly have little to do with the reality of actual business practice. The ability to produce managers that have impact is so much more than analytical reasoning, the main area in which professors excel. Management is as much about synthesis as it is analysis, yet we continue to be beholden to researchers to the very people who often eschew synthetic forms of inquiry for not being rigorous. I think Doug Guthrie’s recent reflections in Forbes on his ouster as dean at GWU are educative here: Even though he was a top professor and researcher who taught leadership and had go-to expertise on China, he crashed and burned when it came to being an actual leader, because he lacked the experience and seasoning to rally people around his ideas: a clear problem of synthesis

    So many business professors have never actually worked in a company as anything more than observers, yet they feel that their research capability and their ability to hammer a case gives them the last word on determining how managers are/should be developed. Let’s get over it.

  • Jimmy

    To summarize: concrete facts vs traditional reputation ranking. Quite interesting and revealing of a certain trend for a school like Thunderbird with a 57 position ranking of difference between what alumni think of their school and its true position in the market today.

  • sss

    Completely agree. The biggest down side of USN is the fact that they stubbornly hold on to “peer assessment” although they are finally reducing its weight in the rankings formula. Professors/ deans at other schools have so much incentive to be biased or untruthful when it comes to rating other institutions.. they have a hard time figuring out what’s going on at their own school…

  • bwanamia

    “The Wharton School of THE UNIVERSITY OF PENNSYLVANIA”

    Problem of Walmart selling a luxury brand.

  • The Truth

    Mediocre schools think they are prestigious because of one successful alum. The true sign of eliteness is how many MBB and top tier BB hirefrom your school and how the firms give your school the dog and pony show.

    UT Austin and USC Marshall may have one MBB hire a year, but they don’t get the dog and pony show treatment HSW get.

  • JohnAByrne

    U.S. News doesn’t survey professors per se, but rather the deans and MBA directors of the business schools for its peer assessment survey.

  • bofdem

    Huh? Are you confusing Pennsylvania State University (System) with the private University of Pennsylvania? If so, you’re so incredibly ignorant that you shouldn’t be allowed to type on a computer keyboard.

    If not, your post does not makes sense.

    If you attended college – not even business school – you’d know enough about Walmart and its EDLP policy that they would never venture into luxury brand(ing).

    Penn and Wharton are completely different – diametrically opposite scenario. Penn *is* a luxury brand in academics, medical, business, and law.

    [I am not an alumni nor in any way affiliated with Penn]

  • http://www.jwmi.com/ Daniel Szpiro

    “Enough”, including the input of deans and directors in the MBA rankings (i.e., a peer review) is not primarily trying to “assess objectively which schools are really having an impact in the business profession.” Instead, the primary purpose of this input is to include an academic evaluation of academic institutions. Business schools are just that – schools. Any school that is part of a tenure-granting university, with research as a part of its mission, is asking to be evaluated on this academic basis as well. According to long-established traditions, academic assessments are peer-reviews.

  • Joe Random

    You could have made your point just as effectively without the vitriol. Many people know Penn is a prestigious academic institution.

    However, the name University of Pennsylvania doesn’t conjure up images of ivy and academics when taken out of that context.

    The original dig at Pennsylvania was unnecessary. But I feel the branding problem still remains. After all, you wouldn’t automatically associate New Jersey with higher ed. But Princeton remains a prestigious brand.

  • Quaker

    Interesting point – whether branding an institution with a distinct name has a positive impact on reputation. Penn remains the only Ivy named after the state in which it was founded. If I’m not mistaken, I think Princeton was called the “College of New Jersey” until the beginning of the 20th century. Look at it now. (Rutgers, on the other hand, is almost as old, but its name hasn’t necessarily elevated its prestige.) Back to Penn, there remains a palpable sense within certain quarters of Wharton that the School is a class above the very university that was innovative to found what became the first collegiate business school. There may have been a time when that disparity was true, but Penn and many of its other schools have long since gotten their acts together. With some of the stuff that’s appeared in the press of late, one must wonder if Wharton is the one that is losing some luster.

  • Danwidth

    I went to Ross about 9 years ago. After initial difficulties, my career has really taken off. I am barely 40 and should be able to retire pretty soon though i would work till i am 70 if I remain healthy. When I looked for a job as a newly minted MBA, it was hard given the lack of brand. The alumni tend to be helpful but they are also not in top tier roles. Now, when someone looks at my resume, they see i went to Ross, chuckle a bit, and ask me why I went there (clear insinuation that i should have gone to Harvard/Stanford/Wharton). I just laugh it off. Truth is i wasn’t good enough to get into those schools and i went wherever someone would take me (I turned down Duke, NYU, Cornell). Michigan gave me a chance and i appreciate that. What you make of your career is up to you. I’ve performed way better than most alumni of the top 3 schools and i attribute luck, hard work, and career savvy to it.

  • Truth Hurts

    I’m glad I don’t want to work for McKinsey, Bain, BCG, or a top tier bulge bracket bank. Once you realize you can get the same “consulting” work from hourlynerd dot com (a Harvard startup) for fractions of what M/B/B charges, for the same associate “strategizing” for you, you’ll think twice about using their services.

  • naan mumukshu

    the degradation of our education system starts when you start conducting assessments like this that do not really reflect the intelligence + ability + drive of the STUDENTS .. its the STUDENTS that make the institution, not the other way around

  • Matt

    Weird to say people laugh at the fact you went to Michigan – it is one of the best schools in the country and your MBA is an extremely respectable top 15 program.

    I guess this does depend on what kind of industries/companies you are working in though..

  • Matt

    Penn State is no “Walmart” either (I use the term loosely because I do love Walmart). It is a top 10 public school per US News and has pretty good programs in Education, Engineering, Law, and yes, also Business.

    Sorry to always be posting stuff like this, but I am continuously amazed at how elitist the people on this webpage can be.

    I don’t mean to be rude, and would choose a nicer word than “elitist” if I could think of any. :)

  • Matt

    John, here is a thought for you. My understanding is that generally US News rankings weigh GMAT, GPA and those kinds of factors more heavily than Businessweek, Forbes, etc. So now consider a program like Fuqua, that is gaining a lot of momentum even with lower GMATs than its peers. It makes sense that US News would have it ranked lower based only on the hard numbers, maybe even lower than it deserves.

    In this regard, you can think of the “underperformers” instead as examples of the flaws in US News’ methodology. That is of course assuming you trust the Peer Assessment scores.. ;)

  • European.Idiot

    Many globally ranked schools are feeding & thriving on this ranking business and students are reeling with mediocre infrastructure & support system. A student is only interested in quality teaching, good infrastructure and excellent career support system. Some brad names like Harvard & MIT sell all across the world while other tend to have a regional appeal only and this is where the school’s intent comes into picture.

  • halo

    I am not sure whether or not you are associated with the aforementioned consulting website, or a naïve current MBA student, but this just shows how the start-up craze-whether it comes from Harvard or Hult is not based in reality. The purpose of every single business is to make as much money as it legally can. And if you think that a service that charges an average of $700 per “consulting” project is a viable businesses than I have a bridge to sell for you.
    I won’t even comment a silly assertion that you can get the SAME consulting from the SAME associates as you would at MBB. Get a handle on reality.

  • Wh

    What business are you in? The only industries I can think of that are almost exclusively populated by H/S/W grads are PE/VC. They are also the most likely to have the type of insecure d-bags that would chuckle in your face if you didn’t go to H/S/W just to make themselves feel better. Talk about soft skills, which bschools are supposed to teach or at least select for.
    At most big companies in industry, though, the majority of people hired come from schools ranked much, much lower. MBAs, even from the top 20 programs, are a minority, considering that most hires are from regional schools that many would consider no-name colleges. So I really can’t see how someone at one of these companies would be a “top 3 or bust” guy.

  • PULman

    I’ll put it another way: “I went to Ross about 9 years ago. When the MBA was an invaluable degree and when it is scarce to find an MBA for your company”

  • Marcia

    University of Pennsylvania is NOT a public school — the public university is Penn StateState. UPenn is private and a member of the Ivy League.

  • public

    Well, it sounds like a public school. Big f****n deal! Who cares if it’s Ivy League. It’s not at the level of Harvard, Yale or Princeton. Even Columbia is better. Most people don’t know that Penn is part of the Ivy League anyway and associate it with PennState. Perception is reality, like it or not.

  • Matt

    Marcia, I was not talking about University of Pennsylvania..

  • Jenny

    I did my MBA at one of those underachiever MBA programs and I thank you for this piece ! I also appreciate John’s annual article re. MBA programs with the worst job placement, usually done after the annual US News MBA ranking. What this does is highlight certain programs that cannot get their act together, rather they fraud students with dubious achievements and often misleading information. Refereeing this grand arena of quality and dubious programs is no easy task given the constant manipulations of ranking data submitted and job placement stats. What it does is allow students and alumni to present the information in a concise manner to said administration or share amongst without being singled out as a malcontent. My program likes to shoot the messenger of reality and Poet & Quants is credible and concise albeit too often focused on Harvard vs Wharton etc.

  • bofdem

    As an anonymous poster, I expect this continued ignorance. The brand is the business school name – Wharton vs. Smeal. Big difference. Now go back to your job at McDonald’s.

Partner Sites: C-Change Media | Poets & Quants for Execs | Tipping the Scales | Poets & Quants for Undergrads