Are Academics Out Of Step With Recruiters & Students?

HAS THE PEER ASSESSMENT GONE ON AUTOPILOT?

The disparities between academic and recruiter surveys produce an insidious cause-and-effect that cascades down the rankings. These higher “peer” scores provide an advantage to top-tier programs, principally since peer scores carry a 10% higher weight than recruiters. In other words, academic sentiment, often far removed from realities found on other far-flung campuses, can cancel out the opinions of recruiters, who often have direct contact with students and can generally vouch for alumni performance in their firms. Bottom line: Any hard-won progress at schools outside the T7 will, in all likelihood, get swallowed up by academic opinion (in the U.S. News ranking, at least).

These peer assessments also act as self-fulfilling prophecies. Take Harvard Business School as an example. Academics have given it a 4.8 average every year starting with the 2007 U.S. News rankings. Alas, that falls a year short of Stanford, which has managed a 4.8 for 12 years and counting. At MIT, that average is 4.7 for the past seven rankings, with NYU Stern posting a 4.2 for six straight. Not to mention, Emory Goizueta has been flip-flopping between 3.6 and 3.7 every ranking since 2006. In other words, 25% of a U.S. News ranking is almost a given for top schools. It almost makes one wonder if the same people are doling out the same scores, year after year, using their own personal logic. It also begs the question: Does every dean and MBA head who comprise U.S. News’s sample truly take an intensive look at how their peers evolve year after year?

Of course, a certain score isn’t necessarily a given. In the 2017 rankings, Yale SOM climbed from a 4.1 to a 4.3 (after being scored at 4.2 from 2012-2015). By the same token, Penn State slipped from 3.4 to 3.2 in 2017, a red flag considering that Smeal had been slotted at 3.4 in nearly every ranking since 2007. Still, Smeal was still highly regarded by recruiters with a 3.5 average.

Over a five-year period, the stagnation in academic sentiment becomes even more evident. Aside from Penn State, just one school (UC-Irvine) either gained or lost 0.2 of a point or more since the 2012 U.S. News rankings. Over that period among the top 10, three schools (Haas, Tuck, and Yale) gained .01 of a point among academics, with Wharton losing .01 and six programs holding steady. The pattern holds when opened up into the top 25, with 17 schools carrying the same average (and another six up by 0.1 and two down by 0.1). Indeed, it’s almost as if their scores were frozen in time.

ARE ALUMNI AND STUDENTS BETTER POSITIONED TO GAUGE A PROGRAM’S PERFORMANCE?

As noted earlier, such complacency is amplified from the likelihood of U.S. News using many of the same survey respondents — not to mention a simplistic evaluation criteria and a stunted five-point grading scale that produces clusters of the same score averages with minimal differentiation among MBA programs. So just how out of step with reality are the U.S. News “peer” surveys? It helps to look at a different set of respondents from the Bloomberg Businessweek rankings, which are even more heavily reliant on qualitative measures than U.S. News. In fact, 80% of the Bloomberg ranking is based on three surveys: employers (35%), alumni (30%), and students (15%).

Setting aside employers (a comparison to come in a future article), alumni and students are uniquely positioned to offer a shrewd comparison with academics. Alumni have already gone through these programs, so they understand the value of the degree — along with the ROI from the network, curriculum, and experience. Similarly, students can speak to whether the brand hype lives up to its execution. The verdict: The U.S. News rankings might benefit from academics spending more time in the field.

Not surprisingly, academics and alumni agree that Harvard and Stanford rank in the top two in both rankings. They also found common ground on UC-Berkeley Haas (sixth with U.S. News academics and fourth with Bloomberg Businessweek alumni), Duke Fuqua (eighth and seventh, respectively), Dartmouth (11th and eighth), Yale (11th and ninth),  Virginia Darden (14th and 18th), and Cornell Johnson (14th and 10th).

That said, there was a sharp difference in rankings at several schools, where academics were more apt to score higher. That is headed by the University of Chicago’s Booth School of Business. Despite being ranked third by academics, alumni pegged Booth at 29th — below other Midwest favorites like Indiana University and Washington University. There is a similar dynamic at the University of Michigan (eighth versus 28th) and NYU Stern (13th versus 35th), with MIT Sloan (third versus 15th), Columbia Business School (eighth versus 19th), and Wharton (third versus 14th).

BUSINESSWEEK ALUMNI SURVEY HAS PROBLEMS OF ITS OWN

Does this mean these programs are overrated in U.S. News’ peer assessments? Probably not, as Bloomberg‘s alumni ranking comes with a few caveats. On the plus side, there is greater clarity on the “who” of the alumni survey. It was based on 12,773 survey responses from students who earned their MBAs from 2007-2009. Although Bloomberg‘s 29.4% response rate fell short of the rate reported by U.S. News’ peer assessment, it does include, at minimum, 30 responses per school. Alumni results also account for more weight in the Bloomberg ranking (30%) than peer assessments do for U.S. News (25%).

That said, it isn’t a pure satisfaction rating. For one, Bloomberg only shares a ranking against other schools, not an actual score like U.S. News. Even more, its alumni ranking is based equally on three criteria: median compensation increase, job satisfaction, and MBA feedback. As a result, just a third of the score was derived from actual graduate sentiment about the school itself — which was expressed through a series of 16 questions “about their MBA experiences, the specific impacts their MBA programs have had on their careers, and whether they would recommend their program to others.” In other words, despite a healthy mix of inputs and outputs with Bloomberg, the alumni ranking lacks the same scoring transparency as U.S. News. In this case, readers know neither which questions were asked nor how the collective answers compare against the weights of responses from salary increase and job satisfaction.

Still, the alumni ranking serves a useful purpose in taking the temperature of alumni opinions. As noted above, certain programs rank lower than people might expect with Bloomberg Businessweek, an indication that certain areas, such as classroom experience or ROI, may be askew at certain touchpoints. Unfortunately, Bloomberg Businessweek doesn’t break this ranking apart so readers can pinpoint exactly where a school may have issues.