Why The Businessweek MBA Ranking Challenge Matters

When Yale School of Management Deputy Dean Anjani Jain went to Bloomberg Businessweek with his concerns over the credibility of its latest MBA rankings, he was told in rather polite if not condescending terms that he was misinformed. Undaunted, Jain followed up with Caleb Solomon, the editor in charge of those rankings. At one point during the telephone conversation, says Jain, Solomon asked him to wait two weeks before going public with his analysis so that he could consult with the magazine’s two data experts who were away.

What Jain had found was disturbing. His analysis questioned the fundamental methodology used by Bloomberg Businessweek to rank the top full-time MBA programs. Jain argued that Businessweek’s rankings did not align with its published index scores and weights. A recalculation of the published index scores by Businessweek’s stated weights would, in fact, change the positions of 23 of the Top 25 U.S. business schools and 79 of the 84 U.S. schools ranked by the magazine. In some cases, the changes were dramatic MIT Sloan and the Wharton School would fall from seventh and ninth to 21st and 28th, respectively.. The change for Yale, ranked 12th, would by Jain’s calculations rise to eighth.

Jain was challenging the ranking from a position of prominence. As the deputy dean for academic programs at Yale SOM, he is a well-known and highly respected academic with deep expertise in algorthisms. Yet, Solomon seemed not to take his challenge all that seriously. He couldn’t even bother to send his data team an email to clear up a major challenge to the magazine’s highly publicized and influential ranking.

‘WE CAN’T DISCLOSE ALL ASPECTS OF OUR METHODOLOGY’

The resulting articles and Bloomberg Businessweek‘s defense of its ranking and methodology is one of the biggest public challenges to MBA rankings ever mounted. The magazine offers little defense of its ranking, claiming that Jain’s analysis is inaccurate. Jain’s highly detailed and public sharing of his spreadsheets analyzing the reported scores of Businessweek‘s ranking metrics has shown that something is not quite right and hasn’t been since Businessweek changed its methodology in 2018. If the latest ranking was recalculated to reflect the true weights assigned to the ranking, 79 of 84 schools would have different rankings. Among other head-scratching outcomes, Wharton would plunge from an already low ranking of ninth place to 28th if the true weights were assigned to the ranking.

Businessweek continues to maintain that it made no mistakes in crunching the numbers. “As I’m sure you can appreciate, we invest a great deal of time and effort into creating a proprietary ranking that can be neither replicated nor gamed,” wrote Caleb Solomon, the best business schools editor, in an email to Jain. “As I said before, we stand by our rankings and our methodology.”

Solomon, however, offers no plausible explanation for the discrepancies. Instead, he insisted that “we can’t disclose all aspects of our methodology beyond what we’ve already published.”

THE BIG QUESTION: DOES IT REALLY MATTER?

A Yale MBA student sums up the stalemate with a logical quote in the Yale Daily News: “While Dean Jain’s findings are significant given their resulting changes to school rankings, what’s more stunning is their implications for integrity in business journalism,” said Penelope Williams. “BBW has either made a mistake they’re bafflingly unwilling to own up to, or concocted an intentional warping of data to produce the desired result. Whether or not readers use these rankings to make personal or professional judgements, such a warping would be an egregious abuse of trust.”

Perhaps the big question over this controversy is: Does it really matter?

In the online world, lists that rank things are the ultimate clickbait. They attract hundreds of thousands of page views that can then be used to sell advertising. And in the world of newspapers and magazines, the same is true. Rankings result in higher newsstand sales that lead to more advertising dollars. So there is a financial incentive to create and publish rankings of any kind.

WHY MBA RANKINGS MATTER TO BUSINESS SCHOOLS

More importantly, though, rankings matter to the schools and their stakeholders. Beyond the obvious bragging rights, they have a significant impact on application volume, student choice, alumni donations, and even faculty recruitment. For many business school deans, rankings have become something of a scorecard that measure their leadership abilities and success. When a ranking improves, it’s a opportunity for celebration. Deans eagerly inform their stakeholders of the improvements in emails and talks. The school’s media machine produces news releases and social media posts to trumpet an advance. Everyone–students, alumni, parents, staff and faculty–feel good about their school when it gets a better ranking. When rankings go the other way, there are no press releases or congratulatory email messages from the dean. Bad news is publicly ignored, though privately, there are more often than not concerns, accusations and demands for changes.

For the vast majority of business schools, the rank of their MBA programs assumes the more import. Typically, full-time MBA programs are the flagship offerings of business schools. These degree programs are also ranked by more organizations than any other degree by U.S. News, the Financial Times, The Economist, Forbes and a bevy of other organizations. Without question, full-time MBA programs are the most scrutinized and ranked degrees offered by a university. So these rankings loom large, often casting a shadow over the entire business school and all its other programs.

This is why the Bloomberg Businessweek controversy over its full-time MBA ranking is important. Rankings have consequence. Rankings really do matter. When the Association of International Graduate Admissions Consultants (AIGAC) surveyed MBA applicants in 2019, the group found that nothing mattered more in school selection than rankings and reputation–two factors that have largely become inseparable. Asked the “top factors influencing specific choice of schools,” 63% of MBA candidates cited “rankings,” with an equal number citing “reputation” (see chart below).

MBA APPLICANTS SAY RANKINGS ARE THREE TIMES MORE IMPORTANT THAN FACULTY QUALITY IN CHOOSING A SCHOOL

In fact, applicants claim that rankings are three times more important in their choice of a business school than faculty quality, cited by only 21% of the nearly 2,000 respondents. Rankings and reputation also trumped the cost of the program (cited by 23%), the GMAT or GRE score achieved by a candidate (27%), or the alumni network (38%).

Another study of prospective MBA students in 2021 by the Graduate Management Admission Council (GMAC), the administrator for the GMAT test, found that candidates rely heavily on rankings as a key information resource for researching programs. Some 45% of domestic MBA applicants and 49% of international MBA applicants said they relied on rankings as much as they did on school-related websites.

As one academic studying rankings once discovered, rankings inevitably shape the trajectory of a school’s reputation, to the point where "rankings and reputation are no longer independent concepts. "To put it simply, the ranking of a college has become its reputation among other college leaders—the same leaders who bemoan the influence of college rankings," wrote Michael Bastedo, a professor of education at the University of Michigan.

THE DARK SIDE: RANKINGS HAVE CAUSED SOME SCHOOLS TO CHEAT

The importance of rankings also is underscored by the schools that have been caught cheating to gain a higher rank. In early November, the former dean of Temple University's Fox School of Business will go on trial for allegedly reporting fraudulent data to U.S. News to game the system (see Anatomy Of A Business School Rankings Fraud). The inflated metrics Temple reported under that dean's leadership allowed the school's online MBA program to be ranked best in the nation for four consecutive years from 2015 to 2018 when U.S. News removed the Fox School off its MBA ranking after discovering that the school misreported critical data on its program.

The Businessweek ranking of full-time MBA programs was first published in 1988--before U.S. News began ranking business schools. The list assumed so much importance in the business school world that it became the subject of numerous academic studies. In one landmark study in 1996, sociologists Kim Elsbach and Roderick Kramer studied the impact of the Businessweek ranking on business school leadership and discovered the dark side of these lists. They found that Businessweek rankings that were inconsistent with a school’s self-perception often led to conflicts over mission, goals, and identity. The consequence of that gap between the perceived quality of a program and its actual Businessweek ranking, the two concluded, could encourage leadership attempts to bring the ranking back in line with their self-perception—by any means necessary.

Sure enough, the rankings scandal at Temple is hardly the first time someone has been caught trying to manipulate the results. In 2013, Tulane University’s Freeman School of Business admitted that it falsely inflated its average GMAT scores by an astounding 35 points for five consecutive years from 2007 to 2011. The school also conceded that it had falsely increased the number of completed applications it received by an average of 116 applications over the same time period to boost its U.S. News' ranking (see Who Cooked The Books At Tulane?).

METHODOLOGICAL RIGOR, TRANSPARENCY AND DATA INTEGRITY RESTS UPON RANKING ORGANIZATIONS

Sadly, rankings--however flawed--are so consequential that some schools are willing to cheat. MBA applicants are obsessed with them. Alumni obviously want their alma maters to improve their status by moving up on such lists. Deans are judged--fairly or unfairly--by how well their schools perform on the rankings. And some faculty are swayed by these lists in deciding whether to accept a school's offer of employment.

These are all reasons why rankings need to be done with great care and consideration. As Anjani Jain, the deputy dean overseeing academic affairs at Yale University's School of Business, rightly points out: "Media rankings of B-schools influence tens of thousands of prospective students each year, and participating schools should indeed compile the requested data with unimpeachable integrity and diligence. A parallel expectation of methodological rigor, transparency, and data integrity rests upon media organizations producing the rankings and seeking public trust."

DON'T MISS: DID BUSINESSWEEK BOTCH ITS MBA RANKING? or BLOOMBERG BUSINESSWEEK'S MBA RANKING CANNOT BE REPLICATED BY THE PUBLISHED DATA & METHODOLOGY