Flawed Again: Bloomberg Businessweek’s New MBA Ranking

Bloomberg Businessweek Mutes its Claim About Stakeholder-Generated Weights

In the last several years, Bloomberg Businessweek’s (BBW) annual MBA ranking has made the following bold claim about their methodology:

“The Bloomberg Businessweek Best B-Schools ranking starts from the basic premise that the best judges of MBA programs are graduating students, recent alumni, and companies
that recruit MBAs. We ask those stakeholders questions about everything from jobs to salaries, classroom learning to alumni networks. Their answers form the heart of this
ranking. … Rather than define the relative weighting of the indexes ourselves, as most rankings do, we let the stakeholders define them.”

Yet, by failing to normalize the raw scores on the five ‘indexes’ they use in the ranking, they have substantially distorted these stakeholder-generated weights in each of the last several years. These distortions have been readily discernible in the data they publish and have received considerable media coverage in Poets&Quants and elsewhere.

The claim quoted above has disappeared from the stated methodology of BBW’s 2023-24 ranking, but they have continued sourcing these weights from their survey respondents. The reader of BBW’s methodology is given no hint that BBW’s ranking calculation continues to distort these weights. These distortions are caused, as in previous years, by their failure to normalize the raw scores (on at least three of the five indexes) before applying the weights. The exercise of crowdsourcing index weights from survey respondents therefore remains pointless and deceptive. BBW does deserve some credit, however, for dampening the reader’s expectation that the crowdsourced weights will be preserved by their computational methodology!

THE EXTENT OF THE DISTORTION

Applying BBW’s stakeholder-generated weights for the five indexes of the ranking (Compensation, Learning, Networking, Entrepreneurship, and Diversity) to the index scores they publish yields the published order for the 75 schools ranked by BBW in 2023-24, but the index scores are not normalized. (It is easy to check if a data vector is normalized; if it is, further normalization leaves it unchanged.)

To see how much the published ranking distorts the stakeholder-generated weights, I computed z- scores from the published index scores and then used a constrained optimization model to determine the effective index weights which, when applied to the z-scores, produce the smallest variance from the published ranking. When applied to the z-scores, the effective weight vector so computed replicates the published BBW ranking of 75 U.S. B-schools (with very minor discrepancies caused by the limited precision of available data):

The analogous computation for the last two rankings had yielded a similar pattern of distorted weights:

A close reading of the BBW methodology page for the 2023-24 ranking provides a clue as to why the ranking continues to underweight learning, networking, and entrepreneurship (while overweighting compensation and diversity): The index scores for both compensation and diversity include some non-survey data and are scaled from 1 to 7 where 1 and 7 correspond to the worst and the best values, respectively. The scores on learning, diversity, and entrepreneurship are all based only on survey responses on a 1 – 7 Likert scale, and are not re-scaled so that the worst value corresponds to 1 and the best to 7.

An equivalent way to think about the distortion of stakeholder-generated weights is that if these weights are preserved (by using z-scores or normalized versions of raw data), the rank order of schools is substantially different from that published by BBW. For schools ranked in the top 25 in 2021-22, the table below shows the weight-preserving ranks (using z-scores) for 2021-22, 2022-23, and 2023-24 juxtaposed against the BBW ranks in those years:

The full list of 75 U.S. schools ranked by BBW in 2023-24, and their corrected (i.e., weight-preserving) rank order, is available in a spreadsheet that the interested reader can download. Also
included in the spreadsheet are calculations showing that the effective index weights stated above replicate the published 2023-24 BBW ranking when applied to z-scores.

A concluding thought. I don’t intend to make a habit of highlighting the ranking’s distortions each year, especially since BBW is likely to continue this practice. Some readers may wonder why BBW
doesn’t employ the obvious fix: apply the stakeholder weights to normalized data, which would eliminate the distortion of weights? Presumably, they don’t like the ranking that this fix would have
produced, placing, for example, Wharton at #19 and Yale at #20, each well behind Michigan (#6), Cornell (#9), Georgia Tech (#11), and Emory (#13) this year.

The other consequence of preserving the stakeholder weights is a much greater year-to-year volatility in the ranks (readily seen in the table above), which would undermine the ranking’s credibility. But, to reiterate what I said last year, if BBW did believe in letting their stakeholders define the relative importance of their indexes, they would have no qualms about publishing the resulting ranking, notwithstanding its departure from conventional wisdom or its volatility. And for schools that currently get buried under the weight of current orthodoxy about rankings, it may create the opportunity to gain prominence.

Anjani Jain is the Deputy Dean for Academic Programs & Professor in the Practice of Management. He holds a bachelor’s degree in physics and mathematics from Indore University in India, an MBA from the Indian Institute of Management, Ahmedabad, and a PhD in operations research from the University of California, Los Angeles. He joined the faculty of the Wharton School of the University of Pennsylvania in 1986 and served for 26 years before joining Yale SOM. At Wharton he taught courses in operations management and management science at the MBA, PhD, and undergraduate levels. In 1993 he became director of Wharton’s MBA Program and was instrumental in several curricular initiatives. From 2000 to 2010 he served as vice dean and director of Wharton’s Graduate Division and from 2010 to 2012 as vice dean of Wharton’s MBA Program for Executives.

DON’T MISS: Harvard & Wharton Slump In New 2023 Businessweek MBA Ranking or The Bloomberg Businessweek 2022-23 MBA Ranking Remains Flawed