Bloomberg Businessweek To Dean Questioning Its Ranking: Fuggedaboutit!

In response to Bloomberg Businessweek’s October 14th letter stating that they “fully stand by [their] ranking,” I sent two messages to cognizant editors to persuade them that the questions raised about the ranking require a fuller response than what they have offered so far.

I had retained some hope that they would be open to a discussion. It appears now that they have foreclosed this option. Because the matter is already in the public domain, with implications for both business schools and their stakeholders, I am now disclosing this communication publicly (with names of BBW editors redacted).
This correspondence has also been an effort on my part to state the issue in clear, concise, and non-technical terms, but I cannot match the buttoned-down conciseness of the BBW’s non-response! Here is the correspondence in chronological order:

‘BY DESIGN, OUR PROPRIETARY RANKING CANNOT BE REPLICATED OR GAMED

From: [cognizant editor at Bloomberg Businessweek]
Sent: Thursday, October 14, 2021 6:11 AM
To: Jain, Anjani <anjani.jain@yale.edu>
Subject: Bloomberg Businessweek B-School Ranking

Dear Anjani,

I am writing in response to your inquiry about Bloomberg Businessewek‘s b-school ranking. We have formally requested a correction to John Byrne’s story in Poets&Quants and your accompanying analysis. Both are inaccurate. Neither the publication nor Yale had access to the raw scores that are used in calculating the ranking.

Once again, we calculate Bloomberg Businessweek‘s B-school ranking by using raw scores, which are not published, and by applying index weights as exactly stated in the methodology. By design, our proprietary ranking cannot be replicated or gamed using published data. Disclosing our raw data would create the possibility that the rankings could be reverse-engineered or gamed by a school for an unfair advantage. In addition, our methodology is repeatedly and carefully vetted by multiple data scientists. We fully stand by our ranking and our methodology.

Sincerely,
[Cognizant editor]

‘I URGE YOU TO PAY CLOSE ATTENTION TO MY ARGUMENT’

On Oct 15, 2021, at 10:56 AM, Jain, Anjani <anjani.jain@yale.edu> wrote:

Dear [Cognizant editors],

I wanted you to be aware of the attached response, which I’ve posted on my LinkedIn page.  I also wanted you to be aware of the article revealing the problems with your 2018 and 2019 rankings.

I urge you to pay close attention to my argument for why, even under the charitable explanation of what happened—i.e., that you applied the weights before normalizing or standardizing the data—the published ranking makes a lie of your stated methodology by greatly distorting the index weights.  This is an incontrovertible mathematical fact which cannot be rebutted by the claim that your private computations with raw data legitimately produce a different ranking.  They do produce a different ranking, but the computation is a serious statistical error that distorts the much-touted and crowd-sourced index weights.

And if this is indeed what happened, then you should re-do the computation, apply the weights after normalizing the raw data, and publish an amended ranking.

I understand why this departure from your dug-in position is difficult, but I hope that ethical considerations will ultimately prevail within the Bloomberg organization.  You refer to “multiple data scientists” having vetted your computations.  Was a single one of them not a Bloomberg employee?  I urge you to retain an independent statistician to vet the computation.

My last comment is that I have done a similar analysis of the USN&WR ranking of b-schools, where I can replicate the published ranking quite closely (the departures from the published version being the result of lack of precision in the published data).  I can also report that USN&WR does standardize the raw data before applying their weights.

Best,
Anjani

‘YOUR DATA SCIENTISTS KNOW WHY THIS MATTERS’

From: Jain, Anjani
Sent: Wednesday, October 20, 2021 10:36 AM
To: [various editors at Bloomberg Businessweek]
Subject: Re: Bloomberg Businessweek B-School Ranking

Dear [Cognizant editors],

Though your non-response to several previous communications does not make me optimistic this time, I wanted one more time to ask you a simple yes-no question, which your data scientists should know right away:

Did you normalize the raw data (i.e., re-scale linearly to a common scale such as 1-7 or 0-100) before applying the index weights?

Your data scientists will also know why this matters. To claim that the answer to this question is proprietary is to resort to abject obfuscation. If the answer is ‘no’, then your computation has greatly distorted the crowdsourced weights and thus the ranking. This can still be corrected and the ranking amended. If the answer is ‘yes’, then the normalized index scores you published have sufficient information for your ranking to be replicated with the stated index weights. And therefore the wide disparity between the published ranking and what the weights yield is the result of some undisclosed post-calculation manipulation not mentioned in the methodology.

I hope you also see that the binary nature of my question also implies that there are only two possible explanations of the disparity between the published ranking and the one that anyone can compute from your data.  To refuse to answer the question is to invite speculation about the underlying reason, and I think you will agree that neither is good for the future of the BBW ranking.

With best regards,
Anjani
[signature]

‘WE DON’T PLAN TO DISCLOSE ANY MORE ABOUT OUR METHODOLOGY THAN WE ALREADY HAVE’

From: [cognizant editor]
Sent: Wednesday, October 20, 2021 1:15 PM
To: Jain, Anjani <anjani.jain@yale.edu>
Subject: Re: Bloomberg Businessweek B-School Ranking

Dear Anjani,

We don’t plan to disclose any more about our methodology than we already have.

Sincerely,
[cognizant editor]

Anjani Jain is the deputy dean for academic programs at Yale University’s School of Management. His research interests include the analysis and design of manufacturing systems, optimization algorithms, and probabilistic analysis of combinatorial problems. He joined the faculty of the Wharton School of the University of Pennsylvania in 1986 and served for 26 years before joining Yale SOM.

DON’T MISS: DID BUSINESSWEEK BOTCH ITS LATEST MBA RANKING? or A CHALLENGE TO BUSINESSWEEK’S EUROPEAN & Asian MBA RANKINGS