BusinessWeek’s Big Oops Ranking Moment

‘WHEN WE DO NOT UPHOLD OUR COMMITMENT TO OUR READERS, WE VIOLATE THEIR TRUST’

“We are committed to providing timely, reliable and unique information, and recognize that when we do not uphold our commitment to our readers, we violate their trust,” wrote Paskin. “We know, too, that by participating in our rankings, your institution has placed its faith in us, and we take very seriously any errors that may jeopardize that faith.”

The mistake had less of an impact on the overall rankings, which more heavily weighs surveys from MBA graduates and corporate recruiters. BusinessWeek said that only two schools had their overall position changed by two positions–the University of Minnesota’s Carlson School and Rice University’s Jones Graduate School of Business–while a dozen other schools suffered a one position change in their overall rank.

According to Paskin, “miscalculations in intellectual capital ranks resulted in amended overall rankings for the following 10 American business schools: UC Berkeley (moved from #13 to #14); Columbia University (from #14 to #13); University of Minnesota (#33 to #35); University of Wisconsin, Madison (#34 to #33); Michigan State University (#35 to #36); Rice University (#36 to #34); Boston College (#47 to #48); University of Florida (#48 to #47); Howard University (#59 to #60); and The University of Tennessee at Knoxville (#60 to #59). Among the international programs, rankings changed for York University (#14 to #13) and Imperial College London (#13 to #14).”

BUSINESSWEEK SAID TWICE THAT ITS OVERALL RANKING FOR THE TOP 12 U.S. SCHOOLS REMAINED UNCHANGED

In an effort to minimize its own mistake, the magazine told business schools that its final overall ranking of the top 12 U.S. and the top 12 international business schools announced last November “were not affected by these miscalculations.”

Oddly, Paskin mentioned this fact twice in her ten-paragraph email dated Oct. 4 to the deans of the business schools. “As noted earlier,” she wrote, “the rank of the top 12 U.S. schools in Bloomberg Businessweek’s Best Business Schools 2012 was not affected by these errors. The top 12 remain, in descending order: University of Chicago, Harvard University, University of Pennsylvania, Stanford University, Northwestern University, Duke University, Cornell University, University of Michigan, Massachusetts Institute of Technology, University of Virginia, Carnegie Mellon and Dartmouth College.

This is not the first time a major ranking organization has made a mistake in calculating its ranking. Only last year, for instance, The Economist declared in its 2012 MBA ranking that the University of Virginia’s Darden School of Business was ranked second right behind Chicago Booth. Delighted, Darden quickly issued a news release to trumpet its achievement.

It took only a few days for The Economist to make its own correction, naming Dartmouth College’s Tuck School of Business the second best school in the world. The British magazine conceded it made a mistake when it calculated its faculty ranking for Tuck. With the adjustment, Tuck essentially traded places with Darden.

Truth is, it’s surprising that more mistakes have not made their way into print. That’s because most rankings are composed of thousands of data points. The BusinessWeek ranking takes into account the opinions of thousands of MBA graduates on more than 30 questions, hundreds of corporate recruiters, and untold number of articles in 20 different journals. The breadth and depth of the project, often done by staffers who are juggling multiple duties, makes it vulnerable to error.

IT TOOK BUSINESSWEEK 11 MONTHS TO CORRECT THE MISTAKES

The BusinessWeek error, however, is especially egregious because it took the magazine 11 months to correct the mistake. It did so not proactively but after complaints from some business schools that attempted to replicate the intellectual capital ranking and realized something was terribly wrong. And the magazine failed to fully inform its readers of the correction.

“We are making a commitment moving forward not only to an error-free process – additional safeguards are being implemented now to ensure success – but also to an evolution in the Bloomberg Businessweek Best Business School rankings,” wrote Paskin in the email. “Over the coming months, we will be evaluating our current process to confirm we are gathering the data and sentiments that matter most to prospective students and employers, and we will strive to be as transparent about our process as possible.”

Just how transparent was BusinessWeek to its readers about the mistake? Here is how the magazine changed the wording of its methodology without ever acknowledging that it had made an error that resulted in numerous changes in its ranking. “The final stage of the ranking process involves calculating the intellectual capital rating for the remaining 82 schools, tallying the number of articles published by each school’s faculty in 20 publications that range from the Journal of Accounting Research to the Harvard Business Review, and awarding extra points if a professor’s book was reviewed in the New York Times, the Wall Street Journal, or Bloomberg Businessweek. When the tally was complete, the scores were adjusted for faculty size.”

Paskin, who is also editor of BusinessWeek’s online site, joined the magazine from The Wall Street Journal three days before BusinessWeek released its 2012 rankings on Nov. 15, 2012.

The BusinessWeek rankings are used by more MBA applicants than any other published list of the best business schools, according to a survey of last year’s applicants by the Association of International Graduate Admissions Consultants (AIGAC). Some 69% of the responding applicants said they consulted the BW list, compared with 61% for The Financial Times, 54% for U.S. News & World Report, and 41% for Poets&Quants. Some 40% said they consulted The Economist, while 31% referred to Forbes‘ biennial ranking.

NOTE: The author of this story originated the BusinessWeek MBA rankings in 1988, and for many years managed and supervised them.