BusinessWeek’s Big Oops Ranking Moment

by John A. Byrne on

Oops BusinessWeekArguably the most famous of all the cover stories published by BusinessWeek magazine carried the simple yet highly provocative headline: Oops! The story chronicled how many of the companies extolled for their excellence had badly stumbled.

Now Bloomberg BusinessWeek has its own “Oops” to contend with, a mistake that significantly impacted its MBA ranking of last year. The errors in the magazine’s calculations change the overall numerical rank of a dozen schools–10 in the U.S. and two non-U.S. schools. More significantly, the “intellectual capital” rank of some schools moved by as many as 10 positions. Harvard Business School, for example, fell to 19th from 9th in intellectual capital as a result of the revision.

This time, Bloomberg BusinessWeek was not so eager to publicized its own failings. The magazine quietly revised its rankings online a month ago with little fanfare or notice. At the top of its corrected table, in small hard-to-read type is this rather vague, less-than-clear statement: “(Corrects to revise 2012 overall rankings, Ranking index and 2012 Intellectual Capital rankings following errors in the calculation of the Intellectual Capital score.)”

Even BusinessWeek’s updated methodology does not acknowledge the mistake. Instead, the magazine simply crosses out part of the description of the methodology (see below).

BUSINESSWEEK ADMITS IT CHANGED METHODOLOGY WITHOUT TELLING ITS READERS OR THE BUSINESS SCHOOLS

Even worse, the magazine had to admit that it had changed its methodology for ranking the intellectual capital produced by business schools without official notice or even a mention in an accompanying story describing the way it tabulates the results of academic research.

In an email obtained by Poets&Quants, Bloomberg BusinessWeek Assistant Managing Editor Janet Paskin informed the schools last month that “we have discovered that some errors were made in calculating one element of the 2012 ranking, the intellectual capital score, which accounts for 10 percent of each school’s total score.”

Paskin went on to describe the problem in more detail that the magazine has still failed to disclose to its readers. “In a recent review of last year’s intellectual capital rankings,” she wrote, “we discovered errors in how we collected and tallied faculty research. We also realized that we had failed to update our stated methodology to reflect our current practice: Faculty are no longer awarded points for reviews of books they have authored, and the time period surveyed spans four years, not five.” In response to a question on how the errors occurred, Paskin said in an email that “inn a few instances, changes in faculty names were missed; some articles were published online but not in print, and vice versa, and were missed.”

THE MISTAKES RESULTED IN SOME DRAMATIC CHANGES IN THE MAGAZINE’S INTELLECTUAL CAPITAL RANKINGS

The screwup by BusinessWeek resulted in some especially dramatic changes in the intellectual capital ranking given to some schools. Yale University’s School of Management, for example, zoomed up eight places to 9th from an inaccurate ranking of 17th. The University of Chicago’s Booth School of Business plunged to 11th from fifth. Harvard Business School dropped 10 places to 19th from ninth. Rice University’s Jones School climbed to 21st from 27th. And Notre Dame moved up six places to 31st from 37th.

All told, BusinessWeek admitted that “miscalculated intellectual capital scores” were made by BusinessWeek for 40 American business schools and 10 international schools–a fact that was buried deep inside the email sent to the schools. The detail regarding the significant drops in intellectual capital rank was in the eighth paragraph of the ten-paragraph email written by Paskin. In journalism school, they call that “burying the lead.”

Paskin’s email, moreover, contains a link to what she refers to as the corrected rankings. However, BusinessWeek makes no mention of its mistake or how its own errors in calculating the ranking resulted in significant changes on the table.

1 2
  • Vucla

    Given how prominently Paskin is featured in the story, it’s little unfair to withhold until the last sentence the fact that she joined BW just three days before the flawed rankings were released. It seems to me that she was put in a very difficult position, trying to correct the errors of her predecessors and/or staff members without showing anyone up.

  • CMcAboy

    Agreed Vucla. In my opinion, Paskin displayed great character in this article. Instead of blaming her predecessors, she stood up for her employer.

  • Dan

    Does anyone even give credence to the BusinessWeek rankings anyway?

  • RankingsGizmo

    Dan,
    Aside from BW being traditionally regarded as one of the premier ranking systems, IMO the people that generally give credence to the BW rankings are likely those whose schools or target schools perform well in the BW rankings. Conversely, the folks that likely generally complain the most about the BW rankings are those whose schools or target schools perform poorly in the BW rankings.
    This is the same with USN and other rankings systems – generally you do not hear from the many who are pleased with the results and instead hear from the many who are not. Logical and natural to human behavior IMO.

  • Dan

    My confusion was coming purely from a perusal of the top 15 schools. I’ve never spoken to anyone who would rank schools like Duke/Cornell/Umich ahead of schools like MIT/Columbia/Hass. This is not to say that those aren’t good programs, I’m just speaking from what I’ve heard from current students and applicants (some of which went to the former schools). I get that students at schools getting the BW bump would pleased, but from a general applicant perspective, I don’t think this jives with the perceived reality (which is very important given the kind of degree the MBA is).

  • RankingsGizmo

    I hear you fully on perception on the one hand, but I guess I have a problem with that becuase aside from the H/S/W – as almost unmovable positions, I think the rest of the rankings do and can move around. And so, I kind of dislike perceived reality becuase it makes rankings stale/stagnant and based on the past, instead of looking ahead. Prospective applicants focus on ideas like – “well that is just the way it is because it has always been”. Not a good way way to think IMO, as history has proven.
    For example, Chicago Booth has moved into a challenging position in recent years accross many rankings for a top 4 spot. You mention Duke – also widely regarded accross many rankings now as challenging for around around the top 10. Cornell also has made huge strides and its applications have surged. It is also developing a campus in NYC, which has garnered enthusiasm.
    I think rankings should account for innovation and change and so perceived reality can also be dangerous, just as it is also comfortable and stable.
    My thoughts only. thx

  • Dan

    That’s the, plainly put, “unfair” part about the rankings. Reputation is very sticky, and it takes a long time to wash off. The aforementioned programs are definitely making strides that should be recognized, and I can see them being strong top 10 school in the future, but at present, I think you’d be hard pressed to find people who’d rank them ahead of places like MIT/Columbia for example (the competition between Columbia/Cornell is a primary reason).

    Side note: the sentiment this application cycle seems to be that the perennial H/S/W trio is shifting towards H/S—–/W

  • RankingsGizmo

    Agreed and I also agree that the sentiment does show a wider space between H/S building distance with W. Also some argue that Chicago Booth may be now essentially tied or have surpassed W? We shall see. I think the real excitement is watching those schools in the 8 – 15 range, which include Duke, and Cornell and many others depending on which ranking you look at (Virginia, Yale, NYU, Michigan, etc). This is the band that is always trying to penetrate the top 10 and seems to have the most potential for change. Fun to watch.
    Good luck to you.

  • Orange1

    Correlate BW,US News and FT. US News and FT correlate well while BW is all over the place compared to them. Three tiers once and for all. To argue about whether #10 is better than #12 is fruitless.

  • Orange1

    I would also add UCLA. It is really a tier. Sometimes Carnegie Mellon and Texas are included. Personally, cannot believe the education or students vary between these schools or ever will. More on where you want to live (California, go to UCLA, NYC, Cornell or NYU) or do (healthcare, Duke, non profit, Yale)

  • Orange1

    Think of the schools ranked 10-17 as also in a “what’s the next hot place” momentum. In 2000, we were all told Duke would be great, then Yale (name, new dean, etc.), Duke again and now Cornell (wow, they are going to Roosevelt Island?). What exactly has changed with them? Students are smarter than they used to be, professors all of a sudden developed better insights? Employers said they are now much more impressive?
    It’s like college football. USC is hot for a few years (gotta play for Pete Carroll), then Texas, Alabama takes its place (must be coached by Nick Saban as he is a Bellichick disciple), and now even Saban might be looking for a new challenge.
    MBA’s get a swirl of activity around a new dean or campus, lots of PR, and then things get down to what happens in the classroom.

  • RankingsGizmo

    I don’t disagree with your insights at all. In fact, they make a lot of sense. Schools in the 8 to 17 or even 20 range are often splitting hairs with one another IMO, and so your point about a tier system is very valid. Any top 20 school is excellent though IMO. But it is easy to get hung up on rankings nevertheless. I do think though that it is important for schools to try and innovate and change and build on momentum and new needs. A rising tide raises the bar for all ships IMO.

  • BSchool Grad

    Thank you for this story. Transparency should matter, from the perspective of journalists as well as the business schools. Speaking of corrections: The journalism term is “burying the lede.”

  • Tipster

    I’m fairly certain the changes impacted more than just rankings. Stats posted for a few schools also changed. For example, I am almost positive that stats
    for the Trulaske College of Business changed substantially. Two stats I noticed that changed were
    employers and salary – the later increased by roughly $10,000.

  • Tipster

    For the record, I just realized that I misspelled “latter” in my earlier post

Partner Sites: C-Change Media | Poets & Quants for Execs | Tipping the Scales