Booth A Likely Victim Of BW Ranking Changes

Bloomberg BusinessWeek today (Nov. 10) revealed that the University of Chicago’s Booth School of Business, which topped its MBA ranking for the past four consecutive times, lost ground in both its corporate recruiter and student surveys.

Booth fell to third from first in BW’s employer survey, while it slipped to 15th from 11th in the magazine’s survey of MBA graduates. Those changes likely mean that when BusinessWeek publishes its 2014 ranking tomorrow morning, Chicago Booth will have lost its No. 1 status.

The disclosures were made in an article published today describing new changes in the methodology used by the magazine to rank full-time MBA programs. Those alternations are likely to have a big impact on the numerical ranks of the business schools. Several underlying changes in the methodology, in fact, appear designed to increase volatility in the ranking, which often lends less credibility to the results.

BUSINESSWEEK INCREASED ITS SURVEY POOL OF RECRUITERS BY NEARLY 15-FOLD

In 2012, the last time BusinessWeek published a full-time MBA ranking, Harvard was second, with Wharton third, Stanford fourth, and Northwestern University’s Kellogg School of Management fifth.

BusinessWeek said it would rank 85 U.S. schools and 27 international business schools this year, the largest group of full-time MBA programs assigned numerical ranks by the magazine since it began the ranking in 1988. In all those years, only three schools have occupied the top spot: Besides Chicago Booth, Wharton and Kellogg have also sat atop the ranking.

The BusinessWeek list is based on three components: a survey of corporate recruiters, which is given a weight of 45% in the overall ranking; a survey of the latest graduating students, which is also give a 45% weight, and the publication record of faculty at each school, dubbed an intellectual capital rating, which accounts for the remaining 10% of the methodology.

IN STUDENT SATISFACTION, BOOTH SLIPPED FROM NO. 2 IN 2010 TO NO. 11 IN 2012 AND NOW 15TH

It is usually difficult for a school to top the BusinessWeek list when it fails to gain a first-place finish in the employer survey. Kellogg accomplished this in 2004 and 2002, when it respectively ranked third and second, among recruiters. In those years, however, Kellogg’s recruiter rankings were offset by high student ratings, No. 1 in 2004 and No. 2 in 2002.

When Chicago Booth first emerged as BusinessWeek’s top MBA program eight years ago in 2006, MBAs gave their MBA experience the highest scores, allowing Chicago to finish first in student satisfaction. That was also the case in 2008. In 2010, however, Booth slipped to second and last time in 2012, fell to 11th place in the student satisfaction portion of the ranking. A decline in both employer and student surveys this year would be hard for Chicago Booth to overcome and maintain it’s No. 1 rank.

Chicago’s fall would occur in a year in which BusinessWeek is making the most substantial changes to its methodology ever. The most dramatic alternation occurs in how BusinessWeek decided to survey corporate recruiters. The magazine substantially increased the number of surveys it sent to recruiters by nearly 15-fold. BW said it surveyed 8,358 recruiters at 4,931 companies this year, up from only 566 recruiters at 566 companies two years ago.

Some 1,320 recruiters responded, representing 614 employers. But the median number of schools rated by the respondents was only three, suggesting that many of the recruiters had highly limited exposure to the business school market and little perspective with which to judge one school’s MBA graduates from another.

MAGAZINE SAYS YALE AND CARNEGIE MELLON HAD NOTICEABLE ‘ENTHUSIASTIC  ALUMNI RECRUITERS’

The increase in sample size results from a decision to survey every recruiter whose name was provided to the magazine by the business schools. In the past, BusinessWeek editorial staff did more front-end due diligence to ascertain the list of mainstream MBA recruiters and then sought to survey only the top human resources official responsible for firm-wide MBA recruitment. That system avoided an inevitable bias that would result when surveying alumni who return to their alma maters to recruit MBAs—a common practice in corporate recruiting.

BusinessWeek said it made at least a partial adjustment to account for this prejudice, but conceded that it still played a role in its new ranking. That’s because BW’s new employer score is based on two components: a school’s average rating by employers as well as the number of good ratings a school receives. The magazine excluded alumni scores for the latter portion of the employer score, but not the former component which accounts for half its employer ranking.

“Alumni tended to rate their own school significantly more favorably than non-alumni rating that school,”  said Jonathan Rodkin, who oversaw this year’s ranking methodology. BusinessWeek said that Yale University’s School of Management and Carnegie Mellon University’s Tepper School of Business “were scored by enthusiastic alumni recruiters.”

BusinessWeek, in reporting on its new methodology, said that when a recruiter who was an alum of a school rated his or her alma mater, the scores were about 18% higher than the average non-alumni rating. Some 38 schools in the sample derived at least 25% of their ratings from alumni, while five schools derived “at least 50%” of their ratings from alumni, according to Rodkin. That means that alumni played a substantial role in ranking nearly four in ten schools–43 out of 112–in this year’s BusinessWeek survey.

BOOTH A LIKELY VICTIM OF THE CHANGE IN METHODOLOGY

“To correct for this imbalance, we excluded alumni ratings from the number of good ratings a school received,” added Rodkin. “Alumni opinions of their own schools, however, remained a factor in the average (quality) rating.”

It’s highly likely that Chicago Booth, which is not known for having as strong an alumni network as several other top schools, was penalized by this change.

The magazine also disclosed that it would use only this year’s data for its employer score, discontinuing the practice of using historical data. In 2012, for example, BW gave the year’s survey results a 50% weight, and then added the results from both 2010 and 2008, each at a weight of 25%. That methodology tended to result in less survey-to-survey variation.

Another change in the employer methodology also could impact the ranking. “Previously, we determined each school’s score by asking recruiters simply to rank up to 20 schools, from best to worst,” explained Rodkin. “Our updated survey asked employers what qualities they seek in hiring MBAs and which graduates offer those qualities, allowing us to record a clearer picture of recruiter priorities and attributes, one less biased by the effects of a school’s reputation.”

In an apparent move to increase transparency, BusinessWeek published its list of student survey questions used for the ranking. In earlier years, the magazine didn’t publicly disclose the questions out of concern that it would make it easier for schools to game the system by telling students how to vote.