How would ChatGPT rank business schools? And how accurate would a list of full-time MBA programs actually be from the AI chatbot?
To put it to the test, we asked the former question to get at the latter query. The result?
Naming the top MBA programs was too complex a task for ChatGPT-4. Though the artificial intelligence program that is all the talk of the technosphere generated a list of reputable programs, the rankings were contextually vague and inconsistently founded upon further interrogation. Some information the AI chatbot identified as its ranking evidence was plainly false. ChatGPT also changed ranking placements in different answers with no real sense of justification, but one always remained first. By and large, ChatGPT’s ranking is more a reflection of a business school’s longstanding reputation, rather than a revealing list that uncovers MBA programs that are more likely to slip through the cracks.
ChatGPT named Harvard Business School the best program in the world. But while HBS is undoubtedly a world-class school, it has been languishing in the rankings lately, generally situated somewhere in the top five but not No. 1 in any reputable list since 2020. Harvard has ranked fifth in Poet&Quants’ aggregate ranking the last two years; in 2023, The Financial Times named HBS’ MBA program the third best, while U.S. News & World Report ranked it fifth in 2022; the latter’s 2023 ranking is due in mid-April.
CHATGPT SELECTS HBS FOR ITS PRESTIGE
“Based on various sources and rankings, HBS is often ranked as the top MBA program,” answers ChatGPT-4, Open AI’s newly-released and fine-tuned large language model accessible only through a $20 monthly subscription.
From 2021-17, HBS ranked the #1 school five times among six different publications. Rankings were from P&Q, U.S. News, Forbes, Bloomberg BusinessWeek, and The Economist.
HBS most-recently ranked #1 by the FT in 2020, it tied with Wharton for #1 a couple of years earlier in P&Qs’ 2018 rankings. It’s worth noting ChatGPT’s #2 B-School, the Stanford Graduate School of Business, actually ranked #1 twice as many times as HBS within the same years and publications.
To why HBS was chosen as one, ChatGPT raddled off a list of reasons that seemed sensible, its brand, prestige, its strong alumni network. But no real data was offered to justify its parameters. When asked to provide further evidence, ChatGPT responded: “According to the Financial Times Global MBA Ranking 2021, Harvard Business School is ranked as the #1 MBA program in the world for the seventh consecutive year, based on factors such as career progress, diversity, and research.”
But that year, HBS along with Stanford and Wharton wasn’t even ranked by the FT because the B-Schools boycotted the rankings during the pandemic. Maybe ChatGPT isn't capable right now of selecting the top program for a particular year, however, it did compose a pretty impressive list of highly-regarded schools that wouldn't be too surprising to find at the top of any ranking.
HOW IT GOT RESULTS
The overall user experience when it comes to rankings just wasn’t there. Data, context and other visuals were missing, the rankings changed several times, and later upon further questioning, ChatGPT revealed a short disclaimer, which to any user, means everything.
According to the ChatGPT, it used multiple sources from the internet that involved synthesizing different rankings, however, it could not provide the exact number of sources used.
Also, the AI bot did not have access to the latest ranking data beyond its knowledge cutoff date of September 2021, though, at one point, it curiously named the FT’s 2022 rankings somehow as a source. Some lists offered by the bot ranked Wharton one place above INSEAD, for reasons it mentioned similar to HBS’s placement, such as prestige, selectivity, strong curriculum and faculty. In other lists, HEC Paris made the list of the Top Ten while Haas was excluded.
Of course, the more specific info you provide the bot, the more likely you'll come up with at least some answer. Asked to name the best MBA programs for a student-centered experience, ChatGPT named the following five schools in this order: Harvard, Stanford, Kellogg, Yale, and Dartmouth Tuck. Of Tuck, the chatbot reported that the school's program is "known for its tight-knit community, with a small class size and a focus on teamwork and collaboration."
It could only list five schools and missed UVA Darden, Michigan Ross, and Duke Fuqua, schools that are known for their devotion to a stellar student experience.
INTRICACIES AROUND THE RANKINGS
Along with inaccuracy, the whole process lacked one very important element. That of a human to spell out the context behind schools’ performances, ultimately, making sense of all the data for the reader.
Open AI’s co-founder and CEO Sam Altman tweeted that the newest version is “still flawed, still limited, and still seems more impressive on first use than it does after you spend more time with it.” That in part seems true. Before even ranking schools, ChatGPT concluded in an accurate fashion how rankings vary depending on the source. It confidently expressed the importance of doing wide-ranging research and considering a variety of factors beyond rankings, including that of one’s personal “best fit” determined by career goals and personal circumstances. And in terms of conversing naturally, the AI model did well.
The purpose of rankings, as expressed by Poets&Quants founder John Byrne, who created the first ranking in 1988 as editor of Businessweek, is to measure the quality of a program. In ranking the best online MBAs, P&Qs frame its methodology according to three attributes, all weighing equally in the rankings: admissions standards, academic experience, and career outcomes. It important to note the business education landscape does change constantly, which is why having the most updated information is important.
In attempting to implicate who is best, rankings also bestow status and add credibility. For MBA candidates new to the scene, rankings are simply a starting place to guide them on a journey of filling out various applications. This is where troves of data come in handy, i.e schools' average test scores, the composition of the latest incoming class, and career stats.
WHAT CHATGPT CAN BE USED FOR
No one is claiming ChatGPT-4 is perfect. Altman said in an interview with Kara Swisher that “there are plenty of things it's still bad at” but in the jump from model 3 to 3.5 to 4, he’s proud of it. He described ChatGPT as having “flashes of brilliance.”
In a test to see how well it wrote an MBA admission essay, its initial responses appeared somewhat forced. Then after entering bits of personalized information, the Emory University’s Goizueta Business School admissions director admitted its answer read surprising well. Also apparently, according to Altman, the flagging errors feature in ChatGPT is the exact kind of user feedback it needs. After hours of back and forth when asked finally if ChatGPT has the capabilities of ranking the top MBA programs, this is what it said:
"As an AI language model, ChatGPT does not have the capability to rank MBA programs on its own. However, it can analyze data and trends from various sources and provide insights that can help prospective MBA candidates make informed decisions about their education and career paths. ChatGPT can provide information such as average alumni salary, employment rate, faculty research output, and other relevant data points that can be used to compare MBA programs."
Comments or questions about this article? Email us.