AI Detection Tools On MBA Essays: False Positives? Fakes?

Blame Adam Markus. The veteran MBA admissions consultant urged us to look into the accuracy of AI detection systems being used on MBA application essays. He had a good reason for his request.

“It is my guess based on anecdotal evidence that Grammarly and other such tools might be generating a lot of false positive results,” wrote Markus who has been doing MBA admissions consulting since 2001. “Given the way natural language AI systems work (and also don’t work), I think there is a real good story here.”

So we gathered a dozen MBA essays written by successful candidates to Harvard Business School and Stanford’s Graduate School of Business, all of them published in full on Poets&Quants (see below), and put them through a leading AI section service, Originality.ai, that claims a false positive rate of less than 3%. Those essays were written by both domestic and international applicants, including those for whom English is a second language.

TESTING AI DETECTION ON SUCCESSFUL MBA ESSAYS AT HARVARD & STANFORD

Originality.ai’s score on all 12 actual MBA essays as well as the two generated by ChatGPT4

According to Originality.ai, “False positives are painful when they occur, and we are working hard to continually improve and release even better models that drive down our false positive rate further. A false positive occurs when an AI detector thinks something was AI-generated when in fact it was written by a human.”

The good news, based on your perspective of course, is that AI was not detected in any of the essays, all of which were written and submitted before ChatGPT was available. In every case, Originality.ai declared each of the 12 essays “likely original.” The confidence level

The biggest surprise, however, was that the tool failed to identify two essays, one for Harvard and one for Stanford, we created using ChatGPT-4. Sure, Originality.ai concedes that its scoring is not perfect. “The tool does produce incorrect predictions,” the software’s website admits. “Incorrectly predicting content that was human-generated as AI generated happens approximately 1% of the time.”

AI DETECTION ON FAKES, NOT SO GOOD

Yet, it took very little effort to fool the AI detection system, particularly after writing a prompt with information that would obviously yield a more personalized result.

For Stanford’s iconic What Matters Most To You And Why prompt, here’s what we asked ChatGPT-4: “Write a 650-word essay for admission to Stanford’s MBA program question: What Matters Most To You And Why? My father is my inspiration for a career in healthcare. A heart surgeon, he has used his scalp on hundreds of patients, saving lives on a weekly basis. In fact, over a 25-year career as a surgeon, he has never lost a patient on the operating room table. While I do not have his medical skills, I want to use the business skills I gained in an MBA to insure that healthcare is delivered fairly and efficiently so that I, too, can save lives and diminish pain.”

Here’s what ChatGPT spat out in seconds:

What matters most to me is making healthcare accessible, efficient, and equitable for all. This conviction stems from my deep admiration for my father, who has dedicated his life to saving others as a heart surgeon. For over 25 years, he has wielded his scalpel with incredible skill, saving lives on a weekly basis. What’s truly remarkable is that, despite the complexity of his work, he has never lost a patient on the operating table. He is my role model, not only because of his technical expertise but also because of his commitment to patient care and human dignity. While I do not possess the medical skills to follow directly in his footsteps, his career has inspired me to pursue a different but complementary path—one that focuses on improving the systems and structures that deliver healthcare.

Growing up, I saw firsthand the impact my father had on his patients and their families. People would thank him for giving them more time with their loved ones. His patients came from all walks of life, from wealthy executives to those barely scraping by, and he treated them all with the same level of care and respect. Witnessing this instilled in me a profound belief in the right to quality healthcare for everyone, regardless of their economic circumstances. However, I also observed the challenges he faced working within a healthcare system that often seemed more focused on profit margins and bureaucracy than on patients.

As I pursued my education and began to develop my own professional ambitions, I realized that while I may not be able to heal patients with a scalpel, I could play a crucial role in healing the system that serves them. My interest in healthcare management began to take shape as I studied the operational inefficiencies and inequities in access that plague healthcare systems worldwide. I learned that millions of people—both in developed and developing countries—are denied proper healthcare due to mismanagement, rising costs, and systemic inequalities. This realization was both disheartening and motivating. I knew that with the right skills and platform, I could contribute to solving these complex problems.

This is where the MBA at Stanford comes in. I believe that the education and network I would gain at Stanford would equip me with the knowledge and tools necessary to address the most pressing issues in healthcare management. I am particularly interested in understanding how technology, policy, and innovative business models can intersect to create more efficient and accessible healthcare systems. For example, telemedicine and digital health records hold enormous potential to reduce costs and improve patient outcomes, but they must be implemented in ways that are inclusive and scalable. Stanford’s focus on innovation and its proximity to Silicon Valley make it the ideal place for me to explore these possibilities.

Moreover, Stanford’s emphasis on leadership and personal development resonates deeply with me. My father has always taught me that a great surgeon isn’t just someone who knows how to perform a procedure but someone who cares about the people on the operating table. Similarly, I believe that a great healthcare leader isn’t just someone who can analyze data and manage resources but someone who understands the human lives behind those numbers. I want to develop into the kind of leader who can balance business acumen with empathy, efficiency with equity, and innovation with compassion.

In addition to acquiring technical skills, I am excited about the opportunity to learn from and collaborate with my peers at Stanford. The diversity of experiences and perspectives within the MBA program would challenge me to think critically about the role of business in healthcare and beyond. I am eager to be part of a community where intellectual curiosity is encouraged, and where I can engage in meaningful discussions about the ethical dimensions of healthcare delivery.

Ultimately, what matters most to me is the ability to use my skills to make a tangible difference in the lives of others. My father has saved countless lives with his hands, and while my path is different, I hope to save lives through the power of better systems and better leadership. Stanford’s MBA program offers me the best opportunity to develop into the kind of leader who can help shape the future of healthcare—one that is fairer, more efficient, and more humane. I am ready to take on that challenge, and I look forward to doing so with the support and guidance of the Stanford community.

Originality.ai gave the AI-generated essay the same score as the original essays, noting that it was “likely original.”

Then, we tried one of Harvard’s new essays questions, asking ChatGPT to write a 300-word essay that answered this prompt: Please reflect on how your experiences have influenced your career choices and aspirations and the impact you will have on businesses, organizations, and communities you plan to serve.”

We fed it the exact same additional info and it came up with the following:

My father, a heart surgeon with over 25 years of experience, has been my greatest inspiration. His skillful hands have saved countless lives, and his unwavering commitment to his patients has always amazed me. Over the years, I have witnessed his dedication to healing, but I have also seen the challenges he faces within an often inefficient and inequitable healthcare system. While I may not have his medical expertise, his career has profoundly shaped my own aspirations. I want to ensure that healthcare is delivered in a way that maximizes its impact on patients and communities.

My experiences observing my father, coupled with my academic and professional journey, have fueled my passion for improving healthcare systems through effective management and innovative solutions. I have come to realize that many of the barriers to quality care are not medical but organizational. Inefficient processes, resource misallocation, and limited access to healthcare are issues that can be addressed through leadership and strategic decision-making.

An MBA from Harvard will equip me with the business acumen and leadership skills necessary to drive meaningful change. I am eager to explore how technology, data, and policy can intersect to create more efficient, patient-centered healthcare systems. Beyond technical knowledge, I am drawn to Harvard’s focus on leadership development, which will help me become a leader capable of balancing efficiency with compassion.

I aspire to make a lasting impact by creating healthcare systems that deliver care more equitably and efficiently. I believe that the skills and network I will gain at Harvard will empower me to make healthcare more accessible, improve patient outcomes, and ultimately, save lives—just as my father has done throughout his career.

THE BOTTOM LINE ON AI DETECTION TOOLS

Same result. It completely fooled Originality.ai. In both examples, no less, the AI-generated copy was the first draft. It didn’t even require additional prompting to get a better response.

We’re not sure if this little experiment would satisfy Markus’ skepticism over AI detection systems. But it may well prove something else: False positives in AI detection software are most likely less common than a system’s ability to recognize a fake.