Findings and Recommendations from Jones Day Investigation into Rankings Information
Provided by Fox School to U.S. News
DISCOVERY OF ERRONEOUS RANKINGS INFORMATION
In January 2018, Temple University learned that the University’s Fox School of Business (“Fox”) had submitted certain inaccurate information to U.S. News & World Report in connection with U.S. News’s 2018 survey of the best Online MBA (“OMBA”) programs. In
particular, the University learned that Fox’s survey submission to U.S. News had erroneously indicated that all new entrants to the OMBA program had provided Graduate Management Admission Test (“GMAT”) scores as part of their applications. Fox self-reported to U.S. News that this GMAT data were incorrect.
Later in January 2018, the University retained Jones Day to conduct an investigation into this and other information reported by Fox to U.S. News. As part of the investigation, Jones Day interviewed 17 University and Fox employees, some on multiple occasions. Jones Day also collected a substantial volume of electronically stored information and hard-copy documents from University and Fox employees, and the investigative team reviewed more than 32,000 documents from this collection. Jones Day also retained a data analytics firm to review, under Jones Day’s direction, certain data submitted by Fox to U.S. News and other related data from University and Fox sources.
Jones Day made multiple factual findings based on the investigation, as set forth below. In short, the investigation revealed that, over the past several years, Fox provided U.S. News with inaccurate information across multiple data metrics that are part of the publication’s OMBA rankings methodology. And while Jones Day focused on information that Fox provided to U.S. News relating to the school’s OMBA program, the investigation revealed that Fox provided U.S. News with erroneous information relating to other programs as well. On certain occasions, Fox’s reporting of inaccurate information to U.S. News was done knowingly and intentionally
for the purpose of improving or maintaining Fox’s standing in the relevant U.S. News rankings.
The key findings of the investigation are as follows:
1. In each year since at least 2014, Fox reported inaccurate information to U.S. News with
respect to one or more data metrics, including: (a) the number of entrants who provided
GMAT scores, (b) the mean undergraduate GPA of entrants, (c) the number of offers of
admission extended to applicants, and (d) student-borrower indebtedness. In addition,
Fox personnel adopted questionable interpretations of survey questions.
2. In various respects, Fox leadership and other employees bore responsibility for creating
or promoting conditions that contributed to the reporting of inaccurate information to
U.S. News and/or for the misreporting itself. Pressure to improve and maintain rankings
contributed to the reporting of inaccurate information.
3. Following a change in procedure initiated by the Dean in approximately mid-2013, Fox
did not establish adequate checks and balances in the process for compiling, verifying,
and submitting information to U.S. News.
4. The employee principally responsible for preparing and submitting Fox’s responses to
ranking surveys knowingly and intentionally misreported certain information to U.S.
News and failed to correct inaccuracies with respect to other information. The
investigative record is inconclusive as to whether this employee, in knowingly and
intentionally misreporting information, was acting at the specific direction of any other
5. There were multiple opportunities for other Fox personnel to observe and/or correct
inaccuracies in information to be or that had been provided to U.S. News, but these
inaccuracies were not corrected either before or after submission.
The inaccurate information that Fox submitted to U.S. News in responses to OMBA
rankings surveys includes the following:
Data Metric Ranking Year(s) Inaccurate Information Reported
Number of entrants who
provided GMAT scores
2015-2018 Reported all new entrants to program
provided GMAT scores; actual figure
was significantly lower
Mean undergraduate GPA 2015-2018 Reported mean undergraduate GPA
figures inflated through use of
Number of offers of
2017-2018 Underreported number of offers of
admission, thereby indicating higher
admissions selectivity and
Student indebtedness 2016-2018 Reported average debt among all
graduates rather than only graduates
with debt as requested by survey
question, thereby decreasing average
Due, in part, to inconsistencies in data maintained in separate Fox and University data systems, the investigation has not determined the precise figures that Fox should have reported to U.S. News with respect to certain data metrics. The investigation has also found evidence that Fox submitted inaccurate information to U.S. News in connection with other data metrics and programs. Further analysis of the nature, extent, and duration of this misreporting is recommended.
1. Environment at Fox
The investigation revealed that (i) the Dean and other Fox personnel, made clear that improving or maintaining Fox’s position in rankings was a key priority; (ii) Fox had in place a concerted, rankings-focused strategy including detailed analyses of U.S. News’s rankings methodology and strategies tied to specific U.S. News data metrics, which strategy was promoted internally by the Dean and other Fox personnel; and (iii) the environment fostered by the school’s emphasis on rankings contributed to the reporting of inaccurate information to U.S. News. Moreover, the Dean’s focus on rankings, coupled with his personal management style, caused Fox personnel who interacted with the Dean on ranking-related matters to feel pressure to perform in this regard.
When Fox achieved rankings success—such as gaining or holding the top spot in the OMBA rankings—the school celebrated and openly promoted that success to internal and external constituencies. For example, Fox touted the OMBA program’s fourth straight #1 U.S.
News ranking in an email on January 22, 2018, notwithstanding that its leadership had learned more than ten days before that the survey response included inaccurate data. Praise for such achievements was also given at the individual employee level. Most
significantly, in annual performance reviews and otherwise, the employee principally responsible for rankings surveys received very favorable assessments of the employee’s rankings-related work, and was even given credit not just for compiling and organizing information for submission to rankings agencies, but also for improvements in Fox’s rankings positions.
2. Inadequate Checks and Balances
The investigation revealed that since approximately mid-2013, controls that existed within Fox were inadequate to ensure the accuracy of information reported to U.S. News. In approximately mid-2013, at the Dean’s initiative, Fox ceased use of a committee
whose members had been meeting regularly to review, and collectively approve, draft responses to rankings surveys. Thereafter, the employee principally responsible for rankings surveys had limited supervision in the collection, verification, and submission of survey data. Neither the Dean nor any other Fox personnel implemented any formal or consistent process related to the review and verification of information contained in draft survey responses prepared by the employee principally responsible for rankings surveys.
3. Misreporting of Certain Data
The investigation determined that the environmental factors noted above, along with the lack of oversight and controls in the process for reviewing and verifying draft responses to rankings surveys, enabled instances of misreporting of rankings-related information to U.S. News.
For example, beginning with Fox’s submission for U.S. News’s 2015 OMBA rankings, the employee principally responsible for rankings surveys (i) knowingly misreported that all new program entrants had provided GMAT scores to Fox, and (ii) allegedly did so at the Dean’s direction in the presence of another employee. The Dean and the other employee deny that such
direction was given.
In addition, beginning at least as early as Fox’s survey responses submitted in 2013 for U.S. News’s 2014 Best Graduate Schools of Business rankings, the employee principally responsible for rankings surveys used various methods to inflate GPA scores of certain entrants, including increasing GPAs listed as a 1/100th value to the next highest 1/10th value (e.g., increasing 3.22 to 3.30), and then reporting the mean—or in some cases, the median—derived from these inflated values to U.S. News as part of Fox’s survey response. (The relevant U.S. News survey questions asked for the mean, rather than the median, GPA values.) Similar methods were also used to increase the mean undergraduate GPA reported to U.S. News for the 2015-2018 OMBA rankings.
Other inaccurate information that Fox reported to U.S. News with respect to the school’s OMBA program included the number of offers of admission extended to applicants and information related to student indebtedness. For at least the 2017-2018 OMBA rankings, Fox underreported the number of offers of admission such that the reported figure was only slightly
higher than the number of new entrants to the program. This resulted in inflated selectivity and
matriculation rates. For U.S. News’s 2016-2018 OMBA rankings, Fox reported the average debt
among all graduates, rather than only among those graduates with debt as requested by the
survey question, resulting in a lower average debt figure. The investigative record is
inconclusive with respect to whether this misreporting was intentional, or was instead the result
of error, incompetence, or confusion about the appropriate source of data on the part of one or
more Fox employees who assisted in compiling and considering the information prior to its
submission to U.S. News.
As to certain other information requested by U.S. News, Fox personnel adopted
questionable interpretations of survey questions, and as a result provided responses that are
likewise questionable. As part of its OMBA survey, U.S. News asks for the number of new
entrants who provided GRE scores and the mean quantitative, verbal, and analytical writing
scores (components of the GRE) among those entrants. When preparing survey responses on
this metric, the employee principally responsible for rankings surveys consistently converted
GRE scores to GMAT scores for purposes of reporting the total number of GMAT test takers,
and then reported that zero entrants had provided GRE scores. As a consequence, these Fox
survey responses indicated no GRE takers and no GRE scores.
With respect to class size, U.S. News asks for the average class size of courses that are
part of the OMBA program. For at least the 2016 through 2018 U.S. News rankings, Fox
reported the average class size on the basis of a student-to-faculty ratio, dividing the number of
students in a class by the total number of instructors and “academic coaches” assigned to the
class. With respect to technical staff, U.S. News’s OMBA survey asks for (a) the number of fulltime
and part-time technical staff the school employed to offer technical assistance to OMBA
faculty members, and (b) the number of faculty members at the institution those technical staff
are employed to assist. For at least the 2016 through 2018 rankings, Fox likely underreported the
number of faculty at the institution that the full-time technical staff support, resulting in an
inflated ratio of full-time technology support personnel to supported faculty members.
4. Missed Opportunities to Prevent Reporting of Inaccurate Data
The investigation revealed that there were numerous opp
ortunities over the last several
years for Fox personnel to prevent the reporting of inaccurate information to U.S. News or to
correct inaccurate information that had been submitted.
Certain of these opportunities arose from the distribution of draft survey responses by the
employee principally responsible for rankings surveys to other Fox personnel. In some cases, the
inaccurate data would have been clear from a thorough review of the draft survey responses but
were missed. At various times, numerous Fox personnel, including members of Fox’s leadership
team, received one or more draft responses to rankings surveys that contained inaccurate
information, but failed to either identify or correct those errors.
In other cases, inaccurate data were explicitly identified by the employee principally
responsible for rankings surveys or by Fox personnel reviewing the draft responses, but were not
corrected prior to submission. For example, during September 2017, Fox personnel raised
concerns about the accuracy of the draft survey responses for the 2018 OMBA rankings with the
employee principally responsible for rankings surveys, the supervisor of this employee, and
others at Fox. Despite these communications, the previously flagged inaccuracies were not
corrected prior to the submission of the survey responses.
U.S. News reached out to Fox on at least two occasions with questions relating to certain
data that Fox had included in a survey response. Most notably, in November 2016, U.S. News
emailed the employee principally responsible for rankings surveys and others regarding Fox’s
survey response (for the then-yet-to-be-published 2017 OMBA rankings) that all 198 of the
program’s new entrants had provided GMAT scores as part of their applications. In its inquiry,
U.S. News noted that Fox’s website indicated that it granted waivers of the GMAT requirement
and questioned the accuracy of the reported GMAT information. The employee responded to
this email without substantively addressing the accuracy of this GMAT information, and the
inaccurate information was never corrected.
RECOMMENDED REMEDIAL AND CORRECTIVE ACTION
Based on the findings of the investigation, Jones Day has recommended that the
University undertake certain remedial and corrective actions:
1. Assess relative responsibility of relevant personnel, and determine and take
appropriate personnel action.
2. Communicate identified instances of misreporting to U.S. News and other
Page 6 of 7
3. Efficiently and effectively determine nature and extent of any other misreporting
by Fox; communicate any such misreporting as appropriate.
4. Institute policy specific to solicitation; compilation; verification; cross-checking;
and submission / reporting of data to external constituencies to address key
aspects of data reporting process, including supervision and oversight.
5. Institute explicit policy addressing mandatory reporting of suspected / potential
inaccurate data reporting.
6. Provide training for employees about best practices in data collection,
verification, and reporting to maximize precision and accuracy.
7. Develop and implement revised rankings process for soliciting, compiling,
verifying, cross-checking, and submitting rankings data consisting of key
a. Form group responsible for reviewing rankings submissions that will:
(1) resolve interpretive questions through consultation with
rankings agencies or otherwise;
(2) memorialize resolution of interpretive issues; and
(3) authorize the collection and reporting of data consistent with
resolution of interpretive decisions, and with all applicable
b. Identify personnel and sources from which / whom to solicit each
category of information / data requested;
c. Develop and implement reconciliation of existing databases;
d. Identify and specifically designate Fox / University employee(s)
responsible for soliciting, compiling, verifying, cross-checking, and
submitting information / data requested;
e. Identify and designate supervisor(s) of employee(s) noted in (d),
f. Create repository for data compilation with access rights limited to,
e.g., group responsible for reviewing rankings submissions, the head
of Institutional Research and Assessment (“IRA”), and select
representative(s) from the Department of Internal Audits;
g. Implement process for saving and archiving supporting data,
including, among other things, a “snapshot” of relevant source data as
it existed pre-submission, the date on which data were submitted to
repository, identity of submitter, etc.;
h. Implement process for verifying / approving submissions, including a
formal review by the group designated as responsible for reviewing
i. Prepare a process map setting forth the processes outlined above; and
j. Develop and administer training program for all relevant personnel.
8. Retain all supporting documentation (e.g., Cognos reports, exports from other
databases, and other data “snapshots”), working papers (e.g., memorialization of
decision-making on interpretive questions, records reflecting data compilation and
associated calculations / analysis, etc.), and final submissions for each ranking
survey for no less than seven (7) years.
9. Periodically review compliance, via Internal Audits.
10. Promote and reinforce institutional commitment to key values in onboarding
orientations; training modules specific to data collection, verification, and
reporting; and in employee handbook.
11. Promote and reinforce key aspects