The Full Jones Day Report On The Rankings Scandal At Temple Fox

Findings and Recommendations from Jones Day Investigation into Rankings Information

Provided by Fox School to U.S. News

DISCOVERY OF ERRONEOUS RANKINGS INFORMATION

In January 2018, Temple University learned that the University’s Fox School of Business (“Fox”) had submitted certain inaccurate information to U.S. News & World Report in connection with U.S. News’s 2018 survey of the best Online MBA (“OMBA”) programs. In

particular, the University learned that Fox’s survey submission to U.S. News had erroneously indicated that all new entrants to the OMBA program had provided Graduate Management Admission Test (“GMAT”) scores as part of their applications. Fox self-reported to U.S. News that this GMAT data were incorrect.

INVESTIGATION

Later in January 2018, the University retained Jones Day to conduct an investigation into this and other information reported by Fox to U.S. News. As part of the investigation, Jones Day interviewed 17 University and Fox employees, some on multiple occasions. Jones Day also collected a substantial volume of electronically stored information and hard-copy documents from University and Fox employees, and the investigative team reviewed more than 32,000 documents from this collection. Jones Day also retained a data analytics firm to review, under Jones Day’s direction, certain data submitted by Fox to U.S. News and other related data from University and Fox sources.

KEY FINDINGS

Jones Day made multiple factual findings based on the investigation, as set forth below. In short, the investigation revealed that, over the past several years, Fox provided U.S. News with inaccurate information across multiple data metrics that are part of the publication’s OMBA rankings methodology. And while Jones Day focused on information that Fox provided to U.S. News relating to the school’s OMBA program, the investigation revealed that Fox provided U.S. News with erroneous information relating to other programs as well. On certain occasions, Fox’s reporting of inaccurate information to U.S. News was done knowingly and intentionally

for the purpose of improving or maintaining Fox’s standing in the relevant U.S. News rankings.

The key findings of the investigation are as follows:

1. In each year since at least 2014, Fox reported inaccurate information to U.S. News with

respect to one or more data metrics, including: (a) the number of entrants who provided

GMAT scores, (b) the mean undergraduate GPA of entrants, (c) the number of offers of

admission extended to applicants, and (d) student-borrower indebtedness. In addition,

Fox personnel adopted questionable interpretations of survey questions.

2. In various respects, Fox leadership and other employees bore responsibility for creating

or promoting conditions that contributed to the reporting of inaccurate information to

U.S. News and/or for the misreporting itself. Pressure to improve and maintain rankings

contributed to the reporting of inaccurate information.

3. Following a change in procedure initiated by the Dean in approximately mid-2013, Fox

did not establish adequate checks and balances in the process for compiling, verifying,

and submitting information to U.S. News.

4. The employee principally responsible for preparing and submitting Fox’s responses to

ranking surveys knowingly and intentionally misreported certain information to U.S.

News and failed to correct inaccuracies with respect to other information. The

investigative record is inconclusive as to whether this employee, in knowingly and

intentionally misreporting information, was acting at the specific direction of any other

Fox personnel.

5. There were multiple opportunities for other Fox personnel to observe and/or correct

inaccuracies in information to be or that had been provided to U.S. News, but these

inaccuracies were not corrected either before or after submission.

INACCURATE DATA

The inaccurate information that Fox submitted to U.S. News in responses to OMBA

rankings surveys includes the following:

Data Metric Ranking Year(s) Inaccurate Information Reported

Number of entrants who

provided GMAT scores

2015-2018 Reported all new entrants to program

provided GMAT scores; actual figure

was significantly lower

Mean undergraduate GPA 2015-2018 Reported mean undergraduate GPA

figures inflated through use of

various methods

Number of offers of

admission

2017-2018 Underreported number of offers of

admission, thereby indicating higher

admissions selectivity and

matriculation rates

Student indebtedness 2016-2018 Reported average debt among all

graduates rather than only graduates

with debt as requested by survey

question, thereby decreasing average

debt

Due, in part, to inconsistencies in data maintained in separate Fox and University data systems, the investigation has not determined the precise figures that Fox should have reported to U.S. News with respect to certain data metrics. The investigation has also found evidence that Fox submitted inaccurate information to U.S. News in connection with other data metrics and programs. Further analysis of the nature, extent, and duration of this misreporting is recommended.

CONTRIBUTING FACTORS

1. Environment at Fox

The investigation revealed that (i) the Dean and other Fox personnel, made clear that improving or maintaining Fox’s position in rankings was a key priority; (ii) Fox had in place a concerted, rankings-focused strategy including detailed analyses of U.S. News’s rankings methodology and strategies tied to specific U.S. News data metrics, which strategy was promoted internally by the Dean and other Fox personnel; and (iii) the environment fostered by the school’s emphasis on rankings contributed to the reporting of inaccurate information to U.S. News. Moreover, the Dean’s focus on rankings, coupled with his personal management style, caused Fox personnel who interacted with the Dean on ranking-related matters to feel pressure to perform in this regard.

When Fox achieved rankings success—such as gaining or holding the top spot in the OMBA rankings—the school celebrated and openly promoted that success to internal and external constituencies. For example, Fox touted the OMBA program’s fourth straight #1 U.S.

News ranking in an email on January 22, 2018, notwithstanding that its leadership had learned more than ten days before that the survey response included inaccurate data. Praise for such achievements was also given at the individual employee level. Most

significantly, in annual performance reviews and otherwise, the employee principally responsible for rankings surveys received very favorable assessments of the employee’s rankings-related work, and was even given credit not just for compiling and organizing information for submission to rankings agencies, but also for improvements in Fox’s rankings positions.

2. Inadequate Checks and Balances

The investigation revealed that since approximately mid-2013, controls that existed within Fox were inadequate to ensure the accuracy of information reported to U.S. News. In approximately mid-2013, at the Dean’s initiative, Fox ceased use of a committee

whose members had been meeting regularly to review, and collectively approve, draft responses to rankings surveys. Thereafter, the employee principally responsible for rankings surveys had limited supervision in the collection, verification, and submission of survey data. Neither the Dean nor any other Fox personnel implemented any formal or consistent process related to the review and verification of information contained in draft survey responses prepared by the employee principally responsible for rankings surveys.

3. Misreporting of Certain Data

The investigation determined that the environmental factors noted above, along with the  lack of oversight and controls in the process for reviewing and verifying draft responses to rankings surveys, enabled instances of misreporting of rankings-related information to U.S. News.

For example, beginning with Fox’s submission for U.S. News’s 2015 OMBA rankings, the employee principally responsible for rankings surveys (i) knowingly misreported that all new program entrants had provided GMAT scores to Fox, and (ii) allegedly did so at the Dean’s direction in the presence of another employee. The Dean and the other employee deny that such

direction was given.

In addition, beginning at least as early as Fox’s survey responses submitted in 2013 for U.S. News’s 2014 Best Graduate Schools of Business rankings, the employee principally responsible for rankings surveys used various methods to inflate GPA scores of certain entrants, including increasing GPAs listed as a 1/100th value to the next highest 1/10th value (e.g., increasing 3.22 to 3.30), and then reporting the mean—or in some cases, the median—derived from these inflated values to U.S. News as part of Fox’s survey response. (The relevant U.S. News survey questions asked for the mean, rather than the median, GPA values.) Similar methods were also used to increase the mean undergraduate GPA reported to U.S. News for the 2015-2018 OMBA rankings.

Other inaccurate information that Fox reported to U.S. News with respect to the school’s OMBA program included the number of offers of admission extended to applicants and information related to student indebtedness. For at least the 2017-2018 OMBA rankings, Fox underreported the number of offers of admission such that the reported figure was only slightly

higher than the number of new entrants to the program. This resulted in inflated selectivity and

matriculation rates. For U.S. News’s 2016-2018 OMBA rankings, Fox reported the average debt

among all graduates, rather than only among those graduates with debt as requested by the

survey question, resulting in a lower average debt figure. The investigative record is

inconclusive with respect to whether this misreporting was intentional, or was instead the result

of error, incompetence, or confusion about the appropriate source of data on the part of one or

more Fox employees who assisted in compiling and considering the information prior to its

submission to U.S. News.

As to certain other information requested by U.S. News, Fox personnel adopted

questionable interpretations of survey questions, and as a result provided responses that are

likewise questionable. As part of its OMBA survey, U.S. News asks for the number of new

entrants who provided GRE scores and the mean quantitative, verbal, and analytical writing

scores (components of the GRE) among those entrants. When preparing survey responses on

this metric, the employee principally responsible for rankings surveys consistently converted

GRE scores to GMAT scores for purposes of reporting the total number of GMAT test takers,

and then reported that zero entrants had provided GRE scores. As a consequence, these Fox

survey responses indicated no GRE takers and no GRE scores.

With respect to class size, U.S. News asks for the average class size of courses that are

part of the OMBA program. For at least the 2016 through 2018 U.S. News rankings, Fox

reported the average class size on the basis of a student-to-faculty ratio, dividing the number of

students in a class by the total number of instructors and “academic coaches” assigned to the

class. With respect to technical staff, U.S. News’s OMBA survey asks for (a) the number of fulltime

and part-time technical staff the school employed to offer technical assistance to OMBA

faculty members, and (b) the number of faculty members at the institution those technical staff

are employed to assist. For at least the 2016 through 2018 rankings, Fox likely underreported the

number of faculty at the institution that the full-time technical staff support, resulting in an

inflated ratio of full-time technology support personnel to supported faculty members.

4. Missed Opportunities to Prevent Reporting of Inaccurate Data

The investigation revealed that there were numerous opp

ortunities over the last several

years for Fox personnel to prevent the reporting of inaccurate information to U.S. News or to

correct inaccurate information that had been submitted.

Certain of these opportunities arose from the distribution of draft survey responses by the

employee principally responsible for rankings surveys to other Fox personnel. In some cases, the

inaccurate data would have been clear from a thorough review of the draft survey responses but

were missed. At various times, numerous Fox personnel, including members of Fox’s leadership

team, received one or more draft responses to rankings surveys that contained inaccurate

information, but failed to either identify or correct those errors.

In other cases, inaccurate data were explicitly identified by the employee principally

responsible for rankings surveys or by Fox personnel reviewing the draft responses, but were not

corrected prior to submission. For example, during September 2017, Fox personnel raised

concerns about the accuracy of the draft survey responses for the 2018 OMBA rankings with the

employee principally responsible for rankings surveys, the supervisor of this employee, and

others at Fox. Despite these communications, the previously flagged inaccuracies were not

corrected prior to the submission of the survey responses.

U.S. News reached out to Fox on at least two occasions with questions relating to certain

data that Fox had included in a survey response. Most notably, in November 2016, U.S. News

emailed the employee principally responsible for rankings surveys and others regarding Fox’s

survey response (for the then-yet-to-be-published 2017 OMBA rankings) that all 198 of the

program’s new entrants had provided GMAT scores as part of their applications. In its inquiry,

U.S. News noted that Fox’s website indicated that it granted waivers of the GMAT requirement

and questioned the accuracy of the reported GMAT information. The employee responded to

this email without substantively addressing the accuracy of this GMAT information, and the

inaccurate information was never corrected.

RECOMMENDED REMEDIAL AND CORRECTIVE ACTION

Based on the findings of the investigation, Jones Day has recommended that the

University undertake certain remedial and corrective actions:

1. Assess relative responsibility of relevant personnel, and determine and take

appropriate personnel action.

2. Communicate identified instances of misreporting to U.S. News and other

constituents.

Page 6 of 7

3. Efficiently and effectively determine nature and extent of any other misreporting

by Fox; communicate any such misreporting as appropriate.

4. Institute policy specific to solicitation; compilation; verification; cross-checking;

and submission / reporting of data to external constituencies to address key

aspects of data reporting process, including supervision and oversight.

5. Institute explicit policy addressing mandatory reporting of suspected / potential

inaccurate data reporting.

6. Provide training for employees about best practices in data collection,

verification, and reporting to maximize precision and accuracy.

7. Develop and implement revised rankings process for soliciting, compiling,

verifying, cross-checking, and submitting rankings data consisting of key

elements, including—

a. Form group responsible for reviewing rankings submissions that will:

(1) resolve interpretive questions through consultation with

rankings agencies or otherwise;

(2) memorialize resolution of interpretive issues; and

(3) authorize the collection and reporting of data consistent with

resolution of interpretive decisions, and with all applicable

policies;

b. Identify personnel and sources from which / whom to solicit each

category of information / data requested;

c. Develop and implement reconciliation of existing databases;

d. Identify and specifically designate Fox / University employee(s)

responsible for soliciting, compiling, verifying, cross-checking, and

submitting information / data requested;

e. Identify and designate supervisor(s) of employee(s) noted in (d),

above.

f. Create repository for data compilation with access rights limited to,

e.g., group responsible for reviewing rankings submissions, the head

of Institutional Research and Assessment (“IRA”), and select

representative(s) from the Department of Internal Audits;

g. Implement process for saving and archiving supporting data,

including, among other things, a “snapshot” of relevant source data as

it existed pre-submission, the date on which data were submitted to

repository, identity of submitter, etc.;

h. Implement process for verifying / approving submissions, including a

formal review by the group designated as responsible for reviewing

rankings submissions;

i. Prepare a process map setting forth the processes outlined above; and

j. Develop and administer training program for all relevant personnel.

8. Retain all supporting documentation (e.g., Cognos reports, exports from other

databases, and other data “snapshots”), working papers (e.g., memorialization of

decision-making on interpretive questions, records reflecting data compilation and

associated calculations / analysis, etc.), and final submissions for each ranking

survey for no less than seven (7) years.

9. Periodically review compliance, via Internal Audits.

10. Promote and reinforce institutional commitment to key values in onboarding

orientations; training modules specific to data collection, verification, and

reporting; and in employee handbook.

11. Promote and reinforce key aspects