The Full Jones Day Report On The Rankings Scandal At Temple Fox by: John A. Byrne on July 10, 2018 | 1,534 Views July 10, 2018 Copy Link Share on Facebook Share on Twitter Email Share on LinkedIn Share on WhatsApp Share on Reddit Temple University’s Fox School of Business in Philadelphia Findings and Recommendations from Jones Day Investigation into Rankings Information Provided by Fox School to U.S. News DISCOVERY OF ERRONEOUS RANKINGS INFORMATION In January 2018, Temple University learned that the University’s Fox School of Business (“Fox”) had submitted certain inaccurate information to U.S. News & World Report in connection with U.S. News’s 2018 survey of the best Online MBA (“OMBA”) programs. In particular, the University learned that Fox’s survey submission to U.S. News had erroneously indicated that all new entrants to the OMBA program had provided Graduate Management Admission Test (“GMAT”) scores as part of their applications. Fox self-reported to U.S. News that this GMAT data were incorrect. INVESTIGATION Later in January 2018, the University retained Jones Day to conduct an investigation into this and other information reported by Fox to U.S. News. As part of the investigation, Jones Day interviewed 17 University and Fox employees, some on multiple occasions. Jones Day also collected a substantial volume of electronically stored information and hard-copy documents from University and Fox employees, and the investigative team reviewed more than 32,000 documents from this collection. Jones Day also retained a data analytics firm to review, under Jones Day’s direction, certain data submitted by Fox to U.S. News and other related data from University and Fox sources. KEY FINDINGS Jones Day made multiple factual findings based on the investigation, as set forth below. In short, the investigation revealed that, over the past several years, Fox provided U.S. News with inaccurate information across multiple data metrics that are part of the publication’s OMBA rankings methodology. And while Jones Day focused on information that Fox provided to U.S. News relating to the school’s OMBA program, the investigation revealed that Fox provided U.S. News with erroneous information relating to other programs as well. On certain occasions, Fox’s reporting of inaccurate information to U.S. News was done knowingly and intentionally for the purpose of improving or maintaining Fox’s standing in the relevant U.S. News rankings. The key findings of the investigation are as follows: 1. In each year since at least 2014, Fox reported inaccurate information to U.S. News with respect to one or more data metrics, including: (a) the number of entrants who provided GMAT scores, (b) the mean undergraduate GPA of entrants, (c) the number of offers of admission extended to applicants, and (d) student-borrower indebtedness. In addition, Fox personnel adopted questionable interpretations of survey questions. 2. In various respects, Fox leadership and other employees bore responsibility for creating or promoting conditions that contributed to the reporting of inaccurate information to U.S. News and/or for the misreporting itself. Pressure to improve and maintain rankings contributed to the reporting of inaccurate information. 3. Following a change in procedure initiated by the Dean in approximately mid-2013, Fox did not establish adequate checks and balances in the process for compiling, verifying, and submitting information to U.S. News. 4. The employee principally responsible for preparing and submitting Fox’s responses to ranking surveys knowingly and intentionally misreported certain information to U.S. News and failed to correct inaccuracies with respect to other information. The investigative record is inconclusive as to whether this employee, in knowingly and intentionally misreporting information, was acting at the specific direction of any other Fox personnel. 5. There were multiple opportunities for other Fox personnel to observe and/or correct inaccuracies in information to be or that had been provided to U.S. News, but these inaccuracies were not corrected either before or after submission. INACCURATE DATA The inaccurate information that Fox submitted to U.S. News in responses to OMBA rankings surveys includes the following: Data Metric Ranking Year(s) Inaccurate Information Reported Number of entrants who provided GMAT scores 2015-2018 Reported all new entrants to program provided GMAT scores; actual figure was significantly lower Mean undergraduate GPA 2015-2018 Reported mean undergraduate GPA figures inflated through use of various methods Number of offers of admission 2017-2018 Underreported number of offers of admission, thereby indicating higher admissions selectivity and matriculation rates Student indebtedness 2016-2018 Reported average debt among all graduates rather than only graduates with debt as requested by survey question, thereby decreasing average debt Due, in part, to inconsistencies in data maintained in separate Fox and University data systems, the investigation has not determined the precise figures that Fox should have reported to U.S. News with respect to certain data metrics. The investigation has also found evidence that Fox submitted inaccurate information to U.S. News in connection with other data metrics and programs. Further analysis of the nature, extent, and duration of this misreporting is recommended. CONTRIBUTING FACTORS 1. Environment at Fox The investigation revealed that (i) the Dean and other Fox personnel, made clear that improving or maintaining Fox’s position in rankings was a key priority; (ii) Fox had in place a concerted, rankings-focused strategy including detailed analyses of U.S. News’s rankings methodology and strategies tied to specific U.S. News data metrics, which strategy was promoted internally by the Dean and other Fox personnel; and (iii) the environment fostered by the school’s emphasis on rankings contributed to the reporting of inaccurate information to U.S. News. Moreover, the Dean’s focus on rankings, coupled with his personal management style, caused Fox personnel who interacted with the Dean on ranking-related matters to feel pressure to perform in this regard. When Fox achieved rankings success—such as gaining or holding the top spot in the OMBA rankings—the school celebrated and openly promoted that success to internal and external constituencies. For example, Fox touted the OMBA program’s fourth straight #1 U.S. News ranking in an email on January 22, 2018, notwithstanding that its leadership had learned more than ten days before that the survey response included inaccurate data. Praise for such achievements was also given at the individual employee level. Most significantly, in annual performance reviews and otherwise, the employee principally responsible for rankings surveys received very favorable assessments of the employee’s rankings-related work, and was even given credit not just for compiling and organizing information for submission to rankings agencies, but also for improvements in Fox’s rankings positions. 2. Inadequate Checks and Balances The investigation revealed that since approximately mid-2013, controls that existed within Fox were inadequate to ensure the accuracy of information reported to U.S. News. In approximately mid-2013, at the Dean’s initiative, Fox ceased use of a committee whose members had been meeting regularly to review, and collectively approve, draft responses to rankings surveys. Thereafter, the employee principally responsible for rankings surveys had limited supervision in the collection, verification, and submission of survey data. Neither the Dean nor any other Fox personnel implemented any formal or consistent process related to the review and verification of information contained in draft survey responses prepared by the employee principally responsible for rankings surveys. 3. Misreporting of Certain Data The investigation determined that the environmental factors noted above, along with the lack of oversight and controls in the process for reviewing and verifying draft responses to rankings surveys, enabled instances of misreporting of rankings-related information to U.S. News. For example, beginning with Fox’s submission for U.S. News’s 2015 OMBA rankings, the employee principally responsible for rankings surveys (i) knowingly misreported that all new program entrants had provided GMAT scores to Fox, and (ii) allegedly did so at the Dean’s direction in the presence of another employee. The Dean and the other employee deny that such direction was given. In addition, beginning at least as early as Fox’s survey responses submitted in 2013 for U.S. News’s 2014 Best Graduate Schools of Business rankings, the employee principally responsible for rankings surveys used various methods to inflate GPA scores of certain entrants, including increasing GPAs listed as a 1/100th value to the next highest 1/10th value (e.g., increasing 3.22 to 3.30), and then reporting the mean—or in some cases, the median—derived from these inflated values to U.S. News as part of Fox’s survey response. (The relevant U.S. News survey questions asked for the mean, rather than the median, GPA values.) Similar methods were also used to increase the mean undergraduate GPA reported to U.S. News for the 2015-2018 OMBA rankings. Other inaccurate information that Fox reported to U.S. News with respect to the school’s OMBA program included the number of offers of admission extended to applicants and information related to student indebtedness. For at least the 2017-2018 OMBA rankings, Fox underreported the number of offers of admission such that the reported figure was only slightly higher than the number of new entrants to the program. This resulted in inflated selectivity and matriculation rates. For U.S. News’s 2016-2018 OMBA rankings, Fox reported the average debt among all graduates, rather than only among those graduates with debt as requested by the survey question, resulting in a lower average debt figure. The investigative record is inconclusive with respect to whether this misreporting was intentional, or was instead the result of error, incompetence, or confusion about the appropriate source of data on the part of one or more Fox employees who assisted in compiling and considering the information prior to its submission to U.S. News. As to certain other information requested by U.S. News, Fox personnel adopted questionable interpretations of survey questions, and as a result provided responses that are likewise questionable. As part of its OMBA survey, U.S. News asks for the number of new entrants who provided GRE scores and the mean quantitative, verbal, and analytical writing scores (components of the GRE) among those entrants. When preparing survey responses on this metric, the employee principally responsible for rankings surveys consistently converted GRE scores to GMAT scores for purposes of reporting the total number of GMAT test takers, and then reported that zero entrants had provided GRE scores. As a consequence, these Fox survey responses indicated no GRE takers and no GRE scores. With respect to class size, U.S. News asks for the average class size of courses that are part of the OMBA program. For at least the 2016 through 2018 U.S. News rankings, Fox reported the average class size on the basis of a student-to-faculty ratio, dividing the number of students in a class by the total number of instructors and “academic coaches” assigned to the class. With respect to technical staff, U.S. News’s OMBA survey asks for (a) the number of fulltime and part-time technical staff the school employed to offer technical assistance to OMBA faculty members, and (b) the number of faculty members at the institution those technical staff are employed to assist. For at least the 2016 through 2018 rankings, Fox likely underreported the number of faculty at the institution that the full-time technical staff support, resulting in an inflated ratio of full-time technology support personnel to supported faculty members. 4. Missed Opportunities to Prevent Reporting of Inaccurate Data The investigation revealed that there were numerous opp ortunities over the last several years for Fox personnel to prevent the reporting of inaccurate information to U.S. News or to correct inaccurate information that had been submitted. Certain of these opportunities arose from the distribution of draft survey responses by the employee principally responsible for rankings surveys to other Fox personnel. In some cases, the inaccurate data would have been clear from a thorough review of the draft survey responses but were missed. At various times, numerous Fox personnel, including members of Fox’s leadership team, received one or more draft responses to rankings surveys that contained inaccurate information, but failed to either identify or correct those errors. In other cases, inaccurate data were explicitly identified by the employee principally responsible for rankings surveys or by Fox personnel reviewing the draft responses, but were not corrected prior to submission. For example, during September 2017, Fox personnel raised concerns about the accuracy of the draft survey responses for the 2018 OMBA rankings with the employee principally responsible for rankings surveys, the supervisor of this employee, and others at Fox. Despite these communications, the previously flagged inaccuracies were not corrected prior to the submission of the survey responses. U.S. News reached out to Fox on at least two occasions with questions relating to certain data that Fox had included in a survey response. Most notably, in November 2016, U.S. News emailed the employee principally responsible for rankings surveys and others regarding Fox’s survey response (for the then-yet-to-be-published 2017 OMBA rankings) that all 198 of the program’s new entrants had provided GMAT scores as part of their applications. In its inquiry, U.S. News noted that Fox’s website indicated that it granted waivers of the GMAT requirement and questioned the accuracy of the reported GMAT information. The employee responded to this email without substantively addressing the accuracy of this GMAT information, and the inaccurate information was never corrected. RECOMMENDED REMEDIAL AND CORRECTIVE ACTION Based on the findings of the investigation, Jones Day has recommended that the University undertake certain remedial and corrective actions: 1. Assess relative responsibility of relevant personnel, and determine and take appropriate personnel action. 2. Communicate identified instances of misreporting to U.S. News and other constituents. Page 6 of 7 3. Efficiently and effectively determine nature and extent of any other misreporting by Fox; communicate any such misreporting as appropriate. 4. Institute policy specific to solicitation; compilation; verification; cross-checking; and submission / reporting of data to external constituencies to address key aspects of data reporting process, including supervision and oversight. 5. Institute explicit policy addressing mandatory reporting of suspected / potential inaccurate data reporting. 6. Provide training for employees about best practices in data collection, verification, and reporting to maximize precision and accuracy. 7. Develop and implement revised rankings process for soliciting, compiling, verifying, cross-checking, and submitting rankings data consisting of key elements, including— a. Form group responsible for reviewing rankings submissions that will: (1) resolve interpretive questions through consultation with rankings agencies or otherwise; (2) memorialize resolution of interpretive issues; and (3) authorize the collection and reporting of data consistent with resolution of interpretive decisions, and with all applicable policies; b. Identify personnel and sources from which / whom to solicit each category of information / data requested; c. Develop and implement reconciliation of existing databases; d. Identify and specifically designate Fox / University employee(s) responsible for soliciting, compiling, verifying, cross-checking, and submitting information / data requested; e. Identify and designate supervisor(s) of employee(s) noted in (d), above. f. Create repository for data compilation with access rights limited to, e.g., group responsible for reviewing rankings submissions, the head of Institutional Research and Assessment (“IRA”), and select representative(s) from the Department of Internal Audits; g. Implement process for saving and archiving supporting data, including, among other things, a “snapshot” of relevant source data as it existed pre-submission, the date on which data were submitted to repository, identity of submitter, etc.; h. Implement process for verifying / approving submissions, including a formal review by the group designated as responsible for reviewing rankings submissions; i. Prepare a process map setting forth the processes outlined above; and j. Develop and administer training program for all relevant personnel. 8. Retain all supporting documentation (e.g., Cognos reports, exports from other databases, and other data “snapshots”), working papers (e.g., memorialization of decision-making on interpretive questions, records reflecting data compilation and associated calculations / analysis, etc.), and final submissions for each ranking survey for no less than seven (7) years. 9. Periodically review compliance, via Internal Audits. 10. Promote and reinforce institutional commitment to key values in onboarding orientations; training modules specific to data collection, verification, and reporting; and in employee handbook. 11. Promote and reinforce key aspects