Penn researcher studies surveys, rankings

Many of this year's early decision applicants knew about the University because of the increased publicity it has received in the last year. And some were impressed by the University's high standing in the U.S. News and World Report college rankings released in September. The University placed 11th -- its highest ranking ever. But prospective freshmen should not take those and other similar reports as fact, according to University administrators. When the rankings came out, Admissions Dean Lee Stetson said they were flawed. "The fact that a school like Johns Hopkins can go from 22 to 10 shows that the method is flawed," Stetson said at the time. "Institutions inherently don't change that quickly." And Assistant Vice President for Planning Susan Shaman, who analyzes the methods and results of surveys like U.S. News and World Report, said much significance should not be placed on the rankings. Shaman oversees the University's institutional research department, which compiles information for the various magazines, college handbooks and other publications that rank and analyze colleges and universities. The department also studies the surveys that the University receives from those publications and the results themselves. And from the work of the four-person institutional research staff, Shaman has been able to understand the significance of the rankings and what they mean to the University. "They are important because they shape public opinion about institutions," Shaman said. "As a result, it is really important that we, as an institution, be considered up there among the top schools in the nation." But Shaman added that the surveys are often very arbitrary, emphasizing different aspects of the institutions that may seem unimportant to the research staff. Each spring, U.S. News and World Report sends out a survey to be completed by every president, provost and dean of admissions of institutions throughout the country. Shaman said some administrators do not express honest opinions on those surveys, putting their institutions higher than they should, or placing every other college or university extremely low on the list. "There are some institutions that don't put Penn in the top quarter," Shaman said. "They also don't put Harvard or Yale or Stanford [there]. It's that absurd. "I have to assume that our president, provost and admissions dean would not purposely rank our competition low," she added. Each school also must fill out a data sheet, which requires statistics in categories such as student selectivity, retention rate and alumni satisfaction. Every year, however, the percentage that each category counts in determining a school's final ranking changes. And many of the categories are questionable or subjective, Shaman added. Alumni satisfaction, for instance, is solely based on monetary contributions. And when analyzing student-faculty ratios, each publication has different cut-off points as to what is favorable and what is not. In some instances, the University will submit different quantitative or qualitative information than that found in the official documentation it provides. The researchers will footnote those passages, explaining where the discrepancy occurred, Shaman said. Often, researchers will contact those who put the rankings together to make clarifications or to lobby them to change the percentage breakdown of categories -- or the survey itself. Since the rankings are arbitrary, Shaman said each school's standing fluctuates from year to year. Depending on what is emphasized in a given survey, the University could be in the top 10 or much lower than that, she added. "There's no reason to think we shouldn't be in the top 10," Shaman said. "But if we fell to 16th or 18th because of something structural in the change in the survey, I wouldn't be altogether shocked -- just disappointed."

Please note All comments are eligible for publication in The Daily Pennsylvanian.