UA surveys may not be up to statistical snuff

The Undergraduate Assembly prides itself on representing the concerns of Penn students. But the survey data that help them do so may only represent students with a specific agenda, experts say.

The UA distributes several online surveys each semester - there have been two this fall - to the entire undergraduate population via e-mail and the UA's own Web site. Responses are strictly voluntary.

But the fact that students choose to respond - rather than being randomly selected - may mean the UA's data are skewed by bias, statisticians and professional pollsters say.

The UA is currently administering two surveys. One - concerning the Penn Video Network - has garnered 400 responses so far. The other - about Mobile Campus, a campus-wide text-messaging system - has roughly 250 replies.

The purpose of UA surveys is to "provide data to back our arguments," according to UA Chairman and Wharton senior Brett Thalmann. They use the information in their proposals and other initiatives.

To this end, "we try to get a high number of people to take" our surveys, Thalmann said. "We also ask for demographic information."

The number of undergraduates who participate in these surveys ranges from a few hundred students to several thousand.

For instance, last year the UA ran a survey on housing that garnered 1,624 replies, or roughly 16 percent of the total undergraduate student population of 10,422, according to UA Secretary and Engineering junior Dipal Patel.

Whatever conclusions the UA draws from this, however, may be tainted by what statisticians call "non-response bias."

In other words: How did the UA know that the opinions of the 16 percent of undergraduates who responded truly mirror the opinions of the 84 percent who chose not to? Those who respond may have an agenda the majority lacks or hold opinions that the majority does not share.

To attain statistical significance, voluntary response surveys must have a higher participation rate, Statistics professor Abraham Wyner said.

Thalmann acknowledges that biased surveys are inevitable, but defended the UA's current surveying practices, saying that "the current way we do it is the best way. . There's nothing we can do to fix it."

However, Wyner suggested the body can change its methods to get more reliable data.

A sample that is chosen randomly, rather than self selected, may help guard against bias, he said. The group doesn't even need to be as large - he suggested a sample size of about 100 for undergraduate surveys.

Wyner said that while large sample groups may provide "an aura of legitimacy," it's preferable to "have a smaller sample of random people with a high return rate."

Joe Donovan - president of WDG Consulting LLC, a firm that does marketing and opinion research - agreed that such an approach would generate more reliable data.

"It's a much more scientific method," he explained.

Please note All comments are eligible for publication in The Daily Pennsylvanian.