Methodology

Collecting Survey Data

Survey researchers employ a variety of techniques in the collection of survey data. People can be contacted and surveyed using a number of different modes including by an interviewer in-person or on the telephone (either a landline or cell phone), via the Internet, or by paper questionnaires delivered in person or in the mail.

The choice of mode affects who can theoretically be interviewed in the survey, the availability of an effective way to sample people in the population, how people can be contacted and selected to be respondents, and who responds to the survey. In addition, factors related to the mode, such as the presence of an interviewer and whether information is communicated aurally or visually, can affect how people respond.

Survey response rates often vary for each mode and can be affected by different aspects of the survey design (e.g., number of calls/contacts, length of field period, etc.), but in recent years, surveyors have been faced with declining response rates for many surveys, which we discuss in more detail in the section The problem of declining response rates.

Although the Pew Research Center primarily conducts landline telephone surveys, we also occasionally conduct cell phone surveys, Internet surveys, and mixed-mode surveys, where people are surveyed by more than one mode. We discuss these types of surveys in the following sections and provide examples from our surveys that used each method. In addition, some of our surveys involve reinterviewing people who we have interviewed previously to see if their attitudes or behaviors have changed. For example, we reinterview voters during presidential election years to understand how their opinions change.

Cell phone surveys

Telephone surveys have traditionally been conducted only by landline telephone. However, as the proportion of Americans who rely solely or mostly on a cell phone for their telephone services continues to grow, more surveys are being conducted with people on their cell phones. Often, cell phone surveys are conducted in conjunction with a landline survey to improve coverage and the data are then combined for analysis. (More information on sampling cell phones) However, there are also unique challenges that arise when surveying people on their cell phones (also see 2007 special issue of Public Opinion Quarterly Cell Phone Numbers and Telephone Surveying in the U.S.).

One of the most important considerations when conducting cell phone surveys is that costs are substantially higher than for a traditional landline survey. Cell phone surveys require many of the same fixed costs as landline surveys such as purchasing a sample, programming the questionnaire, pre-testing, processing and weighting the data, etc. However, since cell phone surveys often accompany a landline survey, these costs have to be factored in twice, once for the landline survey and again for the cell phone survey.

The increase in costs stems from a variety of factors. One of the most expensive aspects of surveying cell phones is that federal regulations require all cell phone numbers to be manually dialed whereas an auto-dialer can be used to dial landline numbers before transferring the call to the interviewer. In the U.S., most people pay for both outgoing and incoming calls on their cell phones. This has led to surveyors offering cash reimbursements to respondents to help offset any costs they might incur while taking the survey. The cost of these payments as well as the additional time necessary for interviewers to collect contact information in order to reimburse respondents also increases the costs of conducting cell phone surveys. Another difference stems from the additional costs of screening respondents in cell phone surveys because a significant number of people reached are under the age of 18 and thus not eligible for the survey. In our most recent analysis, more than 40% of cell phone respondents who were willing to cooperate with the survey were underage and thus could not be interviewed whereas no households were ineligible in the landline survey because they contained no adults (see The Impact of "Cell-Onlys" On Public Opinion Polling for more information).

In addition to costs, most cell phones have caller id or other screening devices where people can see the number that is calling before deciding to answer. People often view calls from surveyors or marketers to their cell phones as an intrusion and may be more reluctant to accept calls from unknown numbers. There is also wide variation in how people use their cell phones (whether they are turned on all the time, used only during work hours, or used only for emergencies, etc.). Cell phone surveys are also influenced by the respondents' environment (as are landline surveys but people responding to landline surveys are always at home). In particular, the increase in restrictions about the use of cell phones while driving has raised the issue of whether people should be responding to surveys on their cell phones while driving in a car. In addition, people are often talking on their cell phones in more open places where they may have less privacy. These two concerns have led some surveyors to ask respondents whether they are in a safe place and whether they can speak freely before continuing with the interview. Lastly, the quality of connection may also impact whether an interview can be completed at that time and more frequent interruptions can occur in connections.

Although some research suggests that response rates may be lower for cell phone surveys, analysis from our most recent report on cell phones suggests that response rates and cooperation rates were similar for the landline and cell samples (see The Impact of "Cell-Onlys" On Public Opinion Polling for more information). Similarly, even though some people have suggested that respondents may be less likely to cooperate with cell phone surveys or may be more distracted, research has not found substantive differences in the quality of responses between landline and cell phone interviews. As discussed in our most recent report, interviewer ratings of respondent cooperation and level of distraction were similar in the cell and landline samples with cell phone respondents showing slightly greater cooperation and less distraction.

Related Reports and Commentaries

Back to top

Internet surveys

More than any other survey mode, internet surveys have increased dramatically in the last 10 years. Web surveys have proliferated because they can be used to conduct random sample surveys of members of selected populations who have access to the internet and the skills necessary to complete a survey on the web (e.g., students, members of voluntary associations, etc.). We have conducted several mixed-mode surveys of journalists where respondents complete the survey via the web or by telephone. In addition, we conducted a web survey of activists involved in Howard Dean's presidential campaign.

Although more surveys are being conducted via the web, internet surveys have not replaced other modes for surveys of the general population. Internet surveys of the general population are subject to significant biases resulting from undercoverage and nonresponse. Not everyone in the U.S. has access to the internet and for those who do, the speed of connection and skills necessary to complete a web survey can vary dramatically. In addition, there are significant differences between those who do have access and those who do not. Fewer people aged 65 and older, people with lower incomes, and those with less education, as well as fewer African-Americans have access to the Internet. These groups and people living in rural areas are also underrepresented among those with high-speed Internet access (see the Pew Internet and American Life Project for the latest trends)

Surveyors use two main strategies for surveying the general population using the internet. Some may rely on panels of respondents who opt-in or volunteer to participate in surveys, while others are simply one-time surveys that invite participation from whomever sees the survey. These surveys are subject to the same limitations facing other surveys using nonprobability-based samples: the relationship between the sample and the population is unknown so there is no theoretical basis for computing or reporting a margin of sampling error and thus for knowing how representative the sample is of the population as a whole. (also see American Association For Public Opinion Research - Opt-In Surveys and Margin of Error). We discuss results comparing responses from an online opt-in panel survey to those obtained using our traditional RDD telephone survey in Online Polling Offer Mixed Results.

Another strategy for creating an online panel is to use another mode (often telephone) to randomly recruit people to participate and then have them respond to surveys via the web. Recruitment using probability-based sampling via another mode allows surveyors to estimate a margin of error for the survey (see Why probability sampling for more information).

Related Reports and Commentaries

Back to top

The problem of declining response rates

As Americans are faced with more demands on their time and a growing number of unsolicited telephone calls, many have armed themselves with increasingly sophisticated technology for screening their calls (e.g., voice mail, caller ID, call blocking, and privacy managers) and are exercising more choice over when and how they can be contacted. Thus, fewer Americans are participating in telephone polls than was the case when telephone surveys first became prevalent; as a consequence, response rates have been declining over the past decade or more.

The Pew Research Center has conducted two survey experiments to gauge the effects of respondent cooperation on the validity of the results. These experiments compare responses from a standard survey, conducted with commonly utilized polling techniques over a five-day period, and a survey conducted over a much longer period that employed more rigorous techniques aimed at obtaining a high rate of response.

Findings from both the 2003 study Polls Face Growing Resistance, But Still Representative and the 1997 study Conservative Opinions Not Underestimated, But Racial Hostility Missed indicate that carefully conducted polls continue to obtain representative samples of the public and provide accurate data about the views and experiences of Americans. These results are also reported in the Public Opinion Quarterly articles.

Related Reports and Commentaries

Back to top

Mixed-mode surveys

Survey researchers employ a variety of techniques in the collection of survey data. People can be contacted and surveyed using a number of different modes including by an interviewer in-person or on the telephone (either a landline or cell phone), via the Internet, or by paper questionnaires delivered in person or in the mail.

The choice of mode affects who can theoretically be interviewed in the survey, the availability of an effective way to sample people in the population, how people can be contacted and selected to be respondents, and who responds to the survey. In addition, factors related to the mode, such as the presence of an interviewer and whether information is communicated aurally or visually, can affect how people respond.

Although the Pew Research Center primarily conducts landline telephone surveys, we also occasionally conduct mixed-mode surveys, where people are surveyed by more than one mode. For example, we have conducted several mixed-mode surveys of journalists where respondents complete the survey via the web or by telephone.

Related Reports and Commentaries

Back to top

Reinterviews

Reinterviews are typically used to examine whether individuals have changed their opinions, behaviors, or circumstances (such as employment, health status, or income) over time. Many terms are used to describe these surveys including panel surveys and longitudinal surveys. The key feature of this survey design is that the same individuals who were interviewed at the time of the first survey are then interviewed again at a later date. The Pew Research Center sometimes conducts reinterviews, especially to learn more about whether and how voter's opinions change during the course of a presidential election campaign. For an example from the 2004 presidential campaign see Swing Voters Slow to Decide, Still Cross-Pressured. For an example comparing foreign policy opinions before and after the events of September 11, 2001 see America's New Internationalist Point of View.

Some of the reports listed below used reinterviews primarily to ask follow-up questions about their opinions rather than to analyze opinion change on the same issues. Survey reports of this sort include Beyond Red vs. Blue and Voters Like Campaign 2004, But Too Much 'Mud-Slinging'.

Related Reports and Commentaries

Back to top