Methodology

Frequently Asked Questions about Polling and Methodology

Why am I never called to be polled?

You have the same chance of being called as anyone else living in the continental United States. This chance, however, is only about 1 in 150,000 for any given Pew survey since there are over 226 million adult-age Americans in the continental United States and our polls typically contact around 1,500 people. Telephone numbers for Pew's polls are generated randomly so that every phone number has the same chance of being called. This process is conducted separately for each poll, so your chances of being contacted are always about 1 in 150,000.

Can I volunteer to be polled?

While we appreciate people who want to participate, we can't base our polls on volunteers. A survey of volunteers is a "non-probability sample" and the results cannot be generalized to the public as a whole. The key to survey research is to have a RANDOM sample so that every type of person has an equal chance of having their views captured. Polls of volunteers would violate this principle since not everyone would have had an equal chance of being included. (See Why probability sampling for more information.) Link to this subtopic in Sampling section And more specifically, the kinds of people who might volunteer for our polls are likely to be very different from the average American - at the least they would probably be more politically interested and engaged.

Why don't your surveys ever reflect the opinions of people I know?

Chances are you don't hang out with a group of friends that represents everyone in America. Your friends, coworkers, and family are probably like you in many ways. If you were to have a group of friends that represents all of the country, you would have acquaintances who are black, white, Asian, rich, poor, Muslim, Catholic, from the South, Northeast, etc. and every combination of those attributes. Few of us are blessed with such a diverse group of friends.

Why should I participate in surveys?

You should participate in surveys for many reasons. First of all, our polls are a way for you to express your opinions to the nation's leaders and the country as a whole. Public officials and other leaders pay attention to the results of polls and often take them into account in their decision-making. If certain kinds of people do not participate in the survey, then the results won't represent the full range of opinions in the nation.

What good are polls?

Polls seek to measure public opinion and document the experiences of the public on a range of subjects. The results inform academics, researchers, and government officials and help to inform the decision-making process. Much of what the country knows about its media usage, labor and job markets, educational performance, crime victimization, and social conditions is based on data collected through polls.

I'm on a "Do Not Call" list. Doesn't that prevent you from calling me?

No. Legitimate survey research is exempt from the Telemarketing Sales Rule, which was adopted by the Federal Trade Commission to fight fraud and protect consumers from harassment. The rule covers marketing but not opinion polling or market research that does not involve an effort to sell you something. Nonetheless, our telephone survey interviewing centers will honor any request not to be called.

Do pollsters have a code of ethics? If so, what is in the code?

The major professional organizations of survey researchers have very clear codes of ethics for their members. These codes cover the responsibilities of pollsters with respect to the treatment of respondents, their relationships with clients, and their responsibilities to the public when reporting on polls.

Most of Pew's pollsters belong to the American Association for Public Opinion Research (AAPOR) and subscribe to AAPOR's code.

Some good examples of a pollster's Code of Ethics include:

American Association for Public Opinion Research (AAPOR)
Council of American Survey Research Organizations (CASRO)

How are political polls different from market research?

There are many similarities but the main difference is the subject matter. Market research explores opinions about products and services, and measures your buying patterns, awareness of products and services, or willingness to buy something. Political polls typically focus on public issues and the views of the public about elected officials. Political polls also try to measure how voters are reacting to candidates in political campaigns and what issues are important to them in elections.

How did you get my number?

Most good telephone surveys of the general public use what is called a random digit dial (or "RDD") sampling technique to generate the sample of phone numbers used in the survey. A database of area codes, telephone exchanges (the first three digits of your seven digit number), and directory listed telephone numbers is used to select "partial numbers" (usually the area code, exchange, and first two digits of the final four numbers) in proportion to the existence of all telephones in a given geographical area. These partial numbers are then filled out by randomly adding the last two digits to them. This ensures that all working telephones, including those not listed in the telephone directory, have a chance of being called.

The goal is to ensure that your telephone has the same chance of being dialed as any other home telephone in the United States. When using this type of telephone sample, we do not know your name when we call. All that we know is that the computer generated your telephone number to call. For more information on our method of selecting telephone numbers, see Random digit dialing - our standard method.

How are people selected for your polls?

Once numbers are selected through random digit dialing (see How did you get my number?), interviewers ask to speak with "the youngest male, 18 years of age or older, who is now at home." If there is no eligible male at home, interviewers ask to speak with "the youngest female, 18 years of age or older, who is now at home." This method of selecting respondents within each household improves participation among young people who are often more difficult to interview than older people because of their lifestyles, and also ensures that we obtain a sufficient number of interviews with men.

What if I only have a cell phone - am I represented in your surveys?

Unfortunately, for most of our surveys, people who only have cell phones are not included in what we call the sampling frame and thus have no chance of being included in these surveys. The number of people who are "cell-only" has been steadily increasing; about 14.5% of the adult population can be reached only by cell phone. This is an issue of continuing concern to pollsters and one which is the subject of a great deal of ongoing research. A recent special issue of Public Opinion Quarterly was dedicated to this issue: Cell Phone Numbers and Telephone Surveying in the U.S.

We can call cell phones and have done so in several studies over the past three years to assess the potential for bias in our landline samples. There are several challenges and extra costs associated with sampling cell phones and conducting cell phone surveys.

Our most recent analyses indicate that when data from both the landline and cell samples are combined and weighted to match the U.S. population on selected demographic measures, the results for key political measures (such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation) are virtually identical to those from the landline survey alone. In 2004, the omission of cell-only voters from election polls did not create a serious bias because these cell-only voters were very similar to others in their age group who could be reached on a landline phone. But there is no guarantee that this will continue to be true in the future.

Don't you have trouble getting anyone to answer your polls?

Yes. The percentage of people we interview, out of all we try to interview, has been declining over the past decade or more There are many reasons for this decline. Some of it is a result of the fact that people are busier and harder to catch at home. Some has to do with the use of technologies such as caller ID, voice mail, and privacy managers. And some is a result of a growing unwillingness on the part of some people to be interviewed. These articles and reports provide greater detail on this issue, and attempt to assess whether declining response rates harm the accuracy of polls (Also see The problem of declining response rates for more information)

Why don't you just conduct surveys on your website?

Good question. Or, more generally, why don't we conduct our polls on the internet, or trust internet polls? It would certainly be less expensive. But polls conducted among people on the internet would not be valid for generalizing to the full population. Two obvious problems: not everyone has internet access (at least 25% of the public does not), and people available and willing to take a poll on the internet do not constitute a representative sample of the public. A good example of the danger of internet polling is described in this 1999 analysis from the height of the controversy over President Clinton's affair with Monica Lewinsky:

Online Polling Offer Mixed Results January 27, 1999

What about people who don't have any telephone service?

Unfortunately, for most of our surveys people who do not have landline telephones are not included in what we call the sampling frame and, therefore, have no chance of being included in these surveys. About 17% of U.S. households do not have landline telephone service. About 2% of households have no telephone service, and about 14.5% of the adult population can be reached only by cell phone. Without using in-person interviewing or a mail survey, there is no way to reach the phoneless households. We can, however, call cell phones and have done so in several studies over the past three years. The results of these studies are summarized here: sampling cell phones and conducting cell phone surveys.

Statistical weighting of our telephone samples helps to correct for the omission of cell-only and no-phone households, but some bias undoubtedly remains for certain kinds of questions. This is an issue of continuing concern to pollsters and one which is the subject of a great deal of ongoing research.

Because people in households with no telephone service are less likely than others to vote, their omission has not seriously damaged the accuracy of pre-election polls. In 2004, the omission of cell-only voters from election polls did not create a serious bias because these cell-only voters were very similar to others in their age group who could be reached on a landline phone. But there is no guarantee that this will continue to be true in the future.

Are election polls accurate?

There has been a surge in interest in election polling given the previous two close presidential elections. In both 2000 and 2004, most national telephone polls were very accurate. Polls in both years forecasted a close election, and in 2004 the average of several major national polls taken in the days leading up to the election showed President George W. Bush with a 1.6 percentage point advantage over Senator John Kerry, just a slight understatement of Bush's actual margin of victory (2.4 percentage points). Of the major news organizations, only one had Kerry ahead, and that was by less than the margin of error.

	Bush	Kerry	Diff	Field Period
Actual 2004 Presidential Vote	50.7	48.3	2.4
Average of 7 polls	48.9	47.3	+1.6
Pew	51	48	+3	Oct 27-30
Newsweek	50	44	+6	Oct 27-29
ABC/Washington Post	49	48	+1	Oct 28-31
Gallup/CNN/USAToday	49	49	0	Oct 29-31
NBC/Wall Street Journal	48	47	+1	Oct 29-31
CBS/New York Times	49	47	+2	Oct 29-Nov 1
FOX	46	48	-2	Oct 30-31

How do you know who is really going to vote?

One of the most difficult aspects of conducting election polls is determining whether a respondent will actually vote in the election. Different pollsters use different sets of questions to help identify likely voters. This report describes an experiment conducted by Pew to identify the best question for predicting which respondents will vote.

Screening Likely Voters: A Survey Experiment May 18, 2001

Does an early lead in the polls usually hold up?

This commentary provides an analysis of whether early front-runners are likely to capture their party's nomination.

Does an early lead in the polls usually hold up? March 4, 1999

Beware of the Bounce

This commentary discusses how candidate ratings may increase during or just after their political conventions but how often these are only short-lived.

Beware of the Bounce August 4, 2000

So Who's Ahead?

This commentary addresses how polling on the presidential horse race can produce different results, particularly early in an election year.

So Who's Ahead? April 14, 2000

Why the Generic Ballot Test?

This commentary discusses "the generic ballot" which is a measure based on the percentage of voters in national surveys who say they would vote for either the Republican or Democratic candidate for the U.S. House of Representatives in their district.

Why the Generic Ballot Test? October 1, 2002

Generic Congressional Measures Less Accurate in Presidential Years?

This commentary addresses how the generic measure of partisan support (see Why the Generic Ballot Test?) can be been less accurate in presidential elections years than in off-years.

Generic Congressional Measures Less Accurate in Presidential Years September 18, 1996

Ignore the Horse Race: Pay Attention to the Trend

This commentary analyzes how early horse race polling may be misleading so more attention should be paid to trends in public attitudes (such as Presidential approval ratings).

Ignore the Horse Race, Pay Attention to the Trend October 30, 1995

Do people lie to pollsters?

We know that not all survey questions are answered accurately but it's impossible to say that any given inaccurate answer necessarily involves lying. In some instances, people simply don't remember their behavior accurately.

More people say they voted in a given election than voting records indicate actually cast ballots, and in some instances researchers have actually verified the voting records of people who were interviewed and found that some of them said they voted but did not. Voting is generally considered a socially desirable behavior, just like attending church or donating money to charity. Studies suggest these kinds of behaviors are overreported. Similarly, socially undesirable behaviors such as illegal drug use, certain kinds of sexual behavior, or driving while intoxicated are underreported.

We take a number of steps to minimize the errors that questions about socially desirable or undesirable activity may entail. For example, questions about voter registration and voting usually include acknowledgements that not everyone takes part in elections. Pew's voter registration question is worded this way:

These days, many people are so busy they can't find time to register to vote, or move around so often they don't get a chance to re-register. Are you NOW registered to vote in your precinct or election district or haven't you been able to register so far?

Do people really have opinions on all of those questions?

People have opinions or attitudes on just about everything. But "I don't know" is a legitimate answer and we always include a category for people who choose not to answer a question.

Why do you typically ask presidential approval first in the survey?

The presidential approval question is a very important political indicator. It is a useful summary measure of the president's standing with the public, and as such, can influence his power in dealing with the Congress, business leaders, and foreign countries. We ask it first in the survey because we do not want any other questions to affect the respondent's answers to it.

For example, if the researcher first asks about the war in Iraq and then asks about presidential approval, the respondent may still be thinking about Iraq when answering. While Iraq may be important in assessing the president's overall performance, so are many other issues. If the respondent is thinking only about Iraq because we reminded him or her of the issue, their response about the president may be biased by what we call a context effect.

Why are demographic questions asked at the end of the survey?

Demographic questions tend to be boring to respondents and also can seem inappropriate and threatening if asked before a level of trust is established in the interview. The interviewer wants to engage the respondent from the beginning of the conversation so that the respondent is interested in the survey and will continue to answer questions. If the interviewer started the survey by asking the respondent's age or sex, the respondent may get bored and decide not to continue with the survey. In addition, if someone called your house and first started asking you how much money you make, your race, how many children you have, etc., you may be put off by these personal questions and decide not to continue.

I looked at your questionnaires. What's all this rotating and randomizing going on in your surveys?

Rotating or randomizing means that items in a list are not asked in the same order to each respondent. We know that answers to questions are sometimes affected by questions that came before them. By presenting questions in a different order to each respondent, we ensure that each question gets asked in the same context as every other question the same number of times (e.g., first, last, or any position in between). This does not eliminate the potential impact of previous questions on the current question, but it does ensure that this bias is spread randomly across all of the questions or items in the list. The same principle applies to the order of response options in a single question. For many questions, we often rotate the order in which the answer choices are presented. For example, if respondents tend to remember (and respond with) the last option read to them, rotating the order of the options means that each option is read last as often as each other option.

How is Form 1 different from Form 2?

We often write two versions of a question and ask each version of half of the survey sample. Thus, we say we have two forms of the questionnaire. Since the respondents are assigned randomly to receive either Form 1 or Form 2, we can assume that the two groups of respondents are essentially identical. If the answers from Form 1 are significantly different to those from Form 2, we can assume that the difference is a result of the way we worded the two versions. For example, in January 2003, we asked this question on Form 1: "Would you favor or oppose taking military action in Iraq to end Saddam Hussein's rule?" On Form 2, we asked: "Would you favor or oppose taking military action in Iraq to end Saddam Hussein's rule even if it meant that U.S. forces might suffer thousands of casualties?" In this experiment, the Form 1 question found 68% favoring removing Hussein from power. But the mention of thousands of U.S. casualties in Form 2 led to far fewer respondents supporting military action: only 43% would favor removing Hussein.

We also use these so-called form splits for the purpose of asking more questions than we would otherwise have room for. If we determine that half of the sample will include enough interviews for a reliable estimate, we will often ask some questions of only half of the respondents. That allows us to include more questions on the survey without burdening any individual respondent with a longer interview.

More Information
About Polling Methods

American Association for Public Opinion Research

National Council on Public Opinion Polls

Roper - Fundamentals of Polling

Pollster.com FAQs

News University Course on Understanding and Interpreting Polls

The Laboratory

Want to participate in a survey experiment with us? Answer the following question, click on Submit Response, and we'll tell you a little more about what we are doing: