Getting the questions right in polling

FILE PHOTO: U.S. President Donald Trump pumps his fist as he departs after attending a CEO town hall on the American business climate at the Eisenhower Executive Office Building in Washington, U.S., April 4, 2017. REUTERS/Kevin Lamarque/File Photo

(NEW YORK) Reuters - Political polling is a tricky business. Controversial issues and politicians can elicit strong reactions from voters, and the phrasing of questions is crucial to how they answer. Part of the challenge, professionals understand, is how a poll question is framed, and any facts or names included in it can often determine the way people respond. So, how do you frame a survey to spot and account for any hidden biases of voters?

That was the task before our polling team, led by U.S. Polling Editor Chris Kahn, for an online survey we conducted in February and March, in which we sought insight into what Americans were thinking about a series of controversial statements made by the country's new president, Donald Trump. (Read the story here)

We divided the 14,000 respondents into two groups. Each group was given a set of nearly identical questions related to statements Trump made during the campaign on topics such as abortion, healthcare and taxes.

With the first group, we made clear that the statements were made by Trump. With the second group, however, we didn’t tell respondents that the words were Trump’s; we simply printed the statements and asked people if they agreed or disagreed with them.

To our readers, from the Editor-in-Chief

Read about Backstory and our Trust Principles

This poll confirmed that opinions can shift dramatically based on a person’s feelings about Trump. For example, 56 percent of Republicans in the group who didn’t know the statements came from Trump himself agreed that a “government official should be forbidden from financially benefiting from their position in any way.” When the words “government official” were switched for “Donald Trump” for the other group, Republicans flipped their responses. Some 70 percent said that Trump should be able to financially benefit “if he puts the country first.”

This method of querying people, in which respondents are randomly given two sets of differently worded questions on the same topic, is known as a split poll. Splits can help pollsters uncover what voters really think. Asking the same question in two ways can show whether like-minded and demographically similar people are swayed by the way facts are presented, or - in the case of the president - people’s feelings about an individual. This poll showed that Trump polarizes the public just by wading into the debate and that people will orient their opinions according to what they think of the president.

As pollsters, it is our duty to drill down deep to see how people actually feel. We will continue to ask questions in different ways to identify and understand nuanced answers.