Whitehorse Daily Star

YG surveys should carry disclaimer: researcher

In the last year,

By Taylor Blewett on December 29, 2017

In the last year, Yukoners may have noticed an uptick in government public engagement surveys calling for input on a wide range of subjects from cannabis legalization to school calendar dates to a carbon tax rebate.

The Yukon Liberal government has iterated on multiple occasions the priority it’s making public engagement.

On Oct. 30, it announced the launch of a new website dedicated to sharing government consultations and their results. Twenty-two engagement initiatives have been completed this year, and many included surveys.

While public consultation has value, market researcher Donna Larsen acknowledges, it’s the blurring of lines between Yukon government engagement surveys and professional public opinion and market research that has prompted her to share a number of concerns with the Star about the way these survey results are being gathered, presented and interpreted.

Larsen is a partner with DataPath Systems, a market research company out of Marsh Lake.

After the Yukon government published its hugely popular cannabis legalization survey results Nov. 9, at least three Yukon media outlets – including the Star – reported these results in a way that made them seem representative of all Yukoners.

A significant number of Yukoners responded to the cannabis survey – more than 3,000.

However, the survey’s finding that 81 per cent of respondents support the federal government’s plan to legalize cannabis, for example, cannot be extended to mean 81 per cent of Yukoners support the federal government’s plan.

Public opinion and market researchers use a variety of methods and controls to obtain survey results that are applicable to a wider population.

A number of factors prevent the Yukon government’s cannabis survey and other engagement surveys from being representative.

“When they invite people to do a survey on a particular topic, they’re inviting people who have a vested interest in the outcome of that survey. So that right away disqualifies it from being a scientific, representative study,” Larsen explained.

Another red flag is that the demographic breakdown of survey respondents does not match the demographic breakdown of the Yukon’s population.

Just over five per cent of respondents, for example, were aged 65 and older.

According to the 2016 Statistics Canada Census, almost 12 per cent of the Yukon’s population is 65 and older.

“It does not represent or reflect the opinions of Yukoners. It only reflects the people that took the time to respond to that survey,” Larsen told the Star.

“The Yukon government seems to be totally now embracing this; they’re doing it on every topic.”

Kara Johancsik is a public engagement advisor with the Executive Council Office.

She told the Star Thursday that the Liberal government has indeed embraced these public surveys on government policy since taking office in December 2016.

“The reason we’re engaging a lot right now is because we’re also taking action on a lot of items as we enter the second year of this mandate,” Johancsik said.

“In order to make those decisions in the best possible way, we want to hear views of Yukoners and have those views reflected in decisions.”

So why is the government making these decisions based on survey results that do not represent the views of the Yukon’s population as a whole?

And why are media outlets reporting on these survey results as if they do?

To the former question, Johancsik said the government isn’t striving to do market research nor create a statistically representative picture of Yukon public opinion with these engagement surveys.

That’s because not everyone cares about or will be affected by every government policy decision, she said.

“We know that depending on the issue that we’re asking people about, some people are going to be very impacted by it, care a lot about it, have a lot to say. Some people may not care at all,” she said.

Johancsik pointed to the Legal Profession Act as one example. An engagement survey on the legislation saw 45 respondents.

“If we were doing a random sample every time we wanted to engage on a policy issue, it might be quite exhausting for people,” Johancsik said.

She was referencing one of the methods market researchers use to obtain representative survey results.

It would also be expensive and time-consuming, she added.

“We want to give people the option to not participate if they don’t have anything to say about it, but also give them the option to participate if they want to contribute.”

And the government does so in a variety of ways, she pointed out.

Surveys are made available online and often in hard copy as well, and face-to-face engagement sessions are also hosted.

“Public engagement in terms of the online surveys we do are designed to be inclusive and accessible rather than representative, and I don’t see that changing,” Johancsik said.

But Larsen argues that distinction needs to be made clearer.

“Listening to Yukoners, getting input from Yukoners, sure, great,” Larsen countered. “But they need to not step over the line between consultation and research.”

The government has an obligation to make it clear that they are participating in the former and not the latter, she explained.

Using graphs and percentages in presenting survey results – as the Yukon government does – blurs the line between professional research and the public engagement the government says it’s facilitating, Larsen said.

The public sees these symbols and thinks they represent the results of a scientific study.

“It’s not research, it’s not scientific, it’s not representative. Don’t pretend it is,” Larsen said. A disclaimer could help, she noted.

Asked about this possibility, Johancsik said she thought it was obvious that the surveys the government advertises are not representative.

The government invites people to participate, so it’s clear it’s not a controlled sample, she said. In presenting survey results, the number of respondents and their demographic breakdown are listed.

“To me, that’s pretty clear that what we’re talking about here is the people who responded. The respondents, not Yukoners as a whole.”

But, Johancsik went on to say, if this in not in fact evident to those reading the survey results, “we can look at tweaking wording.”

Yukon media outlets are not the only ones to misinterpret government survey results.

In Saskatchewan, the provincial government also put out a survey about the legalization of cannabis.

CBC Saskatoon noted in a story on the results that “because it is an online survey, no margin of error is available and it is considered non-representative.”

However, the Regina Leader-Post headlined its story with “Majority of Saskatchewan residents believe legal age for marijuana should be 19: Survey,” implying representativeness.

So is it the media that have to scrutinize survey results more carefully before reporting on them, or the government that has to make it clear that engagement surveys aren’t representative?

“I think it goes both ways,” Johancsik said. “If we can be clearer about it, then we should be.”

Comments (15)

Up 4 Down 0

Hugh Mungus on Jan 3, 2018 at 4:15 pm

I heard Donna Larsen on CBC this morning and it was just painful. Talk about sour grapes that she didn't get contracted to do a survey for YG.

Up 1 Down 0

She is right on Jan 3, 2018 at 8:17 am

Think of elections - we hear about a party or candidate receiving x% of the vote. But 'the vote' includes only those people who get out which is at most 70%. And think of how the results of Local Improvement Charge 'votes' are used. We all need to be more aware of interpreting the results of surveys and votes, and what they mean. The old DIKW pyramid applies - data needs to be interpreted to lead to information, then knowledge, then wisdom... but people tend to stop at 'data'.

Up 4 Down 2

BnR on Jan 3, 2018 at 6:15 am

Dear Premier Pillai
At some point, you Libs are going to have to stop consulting and GOVERN!
You and scuttle about, reversing decisions, dithering, but, that's not leadership.
You have a vision (presumably it's more than just getting Sandys job), so carry that out. Say what you will about the YP, but at least they made decisions.

Up 3 Down 0

Engagement Professional on Jan 2, 2018 at 5:50 pm

A few points that distinguish engagement from public opinion research need to be highlighted:

1. Engagement techniques used to inform decision making are aimed at those who are impacted by the decision. The genesis of the word "stakeholder".

2. For an engagement professional to represent stakeholder input as synonymous with public sentiment is unethical.

3. Stakeholder input prior to a decision is information for decision makers to consider. Public opinion polls is another. Governments pushing a particular agenda tend to blur the distinction when communicating the rationale for their decisions. It is important for engagement professionals to illuminate the distinction in their "findings" .

4. On some public policy issues the public at large are stakeholders, i.e. form the stakeholder community. This is the essence of peace, order and good governance. A rarity these days since the fall of an independent public service.

Up 3 Down 1

Johnson on Jan 2, 2018 at 10:30 am

That's pretty rich commentary coming from DataPath, who's very scientific sampling methods include signing up to their email list to be contacted for surveys and who's 2011 electio 'survey' was so wrong many think it influenced the outcome.

Up 3 Down 6

Willard on Jan 1, 2018 at 9:06 am

The Liberal governing body sure has been open and transparent with their policy objectives and inquiries. The last governing body could not be accused of this as they didn't understand their own policy so how could they bring it forth to the public?

Up 6 Down 3

curious george on Dec 31, 2017 at 4:27 pm

All of the recent surveys have had fixed response options which made it impossible to answer in a way that wasn't 'fixed' - ie, there is a not a true distribution options, such as disagreeing with options as opposed to choosing the least worse option.

The surveys are a slap in the face of anyone who doesn't agree as the replies can be spun as needed.

All for show, nothing else.
They should be ashamed of the crappy puppet show modern day 'governments' put on.

Up 4 Down 0

ProScience Greenie on Dec 31, 2017 at 1:20 pm

There is nothing stopping the Yukon Party from appealing to the 'young, tech-savy, i-Toy addicted demographic' North_of_60. More than a few in that crowd have a fiscal conservative streak but not many are social conservatives. If the YP wants to win they might want to take that into account.

Up 5 Down 1

Groucho d'North on Dec 31, 2017 at 9:02 am

I recall Premier Silver saying, “Our government is looking forward to using feedback from Yukoners to make balanced, evidence-based decisions.” I hope Mr. Silver is familiar with the geek term GIGO an acronym for Garbage In - Garbage Out. It has been used to define fidelity in audio equipment and data recording for many years. It’s similar to signal to noise ratios which define what is pure and what is static.
Data research is very similar where the answers you get are closely related to the questions you ask and how you ask them. All too often, interpretation of data can be skewed by how the data was captured and presented to the scrutineers. Some call it spin too. Where data is cherry-picked to support a pre-identified outcome, rather than determining the outcome based on what the raw data gathered suggests in clear terms. How supportive data is used must also be simple and easy to see how conclusions were arrived at. Who knew the truth could become so sophisticated?

Up 7 Down 3

North_of_60 on Dec 30, 2017 at 5:33 pm

Yukon government 'surveys' are biased toward a young, tech-savy, i-Toy addicted demographic. Since this group predominately votes Liberal, it's not surprising if the governmenmt's bias is purposeful. In an attempt to appear 'transparent' and 'engaged' all they're doing is working in an echo chamber. But hey, they get the results they want, so why should we expect anything different?

Up 8 Down 0

yukon56 on Dec 29, 2017 at 6:25 pm

You can spin a survey however it suits your purpose

Up 5 Down 3

Aunir Jyoti on Dec 29, 2017 at 4:47 pm

Searched Google Scholar for the contribution of Donna Larsen as a researcher. Did not find any. A "researcher" without any scholarly publication! Really? Anything is possible!

Up 2 Down 4

Simon on Dec 29, 2017 at 3:53 pm

I think it's great that the current government is asking for more and more input and then sharing the survey results with citizens. I didn't vote Liberal so don't count me as a supporter, but I'm in favour of any party that is more open than not.

Up 7 Down 0

ProScience Greenie on Dec 29, 2017 at 3:19 pm

I only tend to trust surveys when the people doing the surveys have math degrees specializing in the various branches of statistics. Just say no to things like Survey Monkey etc.

BTW - Happy New Years to the staff at the Star and to all the readers of this most excellent newspaper.

Up 3 Down 3

BB on Dec 29, 2017 at 2:38 pm

If you are not affected or informed, and do not have an opinion, you are less likely to do the survey. That is good that they get out of the way and allow people who are affected, who are informed, and who do have an opinion, to speak up. The problem with random telephone surveys is that they make it very easy for those who don't know what they are talking about and/or who have very little stake in the outcome, to have an equal say in directing policy. You could look at it that way and say the random telephone surveys are flawed.

Voting works the same way. If you don't care and can't be bothered to go vote, you should not be voting. No loss.

Add your comments or reply via Twitter @whitehorsestar

In order to encourage thoughtful and responsible discussion, website comments will not be visible until a moderator approves them. Please add comments judiciously and refrain from maligning any individual or institution. Read about our user comment and privacy policies.

Your name and email address are required before your comment is posted. Otherwise, your comment will not be posted.