Skip to content

Is it statistically valid?

Robert Thomas writes about the validity of a recent survey from city council
statistics shutterstock

A Note to Readers:  For some background, take a read on how the City inferred the 2018 results had some sort of scientific validity from one of Robert Thomas’ old stories.  It is used as background for his final paragraph in this article.  

Is 212 people responding to an on-line survey out of 22,000 voters statistically valid? And if so, is the on-line survey a good indicator of what Moose Jaw residents are thinking? 

These were questions Council faced when the results of the 2019 Budget and City Services Survey were presented at their September 23rd meeting.

“When 212 respondents out of a voting population of 22,000 (respond that) is less than one percent it is not statistically valid,” Councillor Swanson said of the survey.

The question to be asked is Councillor Swanson correct or not? Or was it all just ‘hot air'?

To get the answers we did a Fact Checker and asked an expert on the subject of statistics. 

Dr Andrei Volodin PhD is a Professor of Statistics at the University of Regina, as well as co-editor of the Journal of Statistical Calculations and Simulations and when it comes to the statistical value of the City's survey Councillor Swanson is correct.

For the purposes of statistics, the City's survey fails for two main reasons – too low of a sample size and the way the survey was taken usually means the results are biased, Dr Volodin said.

In order to be statistically valid, surveys need to have a five percent margin of error and a 95 percent confidence level, he said. Something the City's survey does not reach.

“If we use five percent margin of error and 95 percent confidence level, taking into consideration that the (voting) population size (all citizens of Moose Jaw) is 22,000, then the minimal sample size should be 378, which is strictly more than the actual sample size of 212.”

When it comes to bias, Dr Volodin said how the survey is done leads to bias because the sample is not random and representative. The survey must target a group and ensure all group members respond.

“Such things are not possible to achieve by using on-line platforms such as Survey Monkey. Only interested people will reply. 

Such type of research is called an observational study, it is NOT real sampling or correct design of experiment. Usually, when we perform an observational study, we are getting biased results.”

In response, City of Moose Jaw communications manager Craig Hemingway said the City never claimed the survey to be scientific but just another means to get feedback from residents to Council. 

“The City of Moose Jaw has conducted resident engagement on the Budget process since 2015, when 195 respondents took part in the Citizen Budget survey. Public engagement is one of many actions we undertake in an effort to provide City Council with as much pertinent information as possible before Budget deliberations begin. We do consider all feedback important and continue to thank those 212 people that took the time to complete the survey,” Hemingway responded. 

“We noted in our discussion that of the people that responded, this is what they told us,” he responded, adding “We never claimed that these surveys are scientific or spoke for the majority of the people in Moose Jaw...”

“We will continue to search for meaningful ways to engage as many residents as possible on important issues, including Budget,” Hemingway responded.
In 2018 a total of 314 people responded to the non-random on-line survey. 

At that time, Council was told the number of people surveyed was close to the 400 randomly surveyed by marketing firm Insightrix regarding curbside solid waste collection and therefore could be considered a moderate representative view of Moose Javians opinions on the 15 questions asked in 2018.

Robert Thomas is acting editor of the news site MJ Independent. A news site dedicated to fill in some of the gaps and give a different perspective in news coverage. A lengthier full version of this story may be found on 

The views and opinions expressed in this article are those of the author, and do not necessarily reflect the position of this publication.