American Research Group, Inc.
RSS  
 
The National Economy
 
Electoral Vote Calculator
 
The New Hampshire Poll
 
Effective Advertising
Scintillating Grid
2015 Christmas Spending
2015 NFL Predictions
 
Survey Tools -
Sample Size Calculator
Margin of Error Calculator
Ballot Lead Calculator
Sampling Error Example
 
 
 
 

 

February 9, 2016

2015 NFL Prediction Accuracy

 

NOTE: NFL Prediction Accuracy is back by popular demand for 2015 even though both models were invalidated in the second week of the season (as they were in the 2014 season).

Microsoft's Cortana and Nate Silver's fivethirtyeight.com Elo are predicting the outcomes of NFL games each week during the 2015 season. (See the accuracy results for the two models for the 2014 NFL season here.)

The tables below use Brier scores to determine the accuracy of the probabilistic predictions from both models.

Brier scores range from 0 to 1, with 0 meaning the probabilities in the models perfectly match the outcomes of the games and 1 meaning no matches were made (so the closer to 0.000, the better calibration of the model).

If a 50% (0.500) probability were assigned to each team for each game, the Brier score would be 0.250.

For the 2015 season, the Elo model has a Brier score of 0.230 and the Cortana model has a Brier score of 0.229. (In the 2014 NFL season, the Elo model had a Brier score of 0.208 and the Cortana model had a Brier score of 0.213.)

Both the Elo model and the Cortana model failed to correctly pick Denver in the Super Bowl.

 

Brier Scores
2015 NFL Season Summary 538 Elo Cortana
Week 1 0.191 0.211
Week 2 0.286 0.278
Week 3 0.190 0.180
Week 4 0.233 0.236
Week 5 0.207 0.212
Week 6 0.184 0.213
Week 7 0.233 0.196
Week 8 0.180 0.173
Week 9 0.228 0.229
Week 10 0.384 0.382
Week 11 0.194 0.223
Week 12 0.206 0.229
Week 13 0.228 0.233
Week 14 0.248 0.212
Week 15 0.166 0.182
Week 16 0.264 0.249
Week 17 0.234 0.228
Wild Card Weekend 0.226 0.220
Divisional Round 0.184 0.117
Championship Sunday 0.160 0.217
Super Bowl Sunday 0.348 0.415
Season totals 0.230 0.229

 

The 95% confidence intervals for both models for the season include 0.250, making both no better or worse than assigning each team a 50% win probability.

The Brier scores for the season reflect that the average win probabilities among winning teams are 56% for the Elo model and 55% for the Cortana model.

On a weekly basis, the Elo model has failed to perform better than assigning each team a 50% win probability seventeen out of twenty-one weeks and the Cortana model has failed to perform better than assigning each team a 50% win probability seventeen out of twenty-one weeks. The Elo model performed better than assigning each team a 50% win probability in weeks 6, 15, and Championship Sunday and the Cortana model performed better than assigning each team a 50% win probability in weeks 3, 15, and the Divisional Round.

In week 10, both models performed worse than assigning each team a 50% win probability. This is not surprising as both models failed to perform better than assigning each team a win probability of 50% in the second week of the season which invalidated both models (as was the case in the 2014 season).

Weekly results are here.

 


 

February 8, 2016

New Hampshire Presidential Preference Primary

Democratic Primary

 

New Hampshire
Likely Democratic
Primary Voters
Feb
2-3
Feb
3-4
Feb
4-5
Feb
5-6
Feb
6-7
Feb
7-8
Clinton 38% 38% 41% 42% 41% 44%
Sanders 54% 54% 53% 53% 53% 53%
Other * * * * * *
Undecided 8% 8% 6% 5% 6% 3%
Read this table down

*= Less than 1/2 of 1%  - = None

 

Fulll results of our tracking survey for the Democratic primary conducted February 7-8 are here.

 

Republican Primary

 

New Hampshire
Likely Republican
Primary Voters
Feb
2-3
Feb
3-4
Feb
4-5
Feb
5-6
Feb
6-7
Feb
7-8
Bush 8% 8% 8% 9% 9% 9%
Carson 2% 2% 2% 1% 1% 1%
Christie 6% 6% 5% 5% 6% 8%
Cruz 12% 12% 9% 9% 10% 10%
Fiorina 2% 2% 2% 2% 3% 3%
Gilmore - - - - - -
Kasich 13% 14% 17% 17% 16% 17%
Rubio 14% 15% 16% 17% 16% 14%
Trump 34% 36% 34% 31% 30% 33%
Other * * * * * *
Unecided 7% 6% 6% 8% 9% 6%
Read this table down

*= Less than 1/2 of 1%  - = None  ni = Not included

 

Full results of our tracking survey for the Republican primary conducted February 7-8 are here.

 


 

February 2, 2016

2016 Iowa Caucuses Pollster Ratings

 

The following tables list the accuracy of pollsters in the February 1, 2016 Iowa Democratic and Republican presidential caucuses using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

An accuracy measure of 0.00 indicates that the odds ratio of the survey results matches the odds ratio of the actual results (the closer the accuracy score is to 0.00, the better).

Lessons from Iowa in 2016:

  • Sampling error prevents even the perfect poll from representing the true population values.
  • Voter file samples may not be superior as they do not include new voters who register at the caucus sites in the sample frame.
  • Two of the late polls in the Republican race picked up the movement toward Cruz and Rubio and two of the late polls missed the movement toward Cruz and Rubio.
  • The best pollster in politics stumbled again in the late poll on the Republican side, missing  both Cruz and Rubio with accuracy measures falling outside the 95% confidence intervals.
  • Mack Shelley at Iowa State University is the best political pollster in Iowa for the 2016 caucuses.

 

2016 Iowa Democratic Caucus
Clinton Sanders Odds
Ratio
Accuracy
CBS/YouGov 46% 47% 0.97 -0.03 
Iowa State 47% 45% 1.04 0.04 
NBC/Marist 48% 45% 1.06 0.06 
Des Moines Register - L 45% 42% 1.06 0.06 
ARG 45% 48% 0.93 -0.07 
Quinnipiac - L 46% 49% 0.93 -0.07 
Monmouth  47% 42% 1.11 0.11 
FOX 48% 42% 1.14 0.13 
Emerson College - L 51% 43% 1.18 0.16 
PPP 48% 40% 1.19 0.18*
CNN 43% 51% 0.84 -0.18 
Gravis Marketing 53% 42% 1.25 0.23*
Loras College 59% 30% 1.95 0.67*
L = late poll 49.9% 49.6%

There are no differences between the polls at the 95% confidence levels except for the Loras College poll.

*The accuracy measures for the Public Policy Polling (PPP), Gravis Marketing, and Loras Colleges polls fall outside of the 95% confidence intervals.

2016 Iowa Republican Caucus
Cruz Trump Odds
Ratio
Accuracy
Loras College 25% 26% 0.84 -0.17 
Emerson College - L 26% 27% 0.84 -0.17 
Opinion Savvy - L 19% 20% 0.83 -0.18 
Iowa State 26% 19% 1.20 0.18 
Gravis Marketing  27% 31% 0.76 -0.27*
CBS/YouGov 34% 39% 0.76 -0.27*
Des Moines Register - L 23% 28% 0.72 -0.33*
ARG 26% 33% 0.69 -0.37*
NBC/Marist 25% 32% 0.69 -0.37*
Quinnipiac - L 24% 31% 0.68 -0.39*
Monmouth 23% 30% 0.67 -0.40*
PPP 23% 31% 0.65 -0.43*
CNN 26% 37% 0.625 -0.48*
L = late poll 27.7% 24.3%

There are no differences between the polls at the 95% confidence levels.

*The accuracy measures for all but the Loras College, Emerson College, Opinion Savvy, and Iowa State polls fall outside of the 95% confidence intervals.

This is the second straight Republican presidential caucus where the accuracy measure for the the Des Moines Register poll has fallen outside the 95% confidence intervals. In 2012, the Des Moines Register late poll missed Rick Santorum's finish by over 9 points.

These details are here.

 


 

January 21, 2016

Obama Job Approvals and the National Economy

 

Obama Job Approval Ratings
1/20/16 Approve Disapprove Undecided
Overall 45% 51% 4%
Economy 42% 51% 7%

 

A total of 45% of Americans say they approve of the way Barack Obama is handling his job as president and 51% say they disapprove of the way Obama is handling his job. In December, 41% of Americans approved of the job Obama was doing and 55% disapproved.

When it comes to Obama's handling of the economy, 42% of Americans approve and 51% disapprove. In December, 40% approved and 55% disapproved.

Among Americans registered to vote, 45% approve of the way Obama is handling his job as president and 51% disapprove. On Obama's handling of the economy, 42% of registered voters approve and 51% disapprove.

Details from the nationwide survey conducted January 17-20 are available at The National Economy.

 


 

January 13, 2016

Sampling Error Example

 

UPDATE: The Sampling Error Example has been updated to reflect Iowa results as of January 13.

We have dusted off and upgraded our sampling error example from 2000 using the latest polling results from the Iowa Repbulican Presidential Caucus.

See an example that shows when most pundits cry "outlier" in the polls, they are wrong.

The Sampling Error Example is here.

 


 

December 18, 2015

2014 SurveyMonkey Polling Accuracy

 

Accuracy measures for 71 online polls from SurveyMonkey in the 2014 November elections have been calculated.

The polls have an absolute average accuracy of 0.11, but the accuracy measures for 34 of the polls (almost 48%) fall outside of their respective 95% confidence intervals. It would be expected that the accuracy measures for about 4 polls would fall outside their respective 95% confidence intervals.

Details are here.

 


 

Why doesn't most advertising work? Take a look at 10 Rules for More Effective Advertising.

 

American Research Group, Inc.  814 Elm Street  Manchester, NH 03101 603/244-3644
2016 American Research Group, Inc. All rights reserved. Reproduction in whole or in part without permission is prohibited.