When Bad Research Leads to Bad Policy: The Case of New Brunswick

This opinion piece was in response to EFI policy development in NB. It was written by Philip Oreopolous who is an Associate Professor of Economics, University of British Columbia and a University of Toronto Research Fellow, Canadian Institute For Advanced Research. It was written just before the decision was announced on August 5th and unfortunately it was not printed. However, it is a great piece and it should see the light of day. I believe it is still relevant, particularly when we see what has recently been written in the Ottawa Citizen (see post below).

When Bad Research Leads to Bad Policy: The Case of New Brunswick

Governments in Canada pay a lot for research and commission reports regularly to help develop public policy. Recently, policy makers have been placing more emphasis on experimental approaches to research for determining whether social programs are worth maintaining, or whether better alternatives to current policies exist.

These methods, sometimes collectively referred to as ‘evidence based policy’, involve setting up pilot projects or using historical events that allow the comparison of groups of individuals that are eligible or engaged in a particular program against other groups of individuals that are not eligible for the program, or enrolled instead in an alternative program. The experimental approach leads to strong conclusions about the overall impact of one policy compared to another.

Most often, however, governments rely on non-experimental research to draw policy conclusions. Non-experimental reports often use surveys or interviews, without a comparison group to draw policy conclusions. This approach requires working with data that was never intended to answer the questions at hand.

Often these reports draw strong conclusions when they really should not. We are inundated with research and reports drawing strong conclusions, and it is virtually impossible to tell from reading an executive summary or listening to a sound bite whether a study should be taken seriously or not.

Taking a report’s conclusions or sound bites at face value, without initial skepticism about how the report came to these conclusions can sometimes lead to disastrous policy mistakes.

For example, take the case of French Immersion reform in New Brunswick, which I use in my Public Policy class as a case example of when bad research can lead to bad policy. ....

Click here to link to full opinion piece