Ask the Right Questions!

Parents beware!

When research or data is cited to back up proposed policies, ask the right questions!

Much of what is done in education today purports to be “data driven.” This means that school districts, foundations and government officials often cite data analysis or research studies to support the policies they enact. Unfortunately, research is often used inappropriately or incorrectly. It’s important that parents independently assess the evidence being cited in support of policies that will affect their schools.

This primer is designed not to turn you into a statistician, but to give you some things to look out for as you assess the quality of any study cited as support for a specific education policy. In this endeavor, it’s always helpful to recruit someone who is trained in statistics and data analysis to help with specific questions.

Questions to Ask

Here are some key questions to ask of any analysis used to justify a particular policy.

1. Where was the study published and who paid for it? Was the study published in a peer-reviewed journal? Was it supported by a particular group of funders with a political agenda?

2. What was the sample size and time period covered by the analysis?

3. If the analysis involves comparing schools, districts, methods of instruction, etc., are the comparisons appropriate? (the “apples to apples” test).

4. Where can I get a copy of the full study or studies on which a particular policy claim is based?

1. Where was the study published, and who paid for it?

Scholars generally publish their most significant research in peer-reviewed journals. Peer-reviewed means that the analysis is sent to other scholars in the field to determine whether the study was properly conducted and analyzed. Only after other scholars conclude that the study was effectively designed and carried out are these studies published. These are the studies that are most likely to contain meaningful results.

Some studies are commissioned and published by government bodies, foundations, “think tanks” or other policy groups. In such cases, it’s important to look carefully at the credentials of the researcher who carried out the research, and how the results relate to the broader goals of the group that commissioned it.

You should be especially leery of privately commissioned research that is being used to market a specific educational program or product. Promoters of these programs or products often commission their own “research studies,” but they are generally of questionable quality.

2. What was the sample size and time period covered by the analysis?

The most reliable studies involve large sample sizes and lengthy time periods.

One of the most widely admired educational studies is the Tennessee STAR class size project, which established that small classes had a significant positive effect on student achievement. This study randomly assigned more than 7,000 students to smaller or larger classes, and monitored those students’ progress closely for a period of four years. Researchers then continued to check on the students’ progress on a periodic basis for many more years.

A study of this size is much more likely to produce significant results than a study which followed students in a handful of classes for only one or two years. You should be especially cautious about an analysis that relies on data from a single year. Most researchers agree that when it comes to student achievement, at least three years of data are needed in order to start establishing meaningful patterns.

Experimental studies like STAR , which randomly assign students to different “treatments,” are generally the most reliable. Because they are difficult to do, they are also the rarest. For example, while achievement levels between charter and public schools are frequently compared, few of these comparisons are experimental, because the students attending charter schools are self-selected, rather than randomly assigned.

For more information on the STAR study, see: http://www.heros-inc.org/star.htm.

3. If the analysis involves comparing schools, districts, methods of instruction, etc., are the comparisons appropriate?

Comparative analysis is only useful when comparisons are appropriate. This is often very difficult, as no two schools, classes or students are alike.

For example, the Center for American Progress recently released a study that used a popular concept – Return on Investment – to measure the “efficiency” of every school district in the country. The study compared the level of educational achievement in each district to the amount of money that district spent per student. The goal, the researchers stated, was for “inefficient” districts to be able to learn from the practice of “efficient” districts, thus improving the overall “efficiency” of American education.

But the analysis did not include many of the key factors that contribute to per-pupil spending. For example, it did not figure in differences in transportation costs from district to district. As a result, it would be inappropriate to use the study to compare the “efficiency” of rural districts, which generally have high transportation costs, with that of urban districts, where per-pupil transportation costs are generally far lower.

You should be especially careful about large statistical analyses, which combine data from many different studies or sources, because the data is unlikely to be consistent. In this study, for example, researchers calculated each district’s Return on Investment using student performance on state exams. But as the difficulty of state exams varies widely from state to state, much more information would be needed to compare the “efficiency” of a district in one state with a district in another state

For more information on this study, see: http://www.americanprogress.org/issues/2011/01/educational_productivity/report.html.

For a thoughtful critique of the study, which highlights some other significant issues with this kind of analysis, see: http://schoolfinance101.wordpress.com/2011/01/22/if-it-doesnt-work-dont-do-it-caps-roi/.

4. Where can I get a copy of the full study or studies on which a particular policy claim is based?

The data from even the best research studies can be used in inaccurate or misleading ways. It’s always a good idea to have a look at the studies which are being cited, to make sure they’re relevant to the issue at hand.

For example, in the fall of 2010, leaders of Charlotte-Mecklenburg Schools, in Charlotte, North Carolina, sought to close three high-poverty middle schools and create a group of high-poverty K-8 schools instead. In support, district leaders cited a study on the efficacy of K-8 schools, which had been conducted by researchers at Johns Hopkins University and published in a respected journal.

When parents and reporters obtained the full study, however, they learned that while the Johns Hopkins researchers had found some benefits in K-8 schools, they specifically stated that such benefits were far lower in high-poverty K-8s – precisely the kind of schools that CMS leaders proposed to create – and questioned whether creating K-8 schools would be a useful strategy in such cases.

For the study itself, see: “Comparing Achievement Between K-8 and Middle Schools,” found at: http://web.jhu.edu/CSOS/tdmg/results/research.html#middle2.

Good Luck!

While the material in research studies can often seem daunting, especially if the studies involve pages of complex calculations, with a little effort you can start to understand how the field of education research works, and use that knowledge to fight for genuinely effective policies in your schools.

– Pamela Grundy