By: Morgan Jackson
Most people wouldn’t expect used car salesman tactics from the author of a scientific study – but you often have to be as wary of scientists as you are of salesmen.
Too often, the authors of research papers use a compelling title that points to a groundbreaking discovery only to mask the limitations that could undermine the study’s findings. And just like that, you’ve been sold a scientific lemon.
Here are five ways to tell if the study you’re reading is full of sh– well, not science.
Sample size simply refers to the number of subjects a study observes and tests to obtain results and ultimately make conclusions. It is always represented with a lower case ‘n’. If you see “n = ____” that’s how many subjects were tested in a given study (i.e. n = 3452).
The larger the sample size, or the more subjects a study includes, the stronger the foundation on which a study makes its claims. The smaller the sample size, the larger any difference between group scores will have be in order to achieve statistical significance.
In this example, the sample size is not as large as it should be, as acknowledged by the authors as a limitation (at least they acknowledge this!). The study was conducted with 63 subjects, when ideally a larger sample size (three to five times more) would help strengthen the claims made.
Specific Subject Characteristics
Who are the actual subjects? Subject characteristics should be reported and extremely detailed to ensure the population being observed and tested is appropriate for the question or theory being evaluated. If a study makes a strong claim about effects of resistance training on older, inactive adults, the subjects of the study should not be elite college athletes… or rats.
In this example, it’s important to notice that this was conducted on obese individuals of a certain age. Therefore the conclusions being drawn cannot necessarily be drawn for everyone. A healthy 25-year-old is much different than an overweight 45-year-old, and the effects of this specific diet could affect other populations differently.
Amount of Error
Often scientific studies will claim a result is “statistically significant,” meaning that the result did not occur randomly or by chance. However, some results can achieve statistical significance and still have an enormous amount of standard error or variability among the subjects. You can find the amount of error if you look at the reported data that comes with the study – usually shown on a graph or table. A quick eyeball scan, assuming you know what to look for, would show the results of the study are all relatively close to the average – versus them being all over the map. A small amount of error means results are generally replicable, while a large amount suggests results aren’t consistent and should be viewed as a limitation to the study’s claims.
In this example, the error bar is pretty large, indicating a large amount of variability in the subjects’ results. We ideally want to see those bars closer together.
Always be sure to check who wrote the paper and conducted the study. Bias and conflicts of interest are certainly factors that should be considered when evaluating the claims and interpretations made in a science article. A common example of this would be a research study on the effects of a particular drug, conducted by a…pharmaceutical company…that manufactures the drug in question. If you read who the author acknowledges following the conclusions, you can find who funded the study and any conflicts of interest.
In this specific piece, the authors reported funding from the National Institute of Health (an impartial federal organization), suggesting that there was little to no bias regarding the interpretation of the results.
When was the study conducted? Checking this important detail seems simple, but will often go overlooked. Is it guaranteed that a study done 20 years ago doesn’t have value? No, but it should be considered. However, it is very likely that more recent studies will provide a clearer picture of the area being examined.
This study was done in 2003, making it 15 years old. There have certainly been more recent breakthroughs regarding this subject, so you may want to check out what’s been written recently on low carbohydrate diets.
Keep in mind that virtually no scientific research article is without limitation. The number of limitations and the severity of those limitations will vary study to study. Limitations do not completely disqualify the results of a study, they only help us gauge the level of skepticism with which we interpret and discuss the results.
So the next time you see an eye-catching headline about a cancer-killing super fruit, be sure to read the fine print! Double check who or what it was tested on, how many times it was tested, who did the testing, and when the tests were conducted.
The above is from a New England Journal of Medicine study on the effects of a low carbohydrate diet.