Scientific studies, and news coverage about them, often declare that a study “controlled” for certain variables.

For example, the Washington Post reported on a study of GLP-1 drugs and a hormonal disorder. The article noted that “women on GLP-1s lost more weight than those in control groups.”

A couple months earlier, the Post reported on a different study about GLP-1s and bone and tendon injuries. It said “researchers controlled for age, sex, race and tobacco use” and found that people taking GLP-1s had higher risk of skeletal disorders than nonusers.

This language typically gives the impression that having a control enhances credibility. But what exactly does “control” mean? And should you be impressed by it?

I’ve been a statistician for many years, and my recent research has shown that 90% of American adults would base decisions on statistics, at least sometimes, if they understood the basic concepts better. Control is one such concept.

Understanding controls can help you assess how well a study’s conclusions are supported by evidence, and can help you judge the media’s interpretation of new research.

Observed or manipulated?

A control can mean different things depending on a study’s design. Most studies fall into one of two categories: experimental or observational.

In an experimental study, researchers may give a drug or treatment to some participants and not to others. The control group is the people who did not receive the drug or treatment.

In the study of women’s weight loss and hormonal disorders, researchers gave GLP-1s to some of the participants, and did not give them to others, so they could compare outcomes between the two groups. Here, “control” means the researchers directly influenced how the data is generated.

On the other hand, in an observational study, researchers examine existing data, such as electronic health records, to find patterns. They use different statistical tools to see how various factors such as age or sex may have affected outcomes for people, regardless of who received treatment. This step helps make sure that the association between the treatment and the outcome is not being driven by one of the known variables that affects both.

Here’s an example. In the study about GLP-1s and bone injuries, researchers controlled for age, sex, race and tobacco use. What that means is they compared injuries among people who fit in each of those categories, to see if differences existed between those who take GLP-1s and those who did not.

For instance, suppose that smokers are more likely to take GLP-1 medications, and that smoking itself independently increases the risk of bone injuries. If researchers do not control for smoking, it will look like GLP-1 use is linked to more bone injuries simply because many GLP-1 users also smoke.

Here, “control” means to account for elements of data that’s already been gathered, which researchers had no influence over.

Random by design

In an experimental study about a new drug, scientists randomly determine who receives a treatment and who does not. People who do not receive the medication typically receive a placebo, such as a fake pill. They are the control group.

Moreover, if scientists know that certain factors might influence the outcome of a study, such as the severity of an illness or other medications participants may be taking, they can randomize treatment within such groups of people. This is called stratification; it prevents researchers from inadvertently assigning a disproportionate number of participants who share one particular quality to the treatment or placebo group.

For example, one 2021 experimental study investigated weight loss among people who had Type 2 diabetes and were given a GLP-1 drug. The researchers made sure that treatment and control groups had members randomly selected from people with different initial levels of blood sugar, as well as those who were taking different diabetes medications. This way, they are less likely to assign a disproportionate number of participants with, for example, the highest blood sugar to the placebo group by chance. This step also helps the researchers compute the effect of the treatment with greater precision.

Participants in an experimental study may not exactly represent the full variety of people who might someday receive a treatment that is eventually approved. It may be impossible or prohibitively expensive to include enough people to cover every variation.

This challenge is often why researchers turn to observational studies, also called real-world evidence studies. In these cases, researchers use existing records; no randomization can be done. Instead, researchers use different statistical tools to account for factors that may have affected treatments and outcomes.

For example, in an observational GLP-1 study, researchers used medical record databases and estimated how much weight people taking GLP-1 medications lost compared with people taking other Type 2 diabetes medications.

For observational studies, failing to control – failing to account – for important variables can undermine the credibility of the results. For example, an observational study could report health benefits of some nutritional supplement while failing to control for baseline differences in health between users and nonusers. In such cases, healthier individuals might have been more likely to self-select to take the supplement in the first place, making it seem like the supplement has a health benefit even when none exists.

Many such findings subsequently fail to replicate in experimental studies that control for these sources of bias through randomization.

Trying to control for every variable possible would be a problem, too. If they aren’t actually related to the variables of interest – and when running a study researchers don’t know for sure which ones are – more control variables makes mathematical estimates less precise. Controlling for certain variables called colliders can distort the effect a study is interested in.

When you see the findings of observational studies, try to consider if the study falls at one of these extremes – controlling for too few or too many variables.

Judging studies for yourself

To help people make sense of the various kinds of controls, I prepared an online app that lets you compare analyses in observational and experimental approaches, using simulated data.

You don’t need to be a scientist to examine studies that spark your interest. But it is important to recognize that even seemingly intuitive terms such as “controlled” can have very different meanings.

The next time you encounter a story in the media about a new finding, consider whether the study was observational or experimental, and what the researchers controlled for. Keeping the distinctions in mind can help you better judge what conclusions the evidence can confidently support.

Mark Louie Ramos does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.