Not All Science Is Equal

Understanding the Levels of Scientific Proof So You Can Think Critically, Not Cynically

All physicians are trained in science, so when I see a commercial, hear a friend, or relative talk about a new gadget or supplement, I sometimes cringe because they often attribute high scientific evidence based on who said it or where it came from on social media. Even physicians can get this wrong for various reasons. Recently, people have heard the statement, “follow the science,” but far too often, they don’t understand what that means. I’m not here to tell you what to believe, but rather how to apply critical thinking when it comes to a scientific statement.

Here’s the sleight of hand: someone can call something “science” and be right, but not all science is equal. The stronger the study design, the more confidence we can have that the results reflect the truth.

In other words, the level of trust we place in a study depends on the quality of evidence.

Let’s go through the hierarchy of Scientific Evidence (Strongest to Weakest).

1. Double-Blind, Placebo-Controlled Randomized Trials (RCTs)

This is the gold standard of evidence. It’s the strongest scientific evidence.

• Participants are randomly assigned to groups.

• Neither the participants nor the researchers know who is getting the real treatment and who is getting the placebo (the fake treatment).

• This reduces bias and ensures the effect is from the treatment—not expectations or chance.

It is the single best way to prove cause and effect.

2. Systematic Reviews and Meta-Analyses

These are “studies of studies.”

• Researchers gather all high-quality trials on a topic and analyze them together.

• A meta-analysis actually combines the data to look for overall trends.

One good trial is helpful, but results can vary. Combining many trials gives a more reliable answer.

3. Cohort Studies (Observational)

• Researchers follow large groups of people over time to see what happens naturally.

• They don’t assign treatments—they watch.

Example: Tracking tobacco chewers vs non-tobacco chewers over 20 years to see who develops mouth cancer in each group.

But caution: These studies show associations (correlation), but can’t fully prove cause because people aren’t randomly assigned. Scientists will tell you that “Correlation does not imply causation.” For example, ice cream sales and drowning deaths both go up in the summer. They are correlated. BUT, we can all agree that ice cream does not cause drowning. We need controlled studies to prove that one actually causes the other.

---

4. Case-Control Studies

• These studies start with people who already have a disease and look backward to see what might have caused it.

Example: Compare people with colon cancer to those without to see how their diets differed.

This method is weaker than cohort studies because it relies on memory or existing records, which may be incomplete or inaccurate.

---

5. Case Series and Case Reports

• Detailed stories of one patient or a small group.

Example: A doctor notices three patients recovering unexpectedly after a new treatment, or notices three patients developed liver failure after taking the same brand of supplement for sleep.

This is useful for brainstorming, but it is too limited to prove anything. It may serve as a starting point for conducting higher-quality studies to prove or disprove the doctor’s theory.

---

6. Expert Opinion and Mechanistic Reasoning

• What specialists think based on their experience, or what should happen based on biology or theory.

Example: “This antibiotic should help because it reduces inflammation in cells.”

This is the lowest level because it’s not based on real-world testing—just educated best guesses. Again, still “science,” but of very low quality. This is not something you would trust your life with until higher-quality studies are done.

---

SUMMARY

Level

Type of Evidence

Strength

1 Randomized, double-blind, placebo-controlled trials Highest

2 Systematic reviews & meta-analyses Very strong

3 Cohort studies Strong but observational

4 Case-control studies Moderate

5 Case reports/series Weak

6 Expert opinion, lab data, theory Lowest

---

Takeaway

• Top tiers (1 & 2) prove that something works in real people.

• Middle tiers (3 & 4) show a strong association but not absolute proof.

• Bottom tiers (5 & 6) are best for ideas and early warning signs, not solid conclusions.

In short: Not all science is equal. The stronger the study design, the more confidence we can have that the results reflect the truth.

The higher the level, the stronger the evidence for cause and effect in the real world.
Expert opinion and isolated cases can suggest new ideas, but only randomized controlled trials and meta-analyses provide true confidence that something actually works.

One last thing to understand. You may hear of a landmark study.

A landmark study changes the way medicine is practiced, often because it provides strong, clear evidence that leads to new guidelines, treatments, or shifts in thinking. It can occur at any of the top tiers of evidence — but it’s almost always from:

· Randomized controlled trials, or

· Systematic reviews/meta-analyses, or occasionally

· Large cohort studies (especially when RCTs aren’t ethical or possible)

---

Courtroom Analogy:

A landmark study is like a Supreme Court case that sets a new legal precedent.
It doesn’t matter if there were other cases before — this one changes the rules going forward.

---

Characteristics of a Landmark Study:

• Large number of participants

• Strong study design

• Published in a top medical journal (NEJM, JAMA, Lancet)

• Leads to guideline changes, FDA approvals, or new standard-of-care treatments

• Often cited thousands of times

---

Example:

• The Framingham Heart Study (a cohort study) — changed how we understand cholesterol and blood pressure.

• The Women’s Health Initiative RCT on hormone replacement therapy — changed national prescribing practices.

Being “landmark” is about impact, not just design.
A study becomes landmark because it changes what we do in the real world.

A landmark study is not a level of evidence — it is a game-changing study that shifts medical practice or scientific understanding.

In Simple Terms:

A landmark study doesn’t just provide evidence—it changes the direction of medicine or science.

Now you are informed. When someone says to you that you should take this “whatever” because it’s backed by science, ask these questions:

1) What kind of study was it? (because you now know a randomized controlled study is the strongest and expert opinion is the weakest)

2) Does the study show causation or just correlation?

3) How strong was the effect? In other words, was it statistically and clinically meaningful? A 2% improvement may be statistically significant but meaningless in real life.

4) Who funded the study? Was the research funded by a drug company or a group with a vested interest in the findings? Funding does not automatically invalidate a study, but it raises your alert level to examine the study with caution.

5) Has this finding been repeated, or is it a single study? One study is interesting, and many studies with similar results are convincing.

6) Did the study include people like me? What’s true in a 30-year-old healthy person may not apply to a 60-year-old with chronic conditions.

7) What’s the downside? Good science always reports the harms and limitations, not just the benefits.

Science isn’t just about results. It’s about how the results were obtained. Strong evidence can stand up to these questions, but weak evidence falls apart under scrutiny.

Now you realize that following science is more complex than it appears. Keep this in mind for the future because someone, somewhere, will be telling you to use or take this or that. Again, not all science is equal!