Welcome to episode 423 of The Whole View! This week Stacy and Sarah take a step back to share on the philosophy and approach that embodies what The Whole View is all about. Sarah breaks down the definition of the scientific method, the various ways that studies are conducted, and how that impacts the results from that study. You will also get a glimpse into how these variables impact how Sarah interprets and shares this information. Tune in below for all the scientific details!
If you enjoy the show, please review it on iTunes!
The Whole View, Episode 423: The Scientific Method
Welcome back to episode 423 of the Whole View. (0:27)
In episode 400 when we talked about why we were moving to The Whole View, we talked about the necessity of having a podcast that focused on the scientific method.
What Sarah wants to be able to help our audience understand are some of the common pitfalls in the alternative health community.
We see this manifestation of bias, of dismissing science that doesn’t conform to someone’s beliefs.
And we see it in a way that is miscommunicating the value of the different types of studies.
One of the things that Sarah wanted to do is talk about how different studies work, which is what this week’s episode will focus on.
There are different structures for different types of studies, and what the weight of evidence is from different types of studies.
We want to help our listeners look at articles from various educators and influencers and empower everyone to understand how science works and how to interpret information.
When articles are cloaked in jargon and complex language that is oftentimes a red flag.
Sarah wants our listeners to be able to identify the various red flags to be cautious of.
The type of scientific evidence that Sarah draws on when she is writing an article is multiple disciplinary and crosses a lot of fields.
Her ability to dig into the science in the level of detail that she can is an entire career’s worth of training.
This is a skill that takes years to acquire and is very challenging to develop without rigorous training and experience.
However, Sarah does not think it is necessary for everyone to have this skill.
She is hoping to give listeners a broad enough understanding of what scientific evidence actually is to be able to detect bull feces on the internet.
The Whole View Philosophy
We try to always be honest and upfront about when we have the science to show and what we can try to guess from a hypothesis of that science. (7:51)
Stacy knows that we have talked about what some of these terms mean on the show before.
However, we have been addressing a lot of different topics as we expanded to the Whole View philosophy.
We talk a lot about the importance of bio-individuality.
And you can’t learn that if you aren’t able to be open to the idea of learning as you go.
To Stacy that is what science is all about.
For her, the scientific method is always changing and learning as you have new information and staying open to that new information.
Science is all about taking the ongoing, upcoming information, and applying it in a way that helps us continue to evolve ourselves.
Stacy thinks it is good for people to be able to check their sources.
We are all perfectly capable of questioning how people reach the conclusions they are sharing in articles.
Listeners you hold us to that standard.
We want to always be a trusted source for you.
Science is the basis of everything we share.
What we are doing in the academic community is using the scientific method to expand human knowledge.
However, this is not a straight line.
Nutritional Sciences
Sarah shared her thoughts on the field of nutritional sciences, which is a young scientific field. (16:09)
Because of this, we are in a renaissance in terms of the expansion of our understanding.
We are at a point where there are big holes in our basic knowledge of nutritional sciences.
Sarah shared some examples.
It is important to understand that because of where we are in this field, we are in this phase where we are trying to lay the groundwork.
Sarah recommends Death by Food Pyramid.
She took a moment to also talk about the way this industry is not rooted simply in the science, that it is heavily rooted in profit.
We are also talking about a field of science where a lot of the people who are communicators of this research have a lot of bias.
Knowing what a communicator’s bias is helps you to hear their information objectively.
Removing Bias
Scientists in general try to be aware of their biases.
We try to devise experiments where the biases have no opportunity to affect the results of the experiment.
Sarah shared the ways in which this can be achieved.
You are looking for as many qualitative measurements as possible, as opposed to quantitative or semi-quantitative.
Sarah shared examples of how these terms play out in studies.
In science, we look for as many ways to blind or control so that the bias cannot influence the data that is being collected.
The data is then analyzed and as you draw the conclusions that is where bias comes back in.
However, now it comes in the form of the expectations that are shared in the hypothesis.
The Scientific Method
With the scientific method, you make observations, you collect information, and you ask a question.
You are basically interested in looking at something that nobody has looked at before.
The goal is to look at the scientific literature around that question and use it to formulate what would be an educated hypothesis.
We can look at everything we know about the system and make a guess about what we think is going to happen.
You then design an experiment that will help to answer that hypothesis.
And then you conduct the experiment in as objective of a way as possible.
You analyze the results and draw conclusions.
And then you report your findings.
Often, when you get to that piece of sending the study to journals, you then go through peer reviews.
During peer reviews, a scientific article is typically reviewed by three experts in the field.
As an expert in the field, you bring everything you know to looking at this paper.
You read through it very carefully and you try to identify any methodological flaws and alternate interpretation for the data.
There are three ways you can send it back.
The first is that you can say it is accepted as is.
The second response can be accepted as is with revisions, ranging from minor to major.
Or you can reject the paper.
The review process is all done blinded.
It passes three researchers in the field and the editor, and then it can get published.
The peer-review process is very important because it allows fresh eyes to look for any weakness in the paper.
It is because of peer review that there is such a small fraction of papers that ever get retracted, and it works out to be less than 1 in 1,000.
They are often retracted because of the discovery of a mistake.
The person who is performing this research is just interested in expanding human knowledge, and that is why we see so few examples of fraud or bias to the point of needing to retract a paper.
What to Look For
All scientific papers require a disclosure of competing interests, which is a disclosure of funding. (38:39)
They also have to disclose any potential affiliation that could introduce bias into the study.
There is always a statement, usually towards the end, before the citations.
This is always an important thing to keep in mind.
The affiliations of the authors are also important details to look for.
Just because there is a financial disclosure that is linked to industry, does not mean that the research is biased.
The peer-review process is still the main thing that is supposed to detect bias.
Sarah thinks that the most important thing to look at is the body of scientific literature as a whole.
In the alternative health communities, Sarah thinks that there is a tendency to look at a body of scientific literature and to simplify the findings on the one paper that shows something different.
A lot of the interpretation that we get is either ‘this one paper is the truth’ or you get a dismissal.
Neither one of these approaches is the right way to look at that one paper that shows something different.
You want to look for the consensus.
And you want to look for what the majority of the data is pointing to.
Scientific consensus happens when you have enough data that the vast majority of experts looking at that data come to the same conclusion.
It doesn’t mean that every single study ever done supports that consensus view or that every scientist looking at the body of scientific literature comes to the same conclusion.
However, it means that the vast majority of the data aligns with that explanation and that the vast majority of experts looking at the data agree.
A hypothesis is a possible explanation or a preliminary conclusion or an educated guess.
A theory is when the evidence builds up so much so that you have this scientific consensus and you can start to predict based on the theorem.
Sarah shared an example of how this plays out using the theory of gravity.
Types of Studies – Anecdotal & Expert Opinions
The lowest strength of evidence is anecdotes. (47:34)
“Expert opinions” fall into this realm as well.
This is basically, I’m looking around and not necessarily measuring anything.
It also reflects personal experience and is not necessarily representative.
This is what we might call a handwave explanation.
It is not meaningless.
However, it should be viewed as an indication that there might be something interesting to explore there.
There is nothing about an anecdote or expert opinion that is proof.
It is really the formulation of the initial question.
Types of Studies – Case Reports & Case Studies
From there we can get into studies that start to solidify the question and the need to answer this question and expand on our understanding of that answer. (48:31)
The next level is either case studies or case series.
These are purely observational studies.
A case study is typically an account of something that happens to one person.
And a case series is a group.
It is an anecdote with measurements.
Sarah mentioned her anti-keto stance and said listeners can find more on that here.
Types of Studies – Case-Control Studies
The next improvement in terms of strength of evidence is case-control studies. (51:26)
These are retrospective, which means looking back.
However, you have two groups within the study to compare against.
They then trackback to determine an attribute or exposure that could have caused this.
These studies show correlation, but it is hard to prove causation.
Types of Studies – Cohort Studies
From there we can go to prospective studies, which are called cohort studies. (52:44)
This is where you take a group of people and you follow them over time.
So instead of comparing diabetic and non-diabetic people, you take a pile of people and see which ones develop diabetes.
These are much stronger studies because there is much less room for selection bias.
This study provides a much more rigorous data set.
You have a lot less challenge with selection bias.
These studies show correlation, but it is hard to prove causation.
Types of Studies – Mechanistic Studies
Sarah places a high value on mechanistic studies. (54:46)
These are mostly animal studies and cell culture studies.
With these, you are trying to explain how something is linked.
They provide mechanistic insight and improves understanding of an effect.
It also proves causation by explaining why/how an effect is seen.
Subsequent human trials are required to verify the predicted effects of the intervention.
Types of Studies – Randomized Control Trials
This is considered the gold standard. (56:27)
Subjects are randomly assigned to a test group, which receives the treatment, or a control group, which commonly receives a placebo.
You can then design these to control for different types of bias.
“Cross-over” trials: participants switch from control to treatment, or from treatment to control, groups half-way through the trial.
“Blind” trials: participants do not know which group they are in.
“Double-blind” trials: neither the participants nor the experimenters know which group the participant is in.
Blinding trials helps to remove subconscious bias.
Sarah shared other ways you can remove bias.
You still need to combine the data from a randomized control trial, which is the intervention.
When you have enough information to make a prediction about what is going to happen, you run the intervention to see if your prediction is correct.
However, you still need the mechanistic studies to support that data because they explain why.
Types of Studies – Meta-Analyses
From there that are meta-analyses. (1:01:27)
This is where we pool together the data from multiple randomized control trials and look at a much bigger data set.
And this is a really great way to look at a bigger body of evidence to see if an effect still happens even with all these differences.
Looking at a meta-analyses helps us determine whether a difference is based on how a trial was set up versus a true difference.
These are really important types of studies to do.
Types of Studies – Systematic Review
The height of scientific evidence is called a systematic review. (1:02:22)
This is where you go through meta-analyses all the way through the randomized control trials, you look at the cohort studies, and then you look at all the mechanistic studies.
From there you are able to say, ah-ha, here is what this huge body of scientific evidence tells us, and here is the explanation.
This is again something that Sarah puts a lot of stock into as she is going through something.
They take into consideration the quality of the studies included.
Reviews can help mitigate bias in individual studies and give us a more complete picture.
Lastly, they are the best form of evidence, and either develop consensus or indicate the need for further studies.
How to be Objective
The main takeaway from going through those types of studies is to understand that none of them by themselves is the proof. (1:04:42)
The correlation does not equal causation.
Sarah has a hard time when people approach research with a conspiracy theory mindset and think that the one study that shows the difference must be the truth.
When we ignore the 90% of studies that show the other thing.
Sarah also gets really upset over dismissing that one study because it shows something different.
What it shows is complexity.
To Sarah, she wants to understand why that one study shows something different than the rest.
So that is another thing to look out for.
Avoid cherry-picking and dismissing research.
Science is the pursuit of truth, it is not the pursuit of backing up your opinion.
If it doesn’t back up your opinion, change your opinion. That is the scientific method.
The other piece that upsets Sarah is when she sees the dismissal of papers because they were done in a small number of people.
This is the last piece of the thread to pull in here.
Statistical power is related to the standard deviation and the magnitude of the effect.
Sarah explained this in greater detail and why it is a key detail in this all.
Scientists are trained to answer the question with as little use of resources as possible.
This means as small a sample size as possible, and this is because no matter what type of experiment you are doing, increasing the sample number has costs associated with it.
Scientists are trained to do the minimum number of experiments to have statistical significance.
Understanding statistical power is the most important because it is about understanding the magnitude of the effect and how to measure that with confidence.
Not every study needs to be done in 200,000 people to be relevant.
The thing to look for is to look at those P values and look at whether or not that data reaches statistical significance.
The Takeaway
For our listeners, the biggest thing that we want you to take away from this discussion is understanding that if a scientific study is worthy of dismissal, then it is a flaw that is going to get that study retracted. (1:13:48)
The scientific community is really good at policing itself for quality.
Don’t let somebody without a science background who is selling you some kind of supplement tell you that the paper being looked at is irrelevant or wrong.
It is not, because if it is it will get retracted.
We have all these different ways in the alternative health community to dismiss science.
The problem with that is when science is really important and relevant, we are basically training ourselves to be conspiracy theorists.
We are training ourselves to dismiss science.
It becomes a slippery slope when we find excuses to dismiss papers, making data meaningless.
When presented with conflicting evidence, if you see dismissal of that evidence, a red flag should go up.
The building of scientific evidence is what is important.
With an open mind to what conflicting data can actually reveal about a system – that is where we can be informed by the science in a way that improves our lives.
At the end of the day, science is all about improving our lives by increasing human knowledge.
Stacy noted that we need to be mindful that this information is complex and we need to be asking if this is coming from a credible source.
For Stacy, she personally looks to review a summary or an abstract and checks the sources from there for any possible flags.
Sarah took a moment to share on intervention studies.
For Special Consideration
There is a type of animal study that is called an intervention study that is called an intervention study. (1:20:20)
This is like the randomized control trials that we do in humans, but we are doing it in animals.
And this is used a lot in drug development and vaccine development.
This has a different weight to a mechanistic study.
You cannot necessarily draw a straight line between data found in an animal study and what to expect in humans.
A mechanistic study is about understanding the biochemistry – understanding how something happens.
Where an intervention study is about measuring the magnitude of effect from this manipulation point.
This is where you do need bigger sample sizes.
And you can’t say that because something worked in rats, it will work in humans.
Another thing to look for in smaller studies is the definition of the population that is included.
Sometimes that does mean that you take the information with a grain of salt before we start expanding that information to the general population.
When it comes to vaccine development, one of the challenges is that we want to give it to the entire population.
We need studies that show the safety profile in a heterogeneous population, people with genetic differences, and different conditions.
A study also needs a bigger population to determine if those adverse reactions are happening with this vaccine and the frequency of them.
Having those large sample sizes for an intervention trial is about having a sample that represents the general population so that we can identify efficacy as well as safety.
This is the type of study where you want a really big population.
When a covid vaccine is available, we will dedicate an entire episode to this topic and Sarah will dig in on the science.
We have reached a point where science has become politized and it shouldn’t be.
Closing Thoughts
Context is important. (1:27:31)
It is very easy to cherry-pick a quote that someone says and apply it to completely different circumstances.
We are talking about not cherry-picking information, and continuing to educate ourselves.
It is important to look for opportunities to continue learning and expand our understanding of what works.
If you here us have a show about something that contradicts your current opinions/understandings, don’t just listen to twenty minutes of the science and skip past the conclusion.
When you just read/listen to a certain part, or skim the information, you will miss important pieces of the whole.
We do our absolute best to always be broad in terms of what we are talking about.
Stacy shared an example of how this plays out with research on red wine and how this plays out in a large sample size.
It is not fair for us to boil down the complexity of science to a soundbite or a sentence without context.
We appreciate your patience and understanding as we try to navigate and boil oceans sometimes on this show.
Our goal is to keep shows to under an hour, and we are rarely able to do it.
We want to make sure that we aren’t coming out with a show and sharing simple bullet points on the subject.
Sarah shared her thoughts on how our society’s current health standards have shifted the way that we seek information.
It is important to be cautious of predatory marketing practices.
Being willing to revisit recommendations and have a new conversation about something is the scientific method.
It is different than being wrong before – iterating on human knowledge is science.
Sarah shared her excitement for where we are currently at with nutritional sciences.
The reality is that nutritional sciences is young enough that right now we are drawing the best conclusions that we can with the date we have.
However, there is a lot of unanswered questions that science needs to answer for us.
Stacy found it fascinating to look at the relative newness of nutritional sciences when compared to other areas of research.
This is what we are here to do for you.
Sometimes we might find something that contradicts something that we said before.
What Stacy loves is that we are willing to say, well, this is new and it is time to revisit this topic.
You do the best that you can with the knowledge that you have at the time.
The responsibility that Stacy feels we all need to have is to continue to learn so that that knowledge doesn’t stay static.
This is where science is magic.
Sarah hopes that this was helpful for our listeners.
We would love to hear your follow up questions, which you can submit via social media or the contact forms on both of Stacy and Sarah’s sites.
Thank you for listening!
We will be back again next week and we will strive to make it a shorter show! (1:39:39)
