QS 101: The Science of Self Experimentation
Ernesto Ramirez
May 3, 2012
This special guest post in our ongoing QS 101 series comes to us from Dan Gartenberg, our great QS-Washington DC meetup organizer, and his fellow graduate students in the Human Factors and Applied Cognition program at George Mason University.
Turning Scientific Concerns into Strengths for Quantified Self Experimentation
By Dan Gartenberg, Ewart de Visser, and Jonathan Strohl
Quantified Self and Science are not oil and water. They are intertwined with one another and have a long history together. Though some scientists may not hold QS in high regard, and have the following claims:
“these studies lack validity!”
“a study of a single individual will not generalize to the broader population”
“the possibility of experimenter bias makes your findings highly suspicious and inconclusive.”
“is your effect even real?”
While these are legitimate concerns, QS is science. And if we keep in mind the scientific method when conducting QS research, this strengthens the validity of our QS projects.
History speaks for itself. QS studies are actually in line with an age-old scientific tradition: The n=1 study. Back in the day, scientists did not have large labs and took to experimenting on themselves. For example, Hermann Ebbinghaus, one of the first cognitive psychologists, conducted experiments on himself to reveal the process of learning and forgetting. As a scientist he used a level of rigor that was expected of scientists at the time, and more importantly, Eddinghaus contemplated reasonable mechanisms that explained his results. Science gives us the tools to make precise measurements, and QS, with its emphasis on improvement of the self, provides a social framework for people to discuss novel phenomena. In this article we demonstrate how science lends itself to QS and how the scientific method provides us with useful tools for self-discovery. We first recommend a framework for conducting QS experiments and then discuss the scientific methods to keep in mind.
1) Achieve your goal: Unlike most experimental research, in QS our main objective is more often than not self improvement. A frequently used approach to improve yourself is by throwing the kitchen sink at the problem until you get the sought after effect. When we make this process social, we can then discuss with others what they are doing and how they think they are being affected by what they are doing. Based on this information, we get a better understanding of how to most effectively modify our behaviors for the desired outcome.
2) Use a simple design. When presenting your data the scientific establishment might criticize your conclusions because it was not the gold standard “double blind randomized control trial.” But running the right type of design isn’t the be-all-end-all of good science. If you see a difference and have a reasonable mechanism that explains the difference, with no viable alternative explanations – you are solid.
3) Stats don’t matter as much. Just graph it! One of the biggest sources of confusion is how to analyze QS data and knowing the right stats to run. But statistics are only really useful when predicting small effects or for more complex prediction models. In QS, any change is usually meaningful. For example, if you are tracking your mood and you see a small improvement it is likely meaningful to you. So don’t worry too much about statistics.
To discuss the roles of QS and science, we’ll use a dataset that we generated as a case study. After reading the 4 Hour Body by Tim Ferris, three friends all had the same goal of losing weight. None of us were extremely overweight at the time, but we could all stand to lose about 10-pounds. This inspired us to create the 10-Pound Challenge, where we competed to lose weight and either did a slow carb diet or a low carb diet. We then weighed ourselves every morning.
Quantified Friends: The 10-Pound Challenge:
Here are some issues and concepts that you should consider when making sense of your findings and how Science and QS can benefit one another:
| Threat to validity | Definition | 10-pound application |
| Mortality | When your manipulation affects the likelihood of whether or not you respond to the measure of interest (i.e. you don’t respond to a survey out of embarrassment). | There are almost no skipped days over the course of our study. This demonstrates how QS can be used as a way to address the problem of mortality by making data collection more social. This makes people more accountable and motivated to input their data. |
| History | When external events from the environment impact the variables of interest. | The diet was made social when we shared our progress with one another. This socialization, where we competed to lose weight, may explain our progress. We knew that this resulted in alternative explanations, but it didn’t matter because we simply wanted to lose weight and were pulling out all the stops to help us reach our goal. |
| Maturation | When over the course of a study you have changed in other ways that have confounded the impact of the variables in your experiment. | In QS our goal is to actually change and mature. In our QS project the intention was to lose weight and the mechanism was only as important as it was necessary to understand and use in order to promote our increased weight loss. |
| Treatment Fidelity | In science we usually compare a treatment group to a control group, but what if the treatment is not much different from the control? (i.e. there is not a good counterfactual). | We administered relative treatments based on each of our unique situations (this is a common issue with QS). For example, one of us already had a relatively strict diet. He was able to make precise modifications to his diet in order to promote weight loss, whereas; the other two QSers made broader changes. This prevented us from making precise claims about how the diet affected weight loss. Though we still got the general idea that the diet worked. |
| Treatment Interaction | When the variable that you manipulate interacts with other variables that explain the outcome. | When undertaking the 10-pound challenge we frequently told people about the challenge. This in turn made us more accountable for what we consumed due to social pressure. In this example, the response to the treatment interacted with the social environment in a way that made us consume healthier meals and lose weight. Since in QS we are not as fixated on control, we can see how these interactions unfold in the environment and discuss them with others in order to confirm or deny our intuitions. This provides us with ways to explore new ideas and mechanisms. |
| Compensatory Rivalry | When the control group is aware that they are not getting the treatment and in turn seeks out other alternatives. | In the case of the 10-pound challenge, there was no control group. Control groups do not play a large role in QS because of the focus on self-improvement. This is an issue that can be addressed by the scientific method of an A-B-A design where the QSer acts as their own control group. |
| Regression Towards the Mean | This intuitive premise from science is the basic idea that at the extremes of a behavior you are increasingly likely to gravitate towards the mean. | QSers should be particularly sensitive to this because people frequently try to improve on a behavior when they are at an all time low or an all time high. |
| Reactivity | When your response is affected by external factors, for example, social desirability. | In QS, reactivity can actually be used to improve upon outcomes. In our example, there was a social desirability to discuss what was and was not working. At one point the team members independently agreed that eating too many slow-carbs (legumes) was hindering their progress. We then made the appropriate changes to our behaviors and found increased improvements. |
And these are just some of the threats to validity that QSers can consider to improve their projects. So look out for more to come!
We would like to thank Dr. Patrick McKnight and the MRES group (http://mres.gmu.edu) for providing helpful insights on our QS project.
