Azure Grant & Gary Wolf

August 29, 2019

Cite this paper as:

Grant, Azure and Gary Wolf. “Design and Implementation of Participant-Led Research in the Quantified Self Community.” Quantified Self. https://quantifiedself.com/white-paper-design-and-implementation-of-participant-led-research/.

Introduction

Participant-led research (PLR) is a collaborative form of investigation in which researchers and participants are the same individuals. PLR attempts to render investigation equitable and transparent: professional researchers collect personal data in addition to typical research duties, and participants take on organizational, ethical and scientific responsibilities.

In this white paper, we document our experiences organizing and participating in Blood Testers, a PLR project designed to explore what we could learn from frequent measurement of cholesterol and triglycerides. We hope to contribute to the development of best practices for conducting PLR as well as offer a deeper understanding of its benefits and challenges. Finally, this paper contains appendices describing, in detail, the process of Blood Testers, which we invite others to use, challenge, and adapt. The authors also welcome comment via email.

Background

The Development of Participant-Led Research: Beyond Citizen Science

Forms of scientific research have diversified greatly in the past decades, with new approaches expanding both the kinds of researchers who practice science and the structure of the scientific research process. This diversification is driven in part by new information technologies and concomitant ideas of democratizing the scientific process.^[1]

Proponents of increasing public participation in science include, but are not limited to, self-trackers, community health workers, crowdsourced and National Institutes of Health (NIH)/ National Science Foundation (NSF)-funded researchers, leaders of citizen science initiatives, and physicians focusing on N-of-1 research. For example, on citizen science platforms like Zooniverse and Citizen Science Alliance, individuals may contribute to hypothesis development, study design, data collection, data analysis, or dissemination of results.^[2]

Additionally, a few medical centers (including University of California, Davis; University of Alberta; and University of Queensland) have adopted N-of-1 experimentation to personalize treatment plans for patients. In these scenarios, a physician and patient work together to design an experiment that can evaluate the effect of a treatment for that patient’s condition.^[3]  In academic contexts, both community-based participatory research and patient-centered outcomes research help their respective participants work with scientists to shape the research questions most relevant to their needs.^[4] In these scenarios, teams of trained researchers enlist members of the public, subjects, or patients to engage in some, but not all, parts of the research process.

Although these initiatives provide opportunities for individuals to contribute to science, all fall short of equalizing the relationship of subject and scientist. On the continuum of democratization, PLR offers “the most radical form of public participation in science,”^[5] facilitating participant direction of all parts of the research process.^[6] In a definition of PLR offered by Vayena and Tasioulas:

[PLR is an] activity that characteristically aims at the socially valued goal of producing generalizable health knowledge…It is distinctive as being initiated and conducted by the participants themselves. PLR includes individuals interested in acquiring health information, whether about themselves or more generally.^[7]

Common reasons for engaging in PLR include gaining knowledge and support from others dealing with a common health condition; contributing to the creation of useful tools; and, in our case, learning more effectively about oneself via self-observation.^[8] Despite its potential to contribute to the scientific literature, PLR publication is infrequent even within the family of citizen science.^[9]

Each of the forms of participation in science discussed above overlap with an age-old practice in which scientist and subject are the same person: self-experimentation. Santorio Santorii famously studied his metabolism by measuring the weight of his own body and excreta in the early 17th century, Isaac Newton inserted a needle behind his eye to locate his optic nerve, and mid-20th century cardiologists inserted experimental cardiac catheters into their own veins. These are merely a few of the more familiar examples of researchers pursing knowledge by studying themselves.

Self-study is more common in the behavioral sciences than in physiological research, perhaps because interventions are understood to be less risky. Professor Allen Neuringer and his students at Reed College have conducted hundreds of research projects involving self-tracking and self-experiments on topics ranging from learning and memory to mood and digestion.^[10] The results of these experiments are mostly recorded in unpublished student reports; however, Neuringer referred to a number of them in his groundbreaking paper: “Self-Experimentation: A Call for Change.” ^[11] The paper argued that even the simplest of self-experiments can make a positive impact on the individual and the scientific community.

Participant-Led Research in the Quantified Self Community: Blood Testers

Origins

Quantified Self (QS) is a global community of individuals united by an interest in self-observation and self-experimentation. Over its 11-year history, it has grown into a community of thousands of participants around the world. The Blood Testers project grew out of years of informal discussion in the QS community about the reliability and validity of various biometric assays in general and the meaning and value of home cholesterol tests in particular. Some of the questions that interested us included:

• How easy would it be to do common blood lipid tests more frequently at home?
• How do diet and exercise affect cholesterol on short time scales?

Building on community expertise in self-measurement, long time QS organizers and participants Gary Wolf, Martijn de Groot, and Bob Troia explored the possibility of providing a group of community members with the measurement equipment and infrastructure necessary to carry out interesting personal experiments.

Blood lipids were the measure of interest offered to potential collaborators, as high levels of LDL-cholesterol and triglycerides are commonly used as primary risk indicators for cardiovascular disease: the number one killer in the world.^[12] At-home lipid assay became possible in the early 2000s, allowing for easier self-measurement.^[13] The idea was presented at the 2017 QS Conference in Amsterdam, where an open discussion was convened to explore questions attendees had about blood lipid measurement.

Purpose and Innovation

Blood Testers aimed to develop a process for conducting small-group research that addressed participant-generated questions outside an academic institution. For this reason, traditional research methods were augmented to suit the needs of an international group of researchers, engineers, writers, and entrepreneurs.  Responsibility was shared among the participants in all aspects of investigation, save the original selection of the output to be measured, which was selected by a small group of organizers before obtaining funding for the project. Collaborative responsibilities included hypothesis generation, study design, data collection, analysis, and reporting.

Participants engaged in active discussion of risks and benefits of participation prior to, throughout the duration, and at the conclusion of the project. Although this was a group activity with a general, collective goal of learning from high-frequency blood tests of lipids, each participant also developed an individual research question. Participants collected their own data and designed their own experiments, with help from professional researchers among the participants as needed.

Participant-organizers, who provided organizational leadership and research support, also posed their own questions and collected their own data. All participants and participant-organizers subsequently collected and analyzed blood cholesterol and triglycerides as often as once per hour using a commercially available blood lipid testing system. Participants combined this data with their own annotation, additional blood marker data, and wearable data as suited their individual projects.

Preliminary Research

Background research was conducted by Bob Troia, an early participant in the Blood Testers project. His extensive work categorizing extant technology for lipid measurement can be found here. Additionally, Participant-Organizer Azure Grant conducted an internal literature review on the physiology of blood lipids and their changes over short timescales. This review and larger reference list can be found here.

Funding

Funding for the project was obtained from Amgen Inc.’s Customer Experience Division. The project was conceived of by Gary Wolf and proposed to the sponsor as an opportunity to investigate what everyday people could learn from testing personal hypotheses surrounding lipid data (e.g., measures of one’s cholesterol or triglycerides), with resources and training provided by QS and collaborators in academia. As Amgen Inc. produces and markets a lipid-lowering drug, the company has a broad interest in understanding issues and prospects surrounding the emerging practices of self-measurement. The funders did not review the questions asked during the project, the data collected, the manuscripts submitted for publication, nor the participant talks and blogs prior to their dissemination.

Recruitment: Combining the Role of Subject and Scientest

In a typical study, subjects are recruited from student or patient populations of interest and remain distinct from the researchers conducting the experiment. In Blood Testers, however, each active advisor or researcher also participated in the self-study, made observations, and optionally contributed to project leadership by taking on additional project responsibilities. For example, participants led discussions on the risks and benefits of participation as well as provided reviews of lipid physiology research and available technology relevant to the project. In this way, individuals in the study “wore many hats,” and the choices made in terms of project language and requirements aimed to flatten the typical hierarchy among subjects and scientists—both referred to throughout the project as “participants.”

The participant-organizers who initiated the project began recruitment during an information session and discussion at the 2017 QS Global Conference in Amsterdam. This hour-long discussion explained the project concept, invited suggestions for development and feedback from attendees, and gave attendees the opportunity to follow up with participant-organizers to join the project.

Who Were the Blood Testers and What Were Our Roles?

The table below provides a brief description of the individuals who contributed to Blood Testers and their roles. These roles give an idea of the mixture of contributions provided by the participants, and are meant to succinctly convey some but not all of the ways each person shaped the project.

Table 1. Participant Roles and Occupations

Number	Name		Project Role(s)
1	Azure Grant		Participant-Organizer: data collection, led daily organization of project activities and communications; advised on experimental design; troubleshot; drafted publications
2	Bart Timmers		Participant: data collection, contributed knowledge of lipid physiology
3	Ben Best		Participant: data collection, contributed extensive experience with blood testing
4	Benjamin Smarr		Participant: data collection, contributed background on biological rhythms
5	Bob Troia		Participant-Organizer: data collection, conducted extensive research on how to self-collect lipid data
6	Boomer Anderson		Participant: data collection, contributed patient and blood testing experience
7	Camille Nebeker		Participant: contributed background information on research ethics and informed consent; co-authored project publication on governance of PLR
8	Anonymous		Participant: data collection, contributed initial background on cholesterol variability.
9	Dawn Lemanne		Participant: data collection, contributed knowledge of lipid physiology
10	Erica Tanamachi		Participant-Organizer: data collection, provided communication, production, and administrative support throughout the study
11	Gary Wolf		Co-Originator of the Project & Participant-Organizer: data collection, procured funding for the study; provided communication and organizational support throughout the study; co-authored both manuscripts from the study
12	Hannes Feistenauer		Participant: data collection, contributed to our background on the effects of different meal compositions on lipids
13	Anonymous		Participant; contributed extensive experience with blood testing and lipid variability with dietary change
14	Anonymous		Participant; contributed patient and blood testing experience
15	Jos van Dongen		Participant; contributed to our background on exercise and lipid physiology
16	Justin Lawler		Participant; contributed patient and blood testing experience
17	Laila Zemrani		Participant; contributed to our background on exercise and lipid physiology
18	Anonymous		Participant; contributed business experience & assistance developing PLR methods
19	Martijn de Groot		Participant; contributed background experience teaching the practice of self-tracking
20	Niels Bischoff		Participant; contributed to our background on the effects of meal timing on lipids
21	Rob Rothfarb		Participant; contributed extensive patient & blood testing experience, and additional device validation
22	Steven Jonas		Participant; contributed extensive self-tracking and communication experience
23	Whitney Erin Boesel		Participant; contributed extensive self-tracking & blood testing experience

Ethical Review

PLR differs enough from traditional research as to create substantial challenges to the process of ethical review. This topic is addressed in a considerable body of work detailing the difficulties Institutional Review Boards (IRBs)/Research Ethics Boards (REB) face in evaluating participatory research according to traditional academic criteria.^[14] In the early stages of Blood Testers, there was much discussion as to whether the project intended to generate traditional scientific knowledge, which we understood as generalizable health knowledge that typically requires IRB/REB oversight, or whether the project would aim to create personal health knowledge alone, which would not require IRB/REB oversight.

The group settled on a novel approach: to create both personal and generalizable knowledge and to intentionally not seek IRB/REB approval.  As all participants engaged in self-observation rather than directed manipulation by a scientist, the group felt that the very ideas that normally motivate IRB/REB review—protection of individuals from harm from researchers and protection of a university from liability in the event of harm to individuals—did not apply. Simultaneously, a large body of literature attests to the lack of engagement a typical IRB/REB review provides to the individual participants in a study.^[15]

To explore more suitable methods of ethical review, a participant and research-ethicist, Dr. Camille Nebeker, advised the group on the elements of standard ethical review.^[16] Participants then met twice as a group during the project to discuss the risks and benefits of participation and maintained a living document of this discussion. Additionally, throughout the duration of the study, participants evaluated the potential risks, benefits, and risk mitigation strategies of their experiments. At the conclusion of the project, all participants were interviewed about the participatory ethical review process.

Experimental Design: A Group-Wide Experiment and 21 Self-Experiments

Blood Testers was designed as a PLR project with a general, collective goal of learning about lipid variability via high-frequency self-measurement. Notably, the study was designed such that data sharing, in Google Drive or otherwise, was not a requirement for participation. In fact, the only requirement for participation was an active interest in the topic and a willingness to provide feedback about the project to the QS participant-organizers. Almost all participants chose to share data, and the remaining few opted to contribute to the group by leading discussions or writing reports.

The group chose two simple observations for collection across interested members of the group: evaluate the range of values cholesterol and triglycerides (a) within a single 24-hour period (within-a-day) and (b) across many days fasted.

In addition, participants developed hypotheses of personal interest and conducted experiments to address them (see Table 2 for hypotheses, and “results” for completion rate).

The group-wide assessment of variability in lipid levels across a day was chosen such that the data collected in the course of personal experimentation could contribute to the collective goal. Personal questions were developed into individual protocols and sampling calendars stored in a shared Google Drive. This combination of experimental approaches resulted in (1) an easy-to-conduct, group-wide experiment and (2) participant creation of personal, single subject experiments. This combination was crucial to the success of the project in generating personal and generalizable knowledge.

Table 2. Group and Individual Hypotheses

Participant ID(s)	Hypothesis
Group Wide	Our lipids may vary significantly within-a-day.
Group Wide	Our lipids may vary significantly across mornings in the fasted state.
2 Individuals	My blood cholesterol and triglycerides may show ultradian and daily rhythms.
Individual	My lipids may cross a risk category within-a-day.
Individual	My post-prandial triglyceride rise may vary predictably based on the kind of food I eat.
Individual	My cholesterol and triglycerides may show ultradian rhythms that correlate with those in my stomach activity and body temperature.
Individual	I can use my post-prandial triglyceride responses to create a “personal lipidemic index” comparable to a glycemic index of different foods.
2 Individuals	My subjectively and/or heart rate variability (HRV)-estimated stress may correlate with my cholesterol or triglyceride levels within a day.
Individual	Taking repeated multi-time point “baselines” across different days may reveal stereotyped daily variability in my lipids.
Individual	Switching to a plant-based vegan diet may change my lipid levels within two weeks.
Individual	Natural variability in my lipids by time of morning may cause me to cross a risk category.
Individual	My daily fasting lipids and 2-hour lipid profile may change in range or shape during very low, medium low, and moderate carb diets.
2 Individuals	Running may have a short-term effect on my lipids (comparing before versus directly after a 30-, 60-, or 90-minute run).
Individual	A vegan diet may lower my total cholesterol and triglycerides over three months.
Individual	Tracking my lipids may be an effective encouragement for me to lose weight.
2 Individuals	Psychological and physical stressors (as measured subjectively and by HRV) may have distinct, measurable effects on my lipids.
Individual	My post-prandial triglyceride and cholesterol elevation may differ between days in which I eat three meals and days in which I eat only one meal.
Individual	Changing the macronutrient composition of my diet for two-week increments may affect my post-prandial and daily fasted lipid levels.
Individual	I am interested as to whether my lipids and PT/INR (a measure of blood coagulation) co-vary and if this influences the effectiveness of at home blood testing for me. Perhaps if I clot too quickly the test is ineffective.
Individual	I am interested as to whether my lipids change from before and after (a) a long walk or (b) a tai chi class.
Individual	My fasting lipids may vary predictably across my menstrual cycle.
Individual	I hypothesize that marathon training over two months will impact my cholesterol and that my cholesterol may also differently from pre- to post-run depending on run intensity.

Two example individual experiments are described, briefly, below:

• One participant, a 35-year-old, healthy male, took hourly triglyceride samples from 5:00 to 24:00 and recorded his perceived hunger on a four-point scale (i.e., not hungry, mildly hungry, moderately hungry, very hungry) at half-hour intervals. The participant remained at home sitting for much of the day, did no activities beyond light walking, and ate regular meals. Data were transferred from the measurement device into a personal spreadsheet the subsequent day. The participant found an inverse association between his triglyceride levels and his hunger that persisted even in the absence of recent food intake.
• Another participant, a 35-year-old, healthy female, took daily fasting samples between the hours of 7:00 to 9:00 on each day of her menstrual cycle. Fasting was self-reported as at least 12 hours of ingesting water only. The participant was regularly cycling and was not on birth control. The participant recapitulated, at seemingly unprecedented resolution, the menstrual rhythm of total cholesterol (TC).

Methods and Materials

Communication Structure

Blood Testers took place across six countries and nine time zones. Therefore, regular communication among participants and organizers was maintained via a series of eight 1-hour webinars held approximately every two weeks over the course of the project. These were recorded and maintained such that those who could not attend could reference the material at a later time.

Additionally, each participant met individually with the leading participant-organizer several times across the project for everything from device troubleshooting to experimental design and data analysis assistance. Other communications among participants occurred (a) via Slack, (b) via email, (c) via text, (d) via the QS website and forum, (e) via a shared Google Drive, and (f) in person, when possible.

Data Collection and Storage

Equipment was shipped globally to participants, who were trained by a QS participant-organizer on how to self-collect lipid data. Data was stored in a shared Google Sheet accessible by all participants via a private link. This data was stored for the duration of the project and deleted following submission of the manuscript unless a participant expressly wished to keep it accessible. Participants were not required to share their data among the group, but the majority opted to do so. 

All lipid data was collected using an FDA-approved, CLIA-waved finger-prick assay system: the CardioChek Plus and Full Lipid Panel test strips, which directly measure TC, HDL-C, and triglycerides. All data and sampling conditions were self-reported by participants. Participants were strongly incentivized to report data and conditions honestly as all samples were collected voluntarily based on the interest of that participant.

Note that all other measurement devices used for personal experiments were selected by individual participants, and did not involve group or organizer input unless sought by the individual.

Sampling and Data Analysis

Within-a-Day Sampling

Most participants took multiple samples within-a-day to create a dynamic baseline of expected daily variability. Prior to serial within-a-day sampling, participants maintained a stable sleep and meal schedule to the best of their ability and to choose a day for sampling on which they did not have other obligations. On days of serial within-a-day sample collection, participants refrained from strenuous exercise and ate according to their regular habits. By group agreement, participants then collected at least four samples within that day at regular intervals (i.e., 6:00, 12:00, 18:00, 24:00). At the discretion of the participant, sampling frequency was increased, up to hourly, for 24 hours. For a description of sampling frequencies utilized by different participants see our Open Science Framework page’s document “Sampling Stats by Participant”.

Repeated Morning Fasted Sampling

During 12 participants’ personal experiments, 344 total fasted morning samples were collected over a two-month period. A fasting morning sample was defined as one taken after at least 12 hours of not consuming anything except water before 12:00 local time. Note that while there was some overlap, not all individuals who took part in within-a-day sampling took part in repeated morning fasted sampling.

Data Analysis

Data were collated in Google Sheets, Microsoft Excel 2016, and Matlab 2018a. See Open Science Framework for analyses and code as well as the methods section of “Approaches to Governance of Participant-Led Research: a Qualitative Case Study” for details.^[17]  Individual participants analyzed their data in several programs, including Google Sheets, Microsoft Excel and SAS.

Blood Testers Results

Results and Publications

Results of the group-wide study, including validation information, can be found in the publication “Free-Living Humans Cross Cardiovascular Disease Risk Categories Due to Daily Rhythms in Cholesterol and Triglycerides.”^[18] Results of the ethical reflection process can be found in the publication “Approaches to Governance of Participant-Led Research: a Qualitative Case Study”.^[19] Individual projects were captured via a series of blogs and talks at the QS 2018 Cardiovascular Health Symposium, which can be found below:

Link to Laila’s Blog

Link to Bob’s Blog

Link to Justin’s Blog

Link to QSCVD Talks

Retention

During the eight months the PLR took place, 88% of the starting cohort completed a project. Approximately 70% of participants were able to attend a Quantified Self Cardiovascular Health Symposium at UC San Diego, where participants shared their results via on-stage talks and discussions.

What Blood Testers Taught Us About the Process of Participant-Led Research and Self-Collected Data

Contrary to traditional study designs, this work did not exclusively rely on a team made up of academic researchers managing the participation of untrained subjects. Instead, Blood Testers utilized the skills of a variety of individuals in research, academic and non-academic communication, engineering, and data science as well as those with personal experience with self-tracking and cardiovascular illness. The project left us with many benefits, challenges, and questions.

Below, we offer a framework in which individuals of diverse professions can work together to gain individual knowledge and contribute to the scientific literature.

1. 1. Combining the role of researcher and participant encourages commitment to the project. By beginning a project as participants first, researchers/administrators second, individuals in the project took a common stake in its outcomes for both personal and altruistic reasons.^[20]
2. 2. Substantial, longitudinal learning was required for participants to conduct a rigorous self-experiment. This was true even of experienced self-trackers. For example, although individuals were interested in cholesterol when they signed on, many realized it required substantial study time to learn background material on the physiological functions of cholesterol.
3. 3. Choosing to participate is personal. Some individuals had negative past experiences with large lancets (e.g., having to use them to check blood sugar), and this was a deterrent from joining the project.
4. 4. Blood Testers required substantial time commitment. This included ~12 hours of meeting, training and discussion, in addition to the time taken personally to design and execute an experiment. Three individuals wanted to participate but, despite their best intentions, were not able to make time to participate.
5. 5. Organizing a participant-led project requires close attention and flexibility in communication. We found that individuals who communicated more during the project were more likely to draw both personally and generally valuable conclusions from their projects. Individuals are (a) distributed and (b) operating with a high degree of autonomy, making consistent and adaptive communication essential.
6. 6. Physiological measures within an individual vary significantly and rhythmically over time, and these trends require careful analysis to interpret.^[21] Collection of high frequency physiological measures brings up fundamental questions about individual variability and biological rhythmicity. This brings up further questions about the proper interpretation of point measurements in many outputs (e.g., glucose, blood pressure, weight/body mass index).
7. 7. Approaching causality is very difficult in participant-led research; nonetheless, much personal learning occurs. Often, the variability observed among the first few trials of a personal experiment is sufficient to inspire the participant to shift their line of questioning, ultimately leading to an interesting—if not causal—personal takeaway.
8. 8. Personal data can be more difficult to interpret than impersonal data, but collaboration can provide balance. Reasoning “objectively” about oneself is hard. This is an ongoing challenge, as even trained scientists may struggle in choosing and applying appropriate methods to the analysis of self-collected data. For instance, interpreting observations of variability in blood biomarkers is not intuitive; a tendency to interpret every rise and fall is easy to slip into, even for experienced self-trackers. Although this bias has led to the removal of self-experimentation from much scientific work, we learned that collaboration among participants & organizers can help reduce such bias (see 11). 
9. 9. Publishing this work was a non-trivial process. Despite our early intention to not interact with the IRB/REB process, publication required obtaining an IRB/REB exemption. Even preprint servers require proof of ethical review. How can future PLR projects taking place outside academic institutions share generalizable findings when there is no free access to ethical review?
10. 10. The best way to disseminate this work is not known. This work was shared via scientific publication, conference talk, meetup, and blog. What is the most widely useful artifact of such a project? Is there a discrete, shareable unit of knowledge appropriate to PLR? We expect that something akin to a community database and “charticle” or Wikipedia page can fill this role eventually, allowing both professional researchers and academic individuals to share the same knowledge base—but this too remains an area for experimentation.
11. 11. Self-collected, high temporal resolution data has both personal and generalizable value. Self-collected data refers to numerical or categorical information obtained by an individual about themselves, either by use of their own senses or through use of technology, as in the lipid samples collected during Blood Testers’ self experiments. Self-collected data has several advantages from a research perspective over traditional assays conducted by laboratory staff including high temporal resolution, ease of acquisition/scalability, and detailed individual annotation. Self-collected data also has associated challenges, largely due to skepticism (both real and unwarranted) about data quality and management as well as the complexity of measures that can be captured. Blood Testers showed us that these challenges can be overcome by bringing together diverse expertise.

Benefits of Participant-Led Research

Participant-led research, by definition, exists in service of both the participant and general scientific knowledge. The “human right to science” guaranteed in Article 27 of the Universal Declaration of Human Rights is brought to life in PLR.^[22] The fruits of discovery are broadly shared, and the learning associated with the research process itself accrues to the benefit of all involved. We noted a number of specific benefits of our particular project:

1. 1. Participants gained conceptual and practical skills relating to data interpretation. By collaborating with fellow participants who were researchers by profession, other participants gained exposure to concepts in statistics, programming, and interpretation of data.  This included skills relating to reading new types of graphs, normal and non-normal distributions, field-specific knowledge in cardiovascular physiology, and the concept of static measures versus time series. These skills are not usually acquired without a specialized degree yet are valuable skills for decision making in a data-rich world.
2. 2. Participation encouraged active reasoning. Conducting a personal experiment is a way to question one’s assumptions in a non-judgmental and curious manner. This sort of reflection has been reported to promote self-awareness, patience, and curiosity—as attested to in the Blood Testers project and in meta analyses of the practice of self-tracking.^[23] For example, validation of self-collected measurements involved careful reflection on the context of the measurements and the capabilities of the instrumentation. These issues are common in scientific and medical practice but are normally engaged directly only by specialists. Additionally, participant presentations were occasions for developing new knowledge and skills.
3. 3. Learning was motivated by personal interest. Domain knowledge about cardiovascular health, risk, lipid metabolism and circadian rhythms was pursued in the context of personal health questions. An abstract idea of questionable personal relevance, such as a daily rhythm in cholesterol, became worthy of reflection when needed to explain variation in one’s own data. For some participants, this approach to learning about cholesterol and circadian rhythms was a key benefit.
4. 4. Time lag between research and dissemination was minimized. In contrast to traditional approaches to research participation, in which fruits of discovery are shared with participants after a long delay, the learning offered by PLR is ongoing; it begins prior to the experiment with the sharing of motivations and potential study designs, and precedes publication via blog post, show-and-tell talk, forum discussion and personal implementation of experimental learnings.

Academic and Clinical Significance of Participant-Led Research

The QS community shares an interest in using empirical observation to explore personal questions. In this approach, self-collected time series are essential as they represent the self-investigator’s dynamic physiology. As nearly all biological systems are dynamic (e.g., ultradian, circadian, ovulatory, and seasonal rhythms), the time series data of interest to QS participants also hold interest to academic researchers. These data augment existing knowledge based on static measures with observations that are otherwise difficult or impossible to collect.

Access to within-individual time series is a significant barrier to exploring longitudinal, within-individual physiology. For traditional researchers, conducting large-scale human studies is costly, time consuming, and often creates siloed (rather than shared) data. PLR may be useful in addressing these challenges both by offering (1) the possibility of collaboration with self-trackers willing to take part in well-defined projects that collect time series data and (2) the possibility of collaboration with participants who have already collected extensive time series. Collaboration between the QS community; researchers in physiology, medicine, and chronobiology; and data scientists can generate novel insights into the dynamics of human physiology. However, for this potential to be realized, significant challenges in funding, instrumentation, data access, ethical review, lay education, and research dissemination must be addressed.

Road Map for Participant-Led Research

The QS Blood Testers project was a relatively simple experiment in PLR that reveals future prospects and challenges. We propose the following areas of work for stakeholders in the advancement of PLR:  ollaborations across technical sectors to build an open technology stack for PLR, including open instrumentation that allows secure data access and to participants.

1. 1. Collaborations across technical sectors to build an open technology stack for PLR, including open instrumentation that allows secure data access and control to participants.
2. 2. Development of training and educational materials that enable more people to do rewarding and productive self-experiments.
3. 3. Collaboration with data science and physiology specialists to develop shareable methods and frameworks, including data pipelines, suited to statistical and time series analysis of personal data.
4. 4. Partnership with open access movement to ensure participants’ access to personal data and relevant scientific literature.
5. 5. Creation of suitable frameworks for ethical review, allowing participants to work with academic and clinical partners who require traditional IRB/REB review to publish.
6. 6. Contribute to the development of a dissemination system for PLR that creates accessible alternatives to traditional disciplinary publishing, including informal publication and shared data analytical resources.

Appendix I. Under the Hood

• What made the last QS participant-led project successful?
  
  
  • Individual management of data (through personal spreadsheets and a shared Google Drive).
  • Freedom to participate as one saw fit.
  • Observational rather than interventional group experiment.
  • Collaboration among field experts (with knowledge of data science, physiology, medicine, reporting, research ethics, and engineering) and practiced self-trackers (with knowledge of capillary blood testing, wearables, and lived experience).
  • Dedicated staff for project management, leadership, data analysis, and reporting.
• Preparation for the project
  
  
  • Conduct background research
    • Identify a unifying and fairly simple research question likely to be interesting to most/all participants and easily addressable by some common data set that can be shared across participants.
    • Through preliminary dialogue with potential participants or community informants, identify subsidiary research questions that may bear upon instrumentation requirements, recruitment, and methods.
    • Pre-conditions/context: Organizers need sufficient background and familiarity with potential participants to have a general sense of the kinds of questions, instrumentation, and expertise necessary. Not all PLR is community-based, but recruitment is likely to be substantially different than for conventional human subjects research, since organizers are recruiting research collaborators as their research participants.
  • Gather a team of participant-organizers
    • This should include field experts in relevant physiology, medicine, data analysis, statistics, storytelling/reporting, self-tracking, project management, and research ethics. These individuals are ideally participants themselves, but don’t have to be.
  • Conduct instrumentation research
    • Data management: Assess data management platforms that give participants and field experts data access and allow participants to control data sharing permissions.
    • Data sources: Research what devices/tests allow data access in a usable manner (e.g., a CSV rather than a JPEG/PDF of a table). Investigate how data is actually measured if from a commercial wearable. Consider how to help future participants avoid paying too much attention to things like “sleep scores,” which may be poorly constructed but prominent in user interfaces (UIs).
  • Plan analyses
    • Identify analytical methods appropriate to proposed research questions and data. This may be general, as participant-specific questions are not known yet. Consider the temporal and spatial resolution of the data and what data structures are anticipated.
      • Is data likely to be non-normal?
      • Is rhythmic change across the day, ovulatory cycle, or seasons anticipated?
      • Are subjective ratings of symptoms likely to be involved?
    • Identify how analytical pipelines can be shared among all participants. Is it possible to use a primarily shareable platform—Jupyter Notebooks + My Binder, Coda, or even Google Drive—such that participants can run analyses without coding experience?
  • Conduct participant support and training research
    • Identify an infrastructure for participant training and communication.
      • Google Drive, Crowdcast, Slack, Zoom, etc.
    • Identify training materials (e.g., filming device use tutorials, writing instructions).
  • Identify dissemination goals
    • Identify forums for disseminating results. If possible, ask potential participants about their story-sharing/dissemination goals.
      • Past examples include BioArxiv, conference posters, QS Show&Tell talks, blogs, business-pitches, and peer-reviewed research or review publication.
    • Think about authorship for any scientific publications before beginning.
  • Identify funding sources
    • Create a project budget.
    • Identify ideal number of participants (for us, this was no more than 30).
  • Ethical review feasibility
    • Discuss potential risks and benefits of the project with a field expert.
    • Determine what type(s) of IRB/REB review will be sought and how participants will be involved in the ethical review.
  • Write proposal
    • Create a timeline of project events and minimum time commitment (identify expectations of staff and participants).
• Project commencement: data collection, analysis, and sharing loop
  
  
  • Recruitment and onboarding: PLR recruitment is likely to involve a process of exploration of potential questions of interest.
    • Recruitment event: Hold a recruitment event to gauge interest.
    • Questionnaire: Collect information on potential participants.
    • Participant interview: Host discussion with each participant to generate a research plan (i.e., a protocol to execute with timeline and goals).
    • Participant training: Engage with relevant field experts to make sure each participant has needed background knowledge to form a research question, protocol, and understand their results.
    • Ethical review: Host risk and benefit discussion with each potential participant.
    • Group-wide introduction: Host a group meeting to discuss logistics (e.g., timeline with flexible goal end date, cost, research questions, and data management).
  • Project development
    • A participant-organizer and participant meet to discuss:
      • The participants’ health history.
      • Why did you want to join this project?
      • What do you hope to accomplish?
      • What are your possible measures of success?
      • The participant’s interests and occupation (to get a sense of who they are, how they think, and their past research experience, if any).
      • Potential questions (hypotheses).
      • Participant’s knowledge base about their condition.
      • Participant’s goals for the project.
    • If participant is willing, record discussion audio and notes.
    • The goal of the session is to establish a personal relationship, begin to learn about the participant’s health status from their own perspective, assess the participant’s goals for participation, and brainstorm a few specific questions for potential investigation.
    • A goal of this, or a subsequent, session is to identify a set of measures to take as a baseline before any interventions are planned.
      • Create preliminary question and hypothesis if possible.
      • Share source material appropriate to participant (e.g., literature, video).
    • Draft participant protocol
      • Start very simple. This may look like collection of a dynamic baseline of multiple continuous metrics over a period of time (e.g., one week to two months), aiming to assess multiple relevant physiological systems (e.g., heart rate/HRV, blood glucose, body temperature, sleep timing, medication timing).
    • Supported/collaborative data collection and management (the following steps are repeated in a loop until project is done)
      • Disseminate relevant devices/supplies to participants.
      • Participants collect data and store it as they wish using the private/sharing infrastructure proposed.
      • Meetings among field experts and individual participants at regular intervals to assess progress and data.
      • Analyze data on an ongoing basis in a shareable manner.
      • Allow for dynamic, participant-led adjustment of project trajectory (research question or methods can change).
      • Emphasize forums for sharing progress with the group, participant-organizers, and/or public forum (this can be a project log at Open Science Foundation or on the QS forum project logs site).
  • Finish Projects
    • Determine data sharing status of all data.
    • Determine what has been learned or gained by the participant.
    • Determine what has been learned or gained by the assisting participant-organizers.
• Data analysis
  
  
  • A participant-organizer will spend time intensely analyzing data for within- and across-individual patterns.
  • Re-assess which research questions can be answered and if any more data needs to be collected.
  • Put together the most meaningful results.
  • The process of telling the story, and figuring out what kind of story to tell, is fundamentally different in PLR as the motivations for participating may vary among participants. Some participants, including the participant-organizers, may want to prepare the material for scientific publication; others may be more concerned with reporting to a relevant community or recording their lessons only for themselves. We can think about this as a range of opportunities to learn and to disseminate knowledge.
  • This is a responsibility to the participants and also an opportunity to learn. The structure of presentation is an inducement to realize the potential value.
• Disseminate
  
  
  • Submit for publication: Careful consideration of authorship issues/open access/time and budget requirements.
  • Online pre-publication servers (arXiv), which allow individuals to upload scientific manuscripts to the internet without peer revieew.
  • Host talks.
  • Publish blogs.
  • Create an archive of project materials and data (where permitted by participants).
  • Project Log Forum.

Appendix II. Reference Material

• Videos
• Quantified Self Cardiovascular Health Website
• Quantified Self Labs
• Blood Testers Appendix:
  
  
  • Open Science Framework Page
  • Bob Troia’s Report on Lipid Testing Technology
  • Azure Grant’s Training Video on Use of the CardioChek Plus

Bibliography

Banks, Sarah, et al. “Everyday Ethics in Community-Based Participatory Research.” Contemporary Social Science 8, no. 3 (2016): 263-277. https://doi.org/10.1080/21582041.2013.769618.

Bastianelli, Karen M., Stacey Ledin, and Jennifer Chen. “Comparing the Accuracy of 2 Point-of-Care Lipid Testing Devices.” Journal of Pharmacy Practice 30, no. 5 (2017): 490-497. https://doi.org/10.1177/0897190016651546.

Bloss, Cinnamon, et al. “Reimagining Human Research Protections for 21st Century Science.” Journal of Medical Internet Research 18, no. 12 (2016): e329. https://doi.org/10.2196/jmir.6634.

Bonney, Rick, et al. “Next Steps for Citizen Science.” Science 343, no. 6178 (2014): 1436-1437. https://doi.org/10.1126/science.1251554.

Buchanan, David R., Franklin G. Miller, and Nina Wallerstein. “Ethical Issues in Community-Based Participatory Research: Balancing Rigorous Research With Community Participation in Community Intervention Studies.” Progress in Community Health Partnerships: Research, Education, and Action 1, no. 2 (2007): 153-160. https://doi.org/10.1353/cpr.2007.0006.

Camp, G. Van. “Cardiovascular Disease Prevention.” Acta Clinica Belgica 69 (2014): 407-411. https://doi.org/10.1179/2295333714Y.0000000069.

Choe, Eun Kyoung, Bongshin Lee, and Schraefel M.C. “Characterizing Visualization Insights from Quantified Selfers’ Personal Data Presentations.” IEEE Computer Graphics and Applications 35, no. 4(2015): 28-37. https://doi.org/10.1109/MCG.2015.51.

Choe, Eun Kyoung, et al.“Semi-Automated Tracking: A Balanced Approach for Self-Monitoring Applications,” IEEE Pervasive Computing 16, no. 1 (2017): 74-78. https://doi.org/10.1109/MPRV.2017.18.

Choe, Eun Kyoung, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. “Understanding Quantified-Selfers’ Practices in Collecting and Exploring Personal Data,” One of a CHInd – Conference Proceedings, 32nd Annual ACM Conference on Human Factors in Computing Systems, 1143-1152. Toronto, ON: Association for Computing Machinery, 2014. https://doi.org/10.1145/2556288.2557372.

“Cholesterol Testing Technology Research,” Google Docs. https://docs.google.com/document/d/1YLkqd5ssaPU0CPuMeimJ9ti6_a0YT36B5rKGAyfe5vY/edit?usp=sharing.

“Citizen Science Alliance,” Citizen Science Alliance, accessed January 20, 2019, https://www.citizensciencealliance.org.

Cohn, Jeffrey P. “Citizen Science: Can Volunteers Do Real Research?” BioScience 58, no. 3 (2008): 192-197. https://doi.org/10.1641/B580303.

Dehghan, Mahshid, et al. “Associations of Fats and Carbohydrate Intake with Cardiovascular Disease and Mortality in 18 Countries from Five Continents (PURE): a Prospective Cohort Study.” The Lancet 390, no. 10107 (2017): 2050-2062. https://doi.org/10.1016/S0140-6736(17)32252-3.

Dias, Sónia and Ana Gama. “Community-Based Participatory Research in Public Health: Potentials and Challenges.” Revista Panamerica Salud Publica 35, no. 2 (2015): 150–154. https://scielosp.org/pdf/rpsp/2014.v35n2/150-154/pt.

Fleurence, Rachael, Danielle Whicher, Kelly Dunham, Jason Gerson, Robin Newhouse, and Bryan Luce. “The Patient-Centered Outcomes Research Institute’s Role in Advancing Methods for Patient-centered Outcomes Research.” Medical Care 53, no. 1 (2015): 2–8. https://doi.org/10.1097/MLR.0000000000000244.

Follett, Ria, and Vladimir Strezov. “An Analysis of Citizen Science Based Research: Usage and Publication Patterns.” PLoS ONE 10, no. 11 (2015): e0143687. https://doi.org/10.1371/journal.pone.0143687.

Gao, Ying, et al. “Study on The Reliability of CardioChek PA for Measuring Lipid Profile.” Journal of Peking University (Health Sciences) 48, no. 3 (2016): 523–528. https://doi.org/10.3969/j.issn.1671-167X.2016.03.025.

Godlee, Fiona. “Towards the Patient Revolution.” BMJ 348 (2014): g1209. https://doi.org/10.1136/bmj.g1209.

Grant, Azure and Gary Wolf. “Free-Living Humans Cross Cardiovascular Disease Risk Categories Due to Daily Rhythms in Cholesterol and Triglycerides.” Journal of Circadian Rhythms 17, no. 1 (2019): 3. https://doi.org/ 10.5334/jcr.178.

Grant, Azure, Gary Wolf, and Camille Nebeker. “Approaches to Governance of Participant-Led Research: a Qualitative Case Study.” BMJ Open 9, no. 4 (2019): e025633. https://doi.org/10.1136/bmjopen-2018-025633.

Gura, Trisha. “Citizen Science: Amateur Experts.” Nature 496, no. 7444 (2013): 259–261. https://doi.org/10.1038/nj7444-259a.

Hand, Eric. “Citizen Science: People Power.” Nature 446, no. 7307 (2010): 685-687. https://doi.org/10.1038/466685a.

Jones, Peter J. H., and Dale A. Schoeller. “Evidence for Diurnal Periodicity in Human Cholesterol Synthesis.” Journal of Lipid Research 31, no. 4 (1990): 667–673. http://www.jlr.org/content/31/4/667.short.

Khanlou, N., and E. Peter. “Participatory Action Research: Considerations for Ethical Review.” Social Science & Medicine 60, no. 10 (2005): 2333-2340. https://doi.org/10.1016/j.socscimed.2004.10.004.

Kravitz, Richard L. “Medicine, Politics, and the English Language.” Journal of General Internal Medicine 32, no. 8 (2017): 853-854. https://doi.org/10.1007/s11606-017-4081-3.

Lewis, Dana. “Setting Expectations for Successful Artificial Pancreas/Hybrid Closed Loop/Automated Insulin Delivery Adoption.” Journal of Diabetes Science and Technology 12, no. 4 (2018): 533-534. https://doi.org/10.1177/1932296817730083.

Mikesell, Lisa, Elizabeth Bromley, and Dmitry Khodyakov. “Ethical Community-Engaged Research: A Literature Review.” American Journal of Public Health 103, no. 12 (2013): e7-e14. https://doi.org/10.2105/AJPH.2013.301605.

Mirza, RD, S Punja, S Vohra, and G Guyatt. “The History and Development of N-of-1 Trials.” Journal of the Royal Society of Medicine 110, no. 8 (2017): 330-340. https://doi.org/10.1177/0141076817721131.

Mueller, Michael P., Deborah Tippins, and Lynn A. Bryan. “The Future of Citizen Science.” Democracy & Education 20, no. 1 (2011): Article 2. https://democracyeducationjournal.org/home/vol20/iss1/2.

Nebeker, Camille, John Harlow, Rebeca Espinoza Giacinto, Rubi Orozco-Linares, Cinnamon S. Bloss, and Nadir Weibel. “Ethical and Regulatory Challenges of Research Using Pervasive Sensing and Other Emerging Technologies: IRB Perspectives.” AJOB Empirical Bioethics 8, no. 4 (2017): 266-276. https://doi.org/10.1080/23294515.2017.1403980.

Neuringer, Allen. “Self-Experimentation: A Call for Change.” Behaviorism 9, no. 1 (1981): 79-94. https://doi.org/10.2307/27758973.

—. “What I’ve Learned From Thirty Years of Doing and Teaching Self-Experimentation.” Quantified Self. 2018. https://vimeo.com/292507202.

Parikh, Parin, Heidi Mochari, and Lori Mosca. “Clinical Utility of a Fingerstick Technology to Identify Individuals with Abnormal Blood Lipids and High-Sensitivity C-Reactive Protein Levels.” American Journal of Health Promotion 23, no. 4 (2009): 279-282. https://doi.org/10.4278/ajhp.071221140.

Pettibone, Lisa, Katrin Vohland, and David Ziegler. “Understanding the (Inter)disciplinary and Institutional Diversity of Citizen Science: A Survey of Current Practice in Germany and Austria.” PLoS ONE 12, no. 6 (2017): e0178778. https://doi.org/10.1371/journal.pone.0178778.

Plüddemann, Annette, Matthew Thompson, Christopher P Price, Jane Wolstenholme, and Carl Heneghan. “Point-of-Care Testing for the Analysis of Lipid Panels: Primary Care Diagnostic Technology Update.” The British Journal of General Practice 62, no. 596 (2012): e224–26. https://doi.org/10.3399/bjgp12X630241.

Punjaa, Salima, et al. “N-of-1 Trials Are a Tapestry of Heterogeneity.” Journal of Clinical Epidemiology 76 (2016): 47–56. https://doi.org/10.1016/j.jclinepi.2016.03.023.

Quantified Self. “CardioChek Blood Test Instructional Video.” 2017. https://quantifiedself.com/blog/quantified-self-guide-tracking-cholesterol-hdl-triglycerides/.

Quantified Self Labs. “Blood Testers Reference Material.” Quantified Self. http://qscvd.quantifiedself.com/index.html%3Fp=652.html.

Sblendorio, V., B. Palmieri, and G. Riccionii. “Blood Cholesterol Concentration Measured by CR3000: Fingerstick Versus Venous Sampling.” International Journal of Immunopathology and Pharmacology 21, no. 3 (2008): 729-733. https://doi.org/10.1177/039463200802100328.

Stone, Neil J., et al. “2013 ACC/AHA Guideline on the Treatment of Blood Cholesterol to Reduce Atherosclerotic Cardiovascular Risk in Adults A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines.” Circulation 129 (2014): S1-S45. https://doi.org/10.1161/01.cir.0000437738.63853.7a.

Swanson, Alexandra, Margaret Kosmala, Craig Packer, and Chris Lintott. “A Generalized Approach for Producing, Quantifying, and Validating Citizen Science Data from Wildlife Images.” Conservation Biology 30, no. 3. (2016): 520-531. https://doi.org/10.1111/cobi.12695.

UN General Assembly. “Universal Declaration of Human Rights.” 217 (III) A. Paris, 1948. http://www.un.org/en/universal-declaration-human-rights/.

Vayena, Effy, et al. “Research Led by Participants: a New Social Contract for a New Kind of Research.” Journal of Medical Ethics 42, no. 4 (2016): 216-219. https://doi.org/10.1136/medethics-2015-102663.

Vayena, Effy and John Tasioulas. “Adapting Standards: Ethical Oversight of Participant-Led Health Research.” PLoS Med 10, no. 3 (2013): e1001402. https://doi.org/10.1371/journal.pmed.1001402.

Vayena, Effy and John Tasioulas. “The Ethics of Participant-Led Biomedical Research.” Nature Biotechnology 31 (2013): 786. https://doi.org/10.1038/nbt.2692.

Weissman, Joel S., et al. “IRB Oversight of Patient-Centered Outcomes Research: A National Survey of IRB Chairpersons.” Journal of Empirical Research on Human Research Ethics 18, no. 4 (2013): 421-431. https://doi.org/10.1177/1556264618779785.

Westfall, John M, Linda Zittleman, Maret Felzien, Marc Ringel, Alison Lakin, and Don Nease. “Institutional Review Board Training When Patients and Community Members Are Engaged as Researchers.” Family Practice 34, no. 3 (2017): 301–304. https://doi.org/10.1093/fampra/cmw112.

Wicks, Paul, Timothy E Vaughan, Michael P Massagli, and James Heywood. “Accelerated Clinical Discovery Using Self-Reported Patient Data Collected Online and a Patient-Matching Algorithm.”Nature Biotechnology 29 (2011): 411-414. https://doi.org/10.1038/nbt.1837.

Wolf, Gary and Ernesto Ramirez. Quantified Self Public Health Symposium. QS Labs; Robert Wood Johnson Foundation, 2014. https://www.rwjf.org/en/library/research/2014/09/quantified-self-public-health-symposium-2014.html.

Yogeeswaran, Ganesa, Charles E. Jebaraj, and Venkataraman Sriram. “Comparative Evaluation of Ultramicro—and Macro-Chemo Enzyme Based Assays of Glucose, Cholesterol and Triglycerides.” Indian Journal of Clinical Biochemistry 14, no. 2 (1999): 220–228. https://doi.org/10.1007/BF02867922.

Footnotes

^[1] Mueller et al., “The Future of Citizen Science”; and Vayena et al., “Research Led by Participants.”

^[2] Jones and Schoeller, “Evidence for Diurnal Periodicity”; Pettibone, Vohland, and Ziegler, “Understanding the (Inter)disciplinary”; Swanson et al., “A Generalized Approach”; Citizen Science Alliance; Bonney et al., “Next Steps for Citizen Science”; Cohn, “Citizen Science”; Follett and Strezov, “An Analysis of Citizen Science Based Research”; Gura, “Citizen Science: Amateur Experts”; and Hand, “Citizen Science: People Power.”

^[3] Kravitz, “Medicine, Politics, and the English Language”; Mirza et al., “The History and Development of N-of-1 Trials”; and Punja et al., “N-of-1 Trials.”

^[4] Banks et al., “Everyday Ethics”; Buchanan, Miller, and Wallerstein, “Ethical Issues”; Dias and Gama, “Community-Based Participatory Research”; Fleurence et al., “The Patient-Centered Outcomes Research Institute’s Role”; and Weissman et al., “IRB Oversight of Patient-Centered Outcomes Research.”

^[5] Vayena and Tasioulas, “Adapting Standards.”

^[6] Vayena et al., “Research Led by Participants”; and Vayena and Tasioulas, “The Ethics of Participant-Led Biomedical Research.”

^[7] Vayena and Tasioulas, “The Ethics of Participant-Led Biomedical Research.”

^[8] Wicks et al., “Accelerated Clinical Discovery”; Lewis, “Setting Expectations”; Wolf and Ramirez, Quantified Self Public Health Symposium.

^[9] Follett and Strezov, “An Analysis of Citizen Science Based Research.”

^[10] Neuringer, “What I’ve Learned.”

^[11] Neuringer, “Self-Experimentation.”

^[12] Camp, “Cardiovascular Disease Prevention”; and Dehghan et al., “Associations of Fats.”

^[13] Plüddemann et al., “Point-of-Care Testing for the Analysis of Lipid Panels.”

^[14] Buchanan, Miller, and Wallerstein, “Ethical Issues”; Khanlou and Peter, “Participatory Action Research”; Mikesell, Bromley, and Khodyakov, “Ethical Community-Engaged Research”; Nebeker et al., “Ethical and Regulatory Challenges”; and Weissman et al., “IRB Oversight of Patient-Centered Outcomes Research.”

^[15] Westfall et al., “Institutional Review Board Training.”

^[16] Bloss et al., “Reimagining Human Research Protections”; and Nebeker et al., “Ethical and Regulatory Challenges.”

^[17] Grant, Wolf, and Nebeker, “Approaches to Governance.”

^[18] Grant and Wolf, “Free-Living Humans.”

^[19]Grant, Wolf, and Nebeker, “Approaches to Governance.”

^[20] Grant, Wolf, and Nebeker, “Approaches to Governance.”

^[21]Grant and Wolf, “Free-Living Humans.”

^[22] UN General Assembly, “Universal Declaration of Human Rights.”

^[23] Choe, Lee, and Schraefel, “Characterizing Visualization Insights”; Choe et al., “Semi-Automated Tracking”; Choe et al., “Understanding Quantified-Selfers’ Practices.”