Is My Data Valid?

How much do I trust this data?

This question has kept me awake many a night, both in the lab and during self-tracking experiments. Researchers do validation tests even when using expensive and widely trusted laboratory equipment, and these tests often expose unexpected problems. Commercial self-tracking devices present similar challenges, especially because company-sponsored validation tests may not be independently verified and may be difficult to understand and replicate individually. Though this problem won’t be solved overnight, there are several steps you can take to better understand the constraints of your new technology.

In this post, I’m going to take you through some lessons I’ve learned from a recent attempt to validate a widely used lipid test kit. Some of these lessons are generally applicable, and I hope they will be useful to you as you do your own tests.

I’ve been working on a Quantified Self project to support people doing unusually high frequency home lipid testing. For the project to succeed, we need to determine the accuracy and precision of the CardioChek® Plus from PTS diagnostics. (Accuracy refers to how close a device output is to its real value; precision measures how consistent a measure is across identical trials.) We chose this device from among almost a dozen different options and approaches, because it was in common use, was easily accessible, and was approved by the FDA for home use. But in order to ask sensible questions about our data, we needed to know how well it would do under real conditions. With some work, I was able to find the reported accuracy and precision of the device, verify both in my own hands, and alleviate considerable anxiety about generating believable data.

Here are some tips based on my experience. Of course success is not guaranteed; it will depend on the device you have, the time you’re willing to invest, and a bit of luck. But these general suggestions should get you on your way.

  1. Look carefully through the wearable’s website and read the fine print. Some manufacturers report their in-house testing online, but tucked in a corner where it’s hard to find. Although companies typically won’t report bad results, it’s at least a place to start.
  2. Pubmed. This is the watering hole for finding scientific literature. Try searching the name of your wearable here. Abstracts are generally available.
  3. Check the QS forum. Someone there may have the details of your device.
  4. Contact the company, and frame your questions about getting the most accurate and valid data as positively as possible. Many of these companies are small. Yes, they might brush you off, but they might be willing to give you insider tips on how to best use your device, or even raw data from their own trials to compare with your own. If you are able to explain a personal experiment requiring a particular degree of accuracy or precision, a one-on-one conversation is more likely to get you a relevant an honest answer than hours of googling. There are often hidden factors (lighting and humidity in my case) that make a huge difference in your data quality.
  5. Find a medical/industry standard to compare your device to. It’s very important to not only read reports of a device’s accuracy and precision, but to test it in your own hands. For me, this meant making a doctor’s appointment for a fasting lipid panel, taken at the same time as I did my own finger prick test with the CardioCheck. This is not always possible (most of us are unlikely to have access to polysomnography), but do your best.
  6. Replicate your results under similar conditions. This one is often easier.
    If you measuring, say temperature, do so many times in a row to see the amount of variability. To see how accurate your step or distance tracker, walk from your house to the park several times and compare results. In my case, I pricked my fingers a few times in a row (ouch, but necessary).
  7. Take time of day into account when you are doing any measurement. Circadian rhythms are prominent in pretty much every system in your body. This means you should expect variability in any output by time of day. Let’s say that you’re tracking your basal body temperature (BBT) upon waking up as part of tracking ovulatory cycle. Sleeping until 11am on Saturday when you usually record at 6am on weekdays will confound the prediction of your cycle for sure! A perfectly accurate device can’t be a stand-in for good controls in your personal experiment.
  8. Once you know the constraints of your device, work within them. This may seem obvious, but it’s common to put too much faith in unverified data. Numbers aren’t magic. They are the outputs of sensors with strengths and weaknesses and calculations programmed by humans. Even a device with imperfect accuracy, but is consistent, can give useful information: you just have to figure out the right questions to ask.
  9. Don’t give up. This process takes time, but pays off in the long run. The trust gained from putting an honest effort into validation will save you hours, days or even weeks of confusion from trying to explain results that are just noise in the system, or from having to re-do an entire experiment. Save that time now.
  10. Embrace uncertainty. One of the toughest parts about navigating the validation of a new device is getting comfortable with uncertainty. A peek under the hood often reveals a lot we might wish we didn’t know. Sure, it would be nicer if the world delivered perfect data with every wearable purchase, but it isn’t so. Like all learning endeavors, it’s a continually evolving process that will not guarantee perfection. Questioning one’s potentially false sense of certainty, and leaning into the tricky process of confronting unknowns is a good practice to keep us honest anyways.

If you have done a validation test of a self-tracking tool, we’d like to hear about it.

Posted in Lab Notes | Tagged , , | Leave a comment

Meetup This Week in Cambridge, United Kingdom

qs2011europesteveeri_720

This Wednesday the Cambridge QS Group is hosting a meetup on making exercise more fun. If you want to see how multiplayer gaming, science museums, and office furniture hacks can help you get more movement into your day, this is a meetup for you.

Wednesday, August 9

Cambridge, England

To see when the next meetup in your area is, check the full list of the over 100 QS meetup groups in the right sidebar. Don’t see one near you? Why not start your own! If you are a QS Organizer and want some ideas for your next meetup, check out the myriad of meetup formats that other QS organizers are using here.

Posted in Meetups | Tagged , | Leave a comment

QS17 Highlight: My Scars by Ellis Bartholomeus

image (5)

Ellis Bartholomeus gave a talk at QS17 that is a definite “must watch.”

We all accumulate physical scars throughout our lives. My own history consists of scrapes from trees, kitchen burns and a misadventure with liquid nitrogen. Ellis has taken a quantitative and thoughtful look at the form and meaning of her scars. At left, is a map of decades’ worth of Ellis’ scars. In her talk, she walks through that history and the way she turned it into data, which allowed novel findings, such as, if she added the lengths of her scars, the total is over a meter.

We often spend our time trying to appear and feel unworn by our years, covering them with clothing or makeup and ignoring the memories they represent. While Ellis embraces the past, she discusses the frustration associated with the limitations imposed by her injuries. Still, she uses her scars as a visual reminder to appreciate her own history and resilience.

You may watch the entire talk at her project’s page.

Posted in QS17, Videos | Tagged , , | Leave a comment

Meetups This Week

MUTW2

We have meetups on both sides of the pond this week . Manchester and London will have a great program of show&tell talks, and Austin will be talking about accountability groups and fitness.

Tuesday, August 1st
Manchester, England — Only 12 spots left!

Thursday, August 3rd
Austin, Texas, United States
London, England

To see when the next meetup in your area is, check the full list of the over 100 QS meetup groups in the right sidebar. Don’t see one near you? Why not start your own! If you are a QS Organizer and want some ideas for your next meetup, check out the myriad of meetup formats that other QS organizers are using here.

Posted in Meetups | Tagged , | Leave a comment

Meetup This Week in L.A.

IMG_4626
Live near the USC campus and have a QS project to share? The QS L.A. meetup is getting started again by hosting a show&tell session tomorrow.

Tuesday, July 25th
Los Angeles, California

We haven’t been doing these posts as often recently - but meetups have still been happening! To see when the next meetup in your area is, check the full list of the over 100 QS meetup groups in the right sidebar. Don’t see one near you? Why not start your own! If you are a QS Organizer and want some ideas for your next meetup, check out the myriad of meetup formats that other QS organizers are using here.

Posted in Meetups | Tagged , | Leave a comment

QS17 Highlight: Taking on my Osteoporosis

Justin LawlerA persistent theme at the 2017 Quantified Self Conference was how self-tracking can help those with chronic conditions spot associations between symptoms and lifestyle that a clinician might not have time to uncover. These personal discoveries can help improve one’s health.

In this show&tell, Justin Lawler talks about learning that he has early onset osteoporosis and the several metrics, including diet, microbiome, exercise, sleep and bone density, he tracks to help manage and understand the disease.

I love that the talk emphasizes that many QS projects are long term – even lifelong. Most conventional research projects have a start and end date, garnering a lot of information but only addressing a limited window in time. The self awareness that comes with self tracking can be useful across months and years, elucidating subtle patterns that might otherwise be undetectable.

Watch Justin’s show&tell talk at it’s project page.

Posted in Conference, QS17, Videos | Tagged , , , | Leave a comment

QS17 Highlight: Body Temperature and Ovulatory Cycles

Azure TalkI was thrilled to have the chance to do a Show&Tell talk about tracking my ovulatory cycle via minute-by-minute body temperature during the final plenary session at QS17 Conference. It’s an ongoing project that explores what high-temporal-resolution body temperature can help us learn about our reproductive state. Daily body temperature readings are already used to aid fertility tracking, but several of you expressed interest in collecting more frequent data with me. You inspired me to start uploading my cycle tracking code on Github. I’ll be adding to this repository over time, so check back and shoot me a message if you have an idea you’d like me to try!

Posted in Conference, QS17 | Tagged , | Leave a comment

QS17 Amsterdam Highlight: Tracking Crying

Robin_Weis_CrymotionWe’re back from QS17 and eager to share the conference with you from beginning to end. This, our ninth conference, covered a lot of ground: we showcased self-tracking projects, investigated our relationship with technology, and discussed the past and future of QS. Over the coming weeks, we’ll share some conference highlights.

Today I want to share our opening Show & Tell from Robin Weis, which captures the personal discovery and data-driven spirit of QS. If you’re new to QS, you might not know that the community is about much more than tracking your steps or your hours of sleep: it’s about gaining personal insight by putting numbers to any important aspect of your life. Robin Weis tracked an unusual metric – crying – over a long period of time and did an inspiring job tying together her personal story with her data. Click the link above, check it out, and come back in a few days for another talk!

Posted in QS17 | Tagged , | Leave a comment

Stay Tuned for QS Conference Talks and Slides

QS17RegistrationWe’ve just returned from the latest Quantified Self Conference in Amsterdam and we’re working on the slides and videos of talks to share with everybody. It always takes a few weeks of work to get them ready for posting, so be patient.

If you are already nostalgic (or wishing you’d been there) you can check out some random highlights using the QS17 tag on Twitter. Official documentation coming soon!

Posted in QS17 | Tagged | Leave a comment

QS17 Program Updates

QS17 is here!

We can’t wait to see old friends and meet new people who have never been to a QS conference before. If you are reading this because you are attending QS17, you are in for something special. Our program is filled with engaging talks, fascinating discussions, and informative workshops. Here is a PDF of the printed program.

Here is a map of the venue to help you find your breakout discussion and workshop rooms:
Casa400map

Updates to the QS17 program

Our sessions come from our attendees and we try to leave the program open for as long as we can before we send it to the printer. Still, some things come in after the deadline and we can’t resist finding a spot for them. Below are new sessions that will be happening in addition to the printed program:

Saturday

Lunch (13:00-14:00)
Office Hours
GERO
Stanislav Leontenko, Nikolai Kovtunenko
Gero has developed a first digital biomarker of aging which can be measured by your physical activity gathered from your smartphone or wearable device. By using this tool, you can evaluate your personalized effect of healthy or unhealthy lifestyle. 
getgero.com

BLOODTESTERS
Azure Grant
Get your blood cholesterol tested! Lets’ talk about the value of this metric and what could be learned by having the ability to get a lipid profile in your home.

Session 3 (14:00-15:00)
Breakout Session
QS IN THE JUDICIAL CONTEXT (Harvard)
Liza Cornet
In what way could QS help with the understanding, prevention and intervention of offending behavior.

Sunday

Session 5 (10:30-11:30)
Show&Tell
DIABETES VIZ
Peter Kok
In this talk, I will show a custom web app that I’ve created that automatically collects data from continuous glucose monitor and with some novel visualizations, helps me manage my diabetes.

Lunch (13:00-14:00)
Office Hours
ACTITHINGS
Elke Beck
I’m a human-computer interaction researcher and I’m currently exploring how sensor-based self-tracking technologies can help older adults (65+ years) to reflect on their physical activity/ sedentary habits in daily life. I’ve developed several low-tech,  interactive prototypes which people can use for tracking and reflecting their physical activity behaviour.

BLOOD VALUE RESPONSES TO FOOD, FASTING, AND EXERCISE
Ben Best
I have been monitoring the way my blood glucose, ketone, and triglyceride respond to various foods by making hourly measurements 6 to 10 hours after eating the food. I practice a ketogenic diet, and often fast. Every few months I get very detailed laboratory blood tests.

BLOODTESTERS
Azure Grant
Get your blood cholesterol tested! Lets’ talk about the value of this metric and what could be learned by having the ability to get a lipid profile in your home.

Posted in Conference, QS17 | Tagged , | Leave a comment