Tag Archives: Wearables
How much do I trust this data?
This question has kept me awake many a night, both in the lab and during self-tracking experiments. Researchers do validation tests even when using expensive and widely trusted laboratory equipment, and these tests often expose unexpected problems. Commercial self-tracking devices present similar challenges, especially because company-sponsored validation tests may not be independently verified and may be difficult to understand and replicate individually. Though this problem won’t be solved overnight, there are several steps you can take to better understand the constraints of your new technology.
In this post, I’m going to take you through some lessons I’ve learned from a recent attempt to validate a widely used lipid test kit. Some of these lessons are generally applicable, and I hope they will be useful to you as you do your own tests.
I’ve been working on a Quantified Self project to support people doing unusually high frequency home lipid testing. For the project to succeed, we need to determine the accuracy and precision of the CardioChek® Plus from PTS diagnostics. (Accuracy refers to how close a device output is to its real value; precision measures how consistent a measure is across identical trials.) We chose this device from among almost a dozen different options and approaches, because it was in common use, was easily accessible, and was approved by the FDA for home use. But in order to ask sensible questions about our data, we needed to know how well it would do under real conditions. With some work, I was able to find the reported accuracy and precision of the device, verify both in my own hands, and alleviate considerable anxiety about generating believable data.
Here are some tips based on my experience. Of course success is not guaranteed; it will depend on the device you have, the time you’re willing to invest, and a bit of luck. But these general suggestions should get you on your way.
- Look carefully through the wearable’s website and read the fine print. Some manufacturers report their in-house testing online, but tucked in a corner where it’s hard to find. Although companies typically won’t report bad results, it’s at least a place to start.
- Pubmed. This is the watering hole for finding scientific literature. Try searching the name of your wearable here. Abstracts are generally available.
- Check the QS forum. Someone there may have the details of your device.
- Contact the company, and frame your questions about getting the most accurate and valid data as positively as possible. Many of these companies are small. Yes, they might brush you off, but they might be willing to give you insider tips on how to best use your device, or even raw data from their own trials to compare with your own. If you are able to explain a personal experiment requiring a particular degree of accuracy or precision, a one-on-one conversation is more likely to get you a relevant an honest answer than hours of googling. There are often hidden factors (lighting and humidity in my case) that make a huge difference in your data quality.
- Find a medical/industry standard to compare your device to. It’s very important to not only read reports of a device’s accuracy and precision, but to test it in your own hands. For me, this meant making a doctor’s appointment for a fasting lipid panel, taken at the same time as I did my own finger prick test with the CardioCheck. This is not always possible (most of us are unlikely to have access to polysomnography), but do your best.
- Replicate your results under similar conditions. This one is often easier.
If you measuring, say temperature, do so many times in a row to see the amount of variability. To see how accurate your step or distance tracker, walk from your house to the park several times and compare results. In my case, I pricked my fingers a few times in a row (ouch, but necessary).
- Take time of day into account when you are doing any measurement. Circadian rhythms are prominent in pretty much every system in your body. This means you should expect variability in any output by time of day. Let’s say that you’re tracking your basal body temperature (BBT) upon waking up as part of tracking ovulatory cycle. Sleeping until 11am on Saturday when you usually record at 6am on weekdays will confound the prediction of your cycle for sure! A perfectly accurate device can’t be a stand-in for good controls in your personal experiment.
- Once you know the constraints of your device, work within them. This may seem obvious, but it’s common to put too much faith in unverified data. Numbers aren’t magic. They are the outputs of sensors with strengths and weaknesses and calculations programmed by humans. Even a device with imperfect accuracy, but is consistent, can give useful information: you just have to figure out the right questions to ask.
- Don’t give up. This process takes time, but pays off in the long run. The trust gained from putting an honest effort into validation will save you hours, days or even weeks of confusion from trying to explain results that are just noise in the system, or from having to re-do an entire experiment. Save that time now.
- Embrace uncertainty. One of the toughest parts about navigating the validation of a new device is getting comfortable with uncertainty. A peek under the hood often reveals a lot we might wish we didn’t know. Sure, it would be nicer if the world delivered perfect data with every wearable purchase, but it isn’t so. Like all learning endeavors, it’s a continually evolving process that will not guarantee perfection. Questioning one’s potentially false sense of certainty, and leaning into the tricky process of confronting unknowns is a good practice to keep us honest anyways.
If you have done a validation test of a self-tracking tool, we’d like to hear about it.
Like many people paying attention to the press around Quantified Self, self-tracking, and wearable technology I was intrigued by the many articles that focused on a newly published research letter in the Journal of the American Medical Association. The letter, Accuracy of Smartphone Applications and Wearable Devices for Tracking Physical Activity Data, authored by Meredith A. Case et al., described a laboratory study that examined a few different smartphone applications and self-tracking devices. Specifically, they tested the accuracy of steps reported by the three different apps: Moves (Galaxy S4 and iPhone 5s), Withings Health Mate (iPhone 5s), and the Fitbit app (iPhone 5s), three wrist-worn devices: Nike Fuelband, Fitbit Flex, and the Jawbone UP24, and three waist-worn devices: Fitbit One, Fitbit Zip, and the Digi-Walker SW-200. Participants walked on a treadmill at 3.0 MPH for trials of 500 steps and 1500 steps while a research assistant manually counted the actual steps taken. Here’s what they found:
As the data from this research isn’t available we’re left to rely on the authors description of the data. They state that differences in observed vs device recorded steps counts “ranged from−0.3% to 1.0% for the pedometer and accelerometers [waist], −22.7%to −1.5% for the wearable devices [wrist], and −6.7% to 6.2% for smartphone applications [phone apps].” Overall the authors concluded that devices and smartphone apps were generally accurate for measuring steps. However, much of the press around this study dipped into the realm of sensationalism or attention grabbing headlines, for instance: Science Says FitBit Is a Joke.
Part of our work here at Quantified Self Labs is to encourage and help individuals make sense of their own data. After reading this research letter, or one of the many articles which covered it, you might be asking yourself, “I wonder if my device is accurate?” or “Should I be using a step tracking device or just my phone?” In the interest of helping people make sense of their data so that they can come to their own conclusions I decided to do a quick analysis of my own personal data.
For this analysis I examined the step data derived from my Fibit One and the Moves app I have installed on my iPhone 5. (Important note: the iPhone 5 does not have the M7 or M8 chip present on the 5s and 6/6+, respectively, which natively tracks steps.) I had a sneaking suspicion that my data experience differed from the findings of Case and her colleagues. Specifically, I had a hypothesis that the data from every day tracking via the Moves app would be significantly different than data from my Fitbit One.
First, I downloaded and exported my daily aggregate Fitbit data for 2014 using our Google Spreadsheets Fitbit script. I then exported my complete Moves app data via their online web portal. To create a daily aggregate step value from my Moves data I collapsed all activities in the summary_2014.csv file for each day. (Side note: We’ll be publishing a series of how-to’s for doing simple data transformations like this soon). This allowed me to create a file with daily aggregate step data from both Moves and my Fitbit for each day of 2014. Unfortunately I did not have my Fitbit for the first few weeks of 2014 so the data represents steps counts for 342 days (1/24/14 to 12/31/14).
I found that my Fitbit One consistently reports a higher number of total steps per day than my Moves app. Overall, for the 342 days I had 689,192 more steps reported by Fitbit than by the Moves app. The descriptive information is included in the table below:
Another way to look at this is by visualizing both data sets across the full time-frame:
There a few interesting things to point out in this dataset. On two days I have 0 steps reported from my Moves app. One day, Moves was unable to connect with their online service due to me being in an area with little to no cell signal. On the other day my phone was off, probably due to an iOS 8 release and having to reboot my phone a few times.
It is also clear to me that differences in data are related to how I wear my Fitbit and use my phone. For my Fitbit, it is basically on my hip from the time I wake up until the time I go to bed each night. However, my phone isn’t always “on my body” throughout the day. I think this is probably the case for more people.
Since I wear my Fitbit at all times some of the data it captures erroneously is included in the total step count. For instance, for the last few months in this data set I was commuting about 10 miles per day during the week by bike. This data is accurately captured as cycling by Moves, but captured as steps by my Fitbit. Therefore some over-reporting by Fitbit is present in the data.
For my own data I found that the Fitbit reports higher steps on most, if not all days, than the Moves app on my iPhone 5. There are a few caveats with this data and analysis that are worth mentioning. First, this exploration was intended to begin a conversation around the real-world use of activity monitoring apps and devices, and the data they collect. It was not intended as a statement on truth or validity (however I would welcome the help of a volunteer to follow me around with a manual clicker counting all my steps). Second, this analysis was undertaken in part to help you understand that scientists of all types, be it citizen or academic, have the ability to work with their own data in order to come to their own conclusions about what works or doesn’t work for them. Lastly, this analysis was completed very quickly and I am sure that other individuals may have different ideas about how to explore and analyze the data. For this reason I’m posting the daily aggregate values in a open Google Spreadsheet here.
In our work supporting users and makers of Quantified Self tools we pay close attention to how others talk about trends and markets. In the past year, the most-used catch all term for devices that help us track ourselves has been “wearables.” Now, it’s clear that wearables covers only a fraction of QS practices. Many of the ways people are using numbers, computing, and technology to learn about themselves do not involve wearing anything special. However, the term is useful to us in following relevant research. Below you’ll find links to last year’s best reporting on the wearables market, gathered into a single post for easy reference.
Pew Research Center (January 2013)
The most important work in this space remains the Tracking for Health report from the Pew Research Center, which found that 69% of adults track their health or the health of others, and that 21% of those who track use technology.
Link: QS Analysis of the Pew Research Center Tracking for Health
Forrester, January 2013
A report about the market for fitness wearables “like the Nike+ Fuelband and Jawbone UP” predicts that 8 million US online will be purchasing such devices.
Link: Fitness Wearables — Many Products, Few Customers
Nike, August 2013
Announces in a press release for their “Just Do it” campaign that they have over “18 million global” members of their Nike+ ecosystem.
Link: Nike Redefines “Just Do It” With New Campaign
CCS Insight, October 2013
Surveyed over 700 adults in both the UK and US. They found smart watch adoption was low with only 1.3% of adults (both countries) currently owning and using one and 1.5% no longer using (had owned). For “Wearable Fitness Trackers” they found 2.3% currently owned and used one and 1.2% no longer use it.
Link: User Survey: Wearables UK and US
Endeavor Partners, January 2014 (Part 1)
A survey of “thousands of Americans” completed in late 2013 found that 10% own an activity tracker. Activity trackers were most popular with younger adults (25–34 years) when compared to other age groups. They found that 50% of individuals who have owned an activity tracker no longer use it and one third stopped using it within six months.
Link: Inside Wearables
IDC, March 2014
“This IDC study presents the five-year forecast for the worldwide wearable computing devices market by product category. The worldwide wearable computing devices market (commonly referred to as “wearables”) will reach a total of 19.2 million units in 2014”
Link: Worldwide Wearable Computing Device 2014–2018 Forecast and Analysis
Nielsen, March 2014
A survey conducted in late 2013 of 3,956 adults found that 15% currently “use wearable tech—such as smart watches and fitness bands—in their daily lives.” Device ownership leaned heavily toward “fitness bands” with 61% of wearable technology users reporting ownership. This was followed by smart watches (45%), and mobile health devices (17%).
Link: Are Consumers Really Interested in Wearing Tech on their Sleeves?
Rock Health, June 2014
“While the activity tracker segment has about 1-2% U.S. penetration, wearables overall are expected to grow significantly”
Link: The Future of Biosensing Wearables
Endeavor Partners, July 2014 (Part 2)
As of June 2014, they found that the percentage of adult consumers that still wear and use their activity tracker has improved with 88% still wearing it after three months, 77% after 3–6 months, 66% after 6–13 months, and 65% after a year. They also found that majority of respondents (1,024 of 1,700 surveyed) reported obtaining their divide within the last six months
Inside Wearables – Part 2
PWC, October 2014
“21% of American adults already own a wearable device” They also found in their survey of 1,000 adults that 2% no longer use it, 2% wear it a few times per month, 7% wear it a few times a week, and 10% use it everyday.
Links: The Wearable Future, Health Wearables: Early Days
Acquity Group, November 2014
A survey of 2,000 US consumers found that 13% plan to purchase as wearable fitness device with in the next year, and 33% within the next five years. Additionally, smart clothing is on slower trajectory with 3% planning to purchase in the next year and 14% in the next five years.
Link: The Internet of Things: The Future of Consumer Adoption
Gartner, November 2014
Gartner forecasts that worldwide shipments for “wearable electronic devices for fitness” will reach 68 million units in 2015, a slight decrease from the forecasts from 2014 and 2013 (70.2 and 73 million units, respectively). Additionally, according to Angela McIntyre, Gartner has found that “20 million online adults in the U.S. own and use a fitness wristband or other activity monitor and that 5.7% of online adults in the U.S. own and use a fitness wristband.”
Link: Forecast: Wearable Electronic Devices for Fitness, Worldwide, 2014
Berg Insight, December 2014
This is a market research report that states “fitness and activity trackers is the largest product category” and shipments are forecasted to reach 42 million units in 2019. Smart watches are predicted to reach 90 million units.
Link: Connected Wearables
Accenture, January 2015
Using a survey of 24,000 individuals across 24 countries Accenture found that 8% currently own a “Fitness Wearable”. Furthermore, they found that 12% plan to purchase in the next year, 17% in the next 1–3 years, and 11% in the next 2–5 years.
Link: Engaging the Digital Consumer in the New Connected World
Global Web Index, January 2015
In their Q3 2014 Device Summary report, GWI labeled wearable devices as “highly niche” after finding that 7% of US online adults own a “smart wristband” (Nike Fuelband, Jawbone Up, Adidas miCoach) and 9% own a smart watch.
Link: GWI Device Summary – Q3 2014
Rocket Fuel, January 2015
A survey of 1,262 US adult consumers conducted in December of 2014 found that 31% currently use a QS tool to track their health and fitness. This includes apps, devices, and websites. More specifically, 16% use a wearable device and 29% use a website or app not associated with a wearable device to track health and fitness.
Link: “Quantified Self” Digital Tools
Enjoy the first What We’re Reading post of 2015!
Wearable Devices as Facilitators, Not Drivers of Health Behavior Change by Mitesh Patel, David Asch, and Kevin Volpp. This opinion piece seeks to describe the reasons why currently available health wearables are not “bridging the gap” between tracking and changing behavior.
Big Data Not A Cure-All in Medicine by Amy Standen. This story, which first appeared on All Things Considered, sheds some light on concrete examples of how data can be used to treat medical conditions, and the current roadblocks in place.
The Smart, Angry Home by Emily Anthes. Smarter homes, smarter grids, and more data about our energy use is undoubtably on the horizon. In this piece, Emily Anthes describes how providing data back to individuals about energy use, especially in multi-tenant dwellings, can be a source of tension.
Thoughts on the Quantified Self by Kevin Ripka. I really enjoyed this short post about the author’s reactions to Quantified Self. I was especially interested in his description of the “Four Types of Projects” that he believes one can undertake when self-tracking.
What 2439 Reports Taught Me by Sam Bev. Sam has been using the ReporterApp over the last year. Since he began he’s amassed over 2400 reports, and those have provided some interesting insights into his own life. Read this great post and make sure to visit his website where his reports are made visible.
Seen, Read 2014 by Steven Soderbergh. Steven Soderbergh is an acclaimed writer and director, who has been tracking his media consumption for a few years. This post chronicles the books, plays, TV, movies, and records he consumed during 2014.
Map Your Trips Using Pics From Your Phone by Marco Altini. In this how-to post Marco lays out a fun method for tracking travel and location using only the photos you take with your smart phone.
In response to the much anticipated reveal of the Apple Watch I did a bit of digging around to find out where we stand with wrist-worn wearable devices. I found over 60 different devices. The following list focuses on self-tracking tools, I intentionally left out those that work only as notification centers or secondary displays for your phone. I’m sure this isn’t all of them, but it’s as good a place to start as any. If you’re using one of these devices to learn something about yourself, or you’re just interested in these type of wearable tools we invite you to join us in San Francisco on June 18-20, 2015, for our QS15 Conference & Exposition.
(Thank you to all those who commented here, on Twitter, and on our Facebook group pointing us to additional devices to add!)
Sensors: Accelerometer, Heart Rate (optical), Blood Oxygen, Temperature
Sensors: Accelerometer, Pulse Oximeter, Temperature
Sensors: Accelerometer, Gyroscope, Heart Rate (optical)
Sensors: Materials state the ZenWatch houses a “bio sensors and 9-axis sensor.” I assume optical heart rate, accelerometer, and gyroscope.
Sensors: Accelerometer, Gyroscope, Heart Rate (optical)
Sensors: Accelerometer, Temperature, Pressure
Epson Pulsense Band/Watch
Sensors: Accelerometer, Heart Rate (optical)
Fatigue Science Readiband
Today’s post comes to us from Rain Ashford. Rain is a PhD student, researcher, and hardware tinkerer who is interested in how personal data can be conveyed in new and meaningful ways. She’s been exploring ideas around wearable data and the hardware that can support it. At the 2014 Quantified Self Europe Conference, Rain led a breakout session on Emotive Wearables during which she introduced her EEG Visualizing Pendant and engaged attendees in a discussion around wearing data and devices.
By Rain Ashford
It was great to visit Amsterdam again and see friends at the 3rd Quantified Self Europe Conference, previously I have spoken at the conference on Sensing Wearables, in 2011 and Visualising Physiological Data, in 2013.
There were two very prominent topics being discussed at Quantified Self Europe 2014, firstly around the quantifying of grief and secondly on privacy and surveillance. These are two very contrasting and provocative areas for attendees to contemplate, but also very important to all, for they’re very personal areas we can’t avoid having a viewpoint on. My contribution to the conference was to lead a Breakout Session on Emotive Wearables and demonstrated my EEG Visualising Pendant. Breakout Sessions are intended for audience participation and I wanted to use this one-hour session to get feedback on my pendant for its next iteration and also find out what people’s opinions were on emotive wearables generally.
I’ve been making wearable technology for six years and have been a PhD student investigating wearables for three years; during this time I’ve found wearable technology is such a massive field that I have needed to find my own terms to describe the areas I work in, and focus on in my research. Two subsets that I have defined terms for are, responsive wearables: which includes garments, jewellery and accessories that respond to the wearer’s environment, interactivity with technology or physiological signals taken from sensor data worn on or around the body, and emotive wearables: which describes garments, jewellery and accessories that amplify, broadcast and visualise physiological data that is associated with non-verbal communication, for example, the emotions and moods of the wearer. In my PhD research I am looking at whether such wearable devices can used to express non-verbal communication and I wanted to find out what Quantified Self Europe attendees opinions and attitudes would be about such technology, as many attendees are super-users of personal tracking technology and are also developing it.
My EEG Visualising Pendant is an example of my practice that I would describe as an emotive wearable, because it amplifies and broadcasts physiological data of the wearer and may provoke a response from those around the wearer. The pendant visualises the brainwave attention and meditation data of the wearer simultaneously (using data from a Bluetooth NeuroSky MindWave headset), via an LED (Light Emitting Diode) matrix, allowing others to make assumptions and interpretations from the visualizations. For example, whether the person wearing the pendant is paying attention or concentrating on what is going on around them, or is relaxed and not concentrating.
After I demonstrated the EEG Visualising Pendant, I invited attendees of my breakout session to participate in a discussion and paper survey about attitudes to emotive wearables and in particular feedback on the pendant. We had a mixed gender session of various ages and we had a great discussion, which covered areas such as, who would wear this device and other devices that also amplified one’s physiological data? We discussed the appropriateness of such personal technology and also thought in depth about privacy and the ramifications of devices that upload such data to cloud services for processing, plus the positive and the possible negative aspects of data collection. Other issues we discussed included design and aesthetics of prominent devices on the body and where we would be comfortable wearing them.
I am still transcribing the audio from the session and analysing the paper surveys that were completed, overall the feedback was very positive. The data I have gathered will feed into the next iteration of the EEG Visualising Pendantprototype and future devices. It will also feed into my PhD research. Since the Quantified Self Europe Conference, I have run the same focus group three more times with women interested in wearable technology, in London. I will update my blog with my findings from the focus groups and surveys in due course, plus of course information on the EEG Visualising Pendant’s next iteration as it progresses.
Rain Ashford is a PhD student in the Art and Computational Technology Program at Goldsmiths, University of London. Her work is based on the concept of “Emotive Wearables” that help communicate data about ourselves in social settings. This research and design exploration has led her to create unique pieces of wearable technology that both measure and reflect physiological signals. In this show&tell talk, filmed at the 2013 Quantified Self Europe Conference, Rain discusses what got her interested in this area and one of her current projects – the Baroesque Barometric Skirt.
Today’s post comes to us from Rajiv Mehta, our longtime friend and co-organizer of the Bay Area Quantified Self Meetup group. Rajiv is also leading the team behind UnfrazzledCare, a media and application development company focused on the caregiving community.
“What lessons have we learned through Quantified Self meetings and conferences that would benefit entrepreneurs looking to enter this space?” That’s what I was asked to comment on at a recent event on Quantified Self: The Next Frontier in Mobile Healthcare organized by IEEE and TiE. The workshop took place on September 19, 2013, almost exactly five years after the first QS meetup, naturally leading to a theme of 5 years and 5 lessons.
The 5 themes I discussed were:
- How difficult it is to get an accurate measure on the “market size” for self-tracking, though according to some measures it is a very common activity.
- The importance of and excitement surrounding new sensor technologies, but also what we have learned about our in-built human sensors and the challenges of making sense of the data.
- The need to treat feedback loops with caution; that thoughtful reflection is sometimes better than quick reaction.
- About engagement and motivation, about how so many are drawn to QS through a desire to change their own behaviors, and how QS experiences match behavior science research.
- The value of self engagement, and how self-trackers often learn something even when their experiments aren’t successful.
My slides include my talking points, in small text below the slides. If you view this full-screen, you should be able to read the small text.
Several other QS regulars participated in this workshop. Rachel Kalmar, who runs the Sensored meetup group and is a data scientist with Misfit Wearables, gave a keynote on some of the technology challenges facing those working on the sensing devices. These ranged from the fundamental (“What exactly is a step?”) to prosaic (batteries!), and from business issues (data openness vs competitive advantage) to human issues (accuracy vs wearability). Dave Marvit, of Fujitsu Labs, shared some of their work on real-time stress tracking and his thoughts on the issue of “quantifying subjectivity”. Sky Christopherson, of Optimized Athlete, told the audience of his own health-recovery through self-tracking and how he helped the US women’s track cycling team to a dramatic, silver-medal performance at the London Olympics. QS supports his passion for “data not doping” as a better route to athletic excellence. And Monisha Perkash showed off Lumoback.
The second QS European 2013 Conference is coming up. We run our QS global meetings as “carefully curated unconferences,” meaning that we make the program out of ideas and suggestions from the registrants, with a lot of thoughtful back-and-forth in advance. Today we’re highlighting Rain Ashford.
Rain is currently a researcher in the Art and Computational Technology Program at Goldsmiths, University of London. She has been experimenting with wearable electronics since 2008. At first her work centered on interactive wearables for music and gaming, but she soon became interested in mood and social behavior. Her curiosity led her to what she calls “physiological responsive wearables.” Continue reading
Simon Frid moved to California last year because his data told him he was smarter here than in New York. Well, not really. But this funny story begins his journey of figuring out how to track one of the simplest things that we don’t generally know about ourselves: our own posture. Simon designed a wearable sensor shirt with ten built-in accelerometers, and was able to improve his posture significantly from December to January. In the video below, he shares how he trained the shirt to recognize good posture, why he didn’t want immediate feedback, and what question he most wants to ask people. (Filmed by the Bay Area QS Show&Tell meetup group.)