Ernesto Ramirez

June 25, 2015

When the Apple Watch was announced I started waiting with bated breath to see how it could be useful for Quantified Self and self-tracking purposes. Of course this means staying up late and making sure I had one on order as soon as possible. I put in my order shortly after midnight on launch day for a 42mm Space Gray with the black sport band.

On May 19th my Apple Watch arrived, coincidentally just after we wrapped on our first Bay Area Apple Watch Users Group meeting (which was fantastic and I highly recommend joining). I set it up and started figuring out how it worked as an activity tracker. I have a keen interest in activity tracking, not just as a self-tracker, but also as a graduate student studying how people use activity tracker data to understand and impact their lives. In that vein, I’ve been a consistent Fitbit user for over four years, transitioning from the original Fitbit to the Ultra, and then to my current Fitbit One. I’m a big fan of the Fitbit and use it as my personal “gold standard” for activity tracking. It’s accurate, consistent, and easy to use. Does that hold true for the Apple Watch? Let’s find out.

What did I do?

I wore my Apple Watch every day, from the moment I woke up to when I went to sleep at night. I set up my charging station on my nightstand, which is also where my Fitbit One spends its nights. I wasn’t thinking about this data analysis when I first started wearing the watch, but looking back over the past month I am confident saying that if I was wearing my Fitbit I was also wearing the watch.

This data analysis includes data from May 20th to June 23rd, or 35 days of data collection. My activities varied as a normal function of my work and life, meaning I didn’t purposefully mix things up or engage in activities just for testing purposes. Many days were sedentary, some days had longer walking periods, and in the 35 days I ran seven times at distances between four and nine miles.

How did I do it?

Exporting the data from both the Fitbit and the Apple Watch is not a trivial task, but thanks to a few pieces of software I was able to access and analyze both data sets.

Apple Watch
The Apple Watch stores the data it collects in Apple’s Health app using Healthkit. A quick glance into the Health app indicates that it is storing minute-level step data from the Apple Watch. Apple built in a data export function for the Health app, but it’s in a proprietary XML format that I’m not super familiar with. Thankfully there is QS Access. Our team at QS Labs created simple app that connects to Apple Health and allows you to export your data in a easy to use .csv file.

To export my data I first made sure that the Apple Watch had the highest priority for the data sources that feed the “steps” data for Apple Health. This is important because all newer iPhones (5s, 6, 6+) also natively create step data and store it in the Health app. I then used QS Access to create a data export for steps. I chose the hourly function as it’s the highest level of granularity the QS Access app currently offers for data export.

Fitbit
Fitbit recently introduced a data export feature. While this is a great step forward for them, and for their millions of users, the export feature is a bit limited. You can only export daily aggregate data and only one month of data is exportable at a time. Since I had access to hourly data from the Apple Watch I wanted to match that granularity.

I turned to my good friend, and past colleague, Aaron Coleman. Aaron runs a unique startup called Fitabase, which was built to help researchers, organizations, and individuals get easy access to activity tracker data. I spun up my account at Fitabase, which has been collecting and storing my Fitbit data for the last few years, chose the date range and downloaded my hourly step data.

I wanted to get right to my core question, “How accurate is the Apple Watch compared to the Fitbit One?” so I imported both data files into Google Spreadsheets, did a bit of data formatting, created a pivot table, then made some simple graphs. The full data set is available here if you want explore more complex statistics or visualizations.

What Did I Learn?

When compared to the Fitbit One, the Apple Watch is fairly accurate for step tracking. What do I mean by fairly accurate? Let’s dive into the data.

Daily Steps

When I explored my daily step totals it appeared that the Apple Watch counts more steps than my Fitbit One, but not that many more. Here’s the data you need to know:

• Fitbit Total Steps: 308,955
• Apple Watch Total Steps: 317,971
• >Difference: 9,016 or 2.91% of the total steps (counted by Fitbit)

I created a difference category by subtracting Fitbit steps from Apple Watch steps for each day. This allowed me to see how different the data was day over day. The mean difference indicated that Apple Watch counted 258 steps more per day on average. Important to note that the daily difference was highly variable with a standard deviation of 516 steps. Looking at the scatterplot and histogram below you can see a few clear outliers, but what appears to be an otherwise normal(ish) distribution for the difference in step counts.

Hourly Steps
What about when we look at a higher level of granularity? I also explored the hourly steps data and compared the Fitbit and Apple Watch. On average the Apple Watch counted 11 more steps per hour than the Fitbit One during this period. Again, this was highly variable with a standard deviation of 85 steps, and a range from overcounting by 462 steps to undercounting by 696 steps. I haven’t yet filtered out sleep time (0 steps) so the mean difference per hour in this data set is likely skewed low.

I also looked into one more question that I though was interesting. Is there a significant difference in daily or hourly step data as a function of the total steps? Or, more simply, when I’m more active does the Apple Watch still stay consistent?

It appears that being more active doesn’t have a significant impact on how accurate the Apple Watch is tracking and counting steps. I created scatterplots for this relationship and added a simple linear trendline. In both cases, the trendline indicated that only a small amount of variability in the difference between the devices was accounted for by the total steps taken.

So What?

I’m not ready to give up my Fitbit just yet, but I was happy to see that the Apple Watch is an accurate step tracking device. Of course there are caveats to this data set. It’s somewhat small, a little over a month of data, and I didn’t do any “ground truth” testing where I counted my actual steps. However, I feel more confident now that whether I’m walking around my apartment, my nieghborhood, or going on runs, the Apple Watch will accurately reflect those activities.

What’s Next?

Like most other runners who are using the Apple Watch I’m interested to dive into the heart rate data to test it’s accuracy. I’ve already collected a few runs, but will doing a bit more testing to compare to other common heart rate trackers.