Tag Archives: data visualization
In 2009 Tim Ngwena switched on Last.fm and he’s been running in across all his devices ever since. Earlier this year he decided to take a deep dive into his listening data to see what he could learn.
I realized that I was listening to the same old thing and I began to think about changing what I was listening to. But how can I change? Where can I start? I also wanted to learn something about my music, what I was listening to and who was behind the sounds. I decided to focus on music because it was doable.
In this talk, presented at the London QS meetup group, Tim explains how he was able to make sense of almost five years of data and learn more about himself and his listening habits.
What Did Tim Do?
Tim explored his music data along side additional information such as location data from Moves to learn about his musical tastes, listening habits, and explore new visualization and data analysis techniques.
How Did He Do It?
Tim exported his data, used the Last.fm API and some data cleaning and organizational tools to create a simplified and extensive database of his music listening history and associated data. He then visualized that data using Tableau.
What Did He Learn?
Tim learned a lot about himself and what the music he listens to says about him. He describes a few of the most interesting below,
Basically 80% of my listening comes form 10% of the artists that I have in my library.
I’ve listened to Erykah Badu for over a week (7.2 days). It led me to ask what is she saying to me?
Monday is my jam time. I’m listening from the morning into the evening.
I listen to music mostly when I’m walking.
Tim also learned a lot through the process of designing and creating his data visualization. The visualization, which you can explore here, made him think about being able to see the big picture when he has so much linked data.
I think context is important and you need to see all that information in one place and the tools I’m using allows me to do this.
Two weeks ago we announced the release of the QS Access App so you could access your HealthKit data in tabular format for personal exploration, visualization, and analysis. In that short period of time, we’ve seen a good number of downloads and positive feedback.
We know from our experiences hosting in-person and online communication about personal data that seeing real-world examples of what is possible is what inspires people to engage and ask questions of their own data. With that in mind we’re excited to announce our QS Access Visualization Showcase.
We are looking to you, our amazing community of trackers, designers, and visualizers, to show use what you can do with data gathered from using the QS Access App. Make heatmaps in D3, complete analyses and visualizations in Wizard, or just make meaningful charts in Excel. If you’re visualizing your QS Access data we want to see it.
We also know that data visualization design and creation is not trivial work. To support the community and help expose the visualization work we’ll be awarding free tickets to our QS15 Global Conference & Exposition to individuals who use QS Access to create unique and interesting visualizations. We’ve earmarked two tickets (a $700 value) for outstanding work. If you’re selected, we’ll also work with you to showcase your work at the QS15 Conference and Exposition so other community members and attendees can explore and learn from their own data.
If you’re in the Bay Area come to our QS Meetup on November 11th at the Berkeley Skydeck. You can showcase your visualization and tell our community what you’ve learned from accessing and visualizing your data.
HealthKit is still new and the number of apps that integrate with it is growing by the day. At QS Labs we’ve done a bit of work making simple visualizations that are meaningful to us.
Steps and Sedentary Activity
Gary has an iPhone 5s which has native step tracking. We used the QS Access app to export his hourly step totals and made these simple line graphs in Excel. You can read more about what he learned from these simple data visualizations here.
How Much Do I Run?
Ernesto is an avid runner and enjoys running along the quiet trails in Los Angeles. He was interested to see how often he actually runs and if there’s any pattern to his running. Using a well-designed D3 template he was able to make a calendar heatmatp of his running distance.
If you don’t have any HealthKit data to work with, or just want to play with some example data we’ve created a few files that you can use as examples. Download the files below from our GitHub account and make sure to read the documentation to understand where the data is coming from. Descriptions of the data files and sources are available in our QS Access Data Examples repo on Github.
At our 2013 Quantified Self Global Conference we were excited to share a variety of beautiful and insightful data visualizations from our community. In the months leading up to the conference we asked attendees to send in their own personal data visualizations along with a short description. In our 6 years of hosting Quantified Self meetups and events, as well as running this website, our forum, and social channels, we’ve seen the power of data visualization as a story telling medium. We exist in part to help people tell their stories – about the data they collect, the changes they create, and the insights and new knowledge they’re excited to share.
Today we’re sharing a few of our favorite visualizations from past conferences. The images and descriptions below represent a wide a variety of tracking experiences and techniques, and we hope to showcase eve more unique personal data projects at our upcoming QS15 Conference & Exposition.
Tracking Sleep by Anita Lillie
This is concatenation of screenshots from my sleep app. Most sleep apps don’t let you zoom out like this and still see daily/nightly detail, so I just made it myself. I like that it shows how almost-consistent I am with my sleep, and made me ask new questions about the “shape” of a night of sleep for me.
2.5 Years of My Weight by Mette Dyhrberg
I gained a lot of insights from this heat map. The most obvious weight gain was no surprise — that’s when I periodically don’t track. In any case, the big picture patterns are easily identified with a heat map. Realized looking at this heat map that the point of no return was mid-April 2012 — my data shows that was when I switched protein shakes with an egg based breakfast. I have since experimented and seen that protein shake in the morning seems to keep my blood sugar more stable and as a result my weight under control!
One Month of Blood Sugar by Doug Kanter
This is a visualization of one month of my blood sugar readings from October 2012. I see that my control was generally good, with high blood sugars happening most often around midnight (at the top of the circle).
Tracking Productivity by Nick Winter
My percentile feedback graph of my development productivity helps my motivation.
Six Months of My Life by David El Achkar
This is my life during the past six months. Each square = 15 minutes. Each column = 1 day. This picture represents 138 days or 3,000+ activities.
My Thesis Self Portrait by Sara M. Watson
Here’s a period of a few days of webcam images taken using Stan James’ LifeSlice during the final days of editing my thesis on Quantified Self uses of personal data. Serious business!
Sleep and Meaningful Work by Robby Macdonell
In an average work day, I don’t consider communication (email, instant message, etc) to be terribly meaningful work. I’d much rather be working on building software. Getting more sleep the night before increases the amount of meaningful work I’m likely to do in a day.
70 Days of Pulse by Laurie Frick
Pulse rate over 24 hours for 70 days from my Basis watch. Grey=null, blues=85
Rain Ashford is a PhD student in the Art and Computational Technology Program at Goldsmiths, University of London. Her work is based on the concept of “Emotive Wearables” that help communicate data about ourselves in social settings. This research and design exploration has led her to create unique pieces of wearable technology that both measure and reflect physiological signals. In this show&tell talk, filmed at the 2013 Quantified Self Europe Conference, Rain discusses what got her interested in this area and one of her current projects – the Baroesque Barometric Skirt.
We’re back with another great set of articles, show&tells, and visualizations for you.
How to Make Government Data Sites Better by Nathan Yau. Government entities are some of the largest holders of interesting data. Nathan focuses this article on the difficulties of accessing and making sense of data from the United States Centers for Disease Control and offers some good ideas on how to make it better.
Project Eavesdrop: An Experiment At Monitoring My Home Office by Steve Henn. What happens when you start monitoring yourself in the same manner the NSA might be doing? The author employs some technical help to learn what data leaks are possible and what you can figure out from your digital trails.
Sitting is Bad for You. So I Stopped. For a Whole Month. by Dan Kols. As a past frequent user of a treadmill desk and a sedentary behavior researcher I found this article intriguing. Yes, a bit silly in nature, but an interesting look at what happens when you go to the extreme. I especially enjoyed the integration of personal tracking in the piece.
Analyzing Squash Performance Using Fitbit by Ben Sidders. Ben sought out to see if he could learn anything from his step data to improve his squash playing. In this post he explains how he used R to access his data and plot it against his squash records, which he also records.
My Life As Seen Through Fitbit. Reddit user, VisionsofStigma, plots a year and a half of Fitbit data to find out what is related to the rise and fall of his activity.
Freeing My Fitbit Data by Bonnie Barrilleaux. Bonnie used our instructions for accessing Fitbit data in Google Spreadsheets then used Python to visualize her data. I especially like the histogram pictured at left. If you want to visualize your Fitbit data, she’s included her code in the post.
Diurnal Plot of Fitbit Data by Matthew Gaudet. Matthew was inspired by the diurnal plots from Stephen Wolfram’s in-depth personal data review. He implemented the same methods to better understand his activity.
Iconic History by Shan Huang. As part of a the Interactive and Computational Design class at Carnegie Mellon University, Shan created a Chrome browser extension that visualizes your browser history. More about the project here.
Visualizing Last.fm History by Andy Cotgreave. Andy has been using Last.fm since 2006 to track his music listening activity. As a data scientist he was interested in what he could learned from all that data. In this four-part series, he explores his data along side data from eight of his friends. (If you want explore your Last.fm data you can export it using this awesome CSV export tool.)
From the Forum
23andMe Data Analysis Tools
Want to receive the weekly What We Are Reading posts in your inbox? We’ve set up a simple newsletter just for you. Click here to subscribe.
In the Quantified Self community we focus on projects and ideas that help people access and get meaning out their personal data, including the information you can collect with your smartphone. If you have an iPhone, Android, or Windows phone you’re already have carrying of the world’s most sophisticated self-tracking tools. The GPS, accelerometer, the microphone, all of these tiny sensors make up a great set of tools you can use to understand how you move around the world.
I’m going to focus this short “how to” on geolocation data and mapping your movement, specifically using data gathered by the Moves application. Moves is a passive activity and location tracking tool available for the iPhone and Android. We’ve written a bit about it in the past and had a chance to interview their CEO, Sampo Karjalainen. I’ve been using it since May, 2013 and I wanted to share some neat tools and methods for getting a bit more out of the data Moves collects.
I find that visualizing my data on a map to be incredibly powerful. It might by my inner cartographer, but seeing my patterns of movement (or lack there of) in reference to known places and landmarks is a great mechanism for inducing recall and reflection on where I’ve been and what I’ve done. Hopefully you’ll use one of the tools or methods below to map you data and learn something new!
Moves Connected Apps
Like many self-tracking applications and devices, Moves has a API that many different developers have built services on top of. Here are just a few of the services that allow you to see your data on a map. Be advised that each of these services has access to your data. Make sure to read their Terms of Service before agreeing to the data transfer.
WebTrack. This is by far the most utilitarian data mapping tool. However, you shouldn’t get discouraged by the lack of fancy design because it gives you an very unique data view. When you use Moves on your phone you typically only see the “storyline” and the detected places you’ve spent time at. However, Moves is constantly pinging and recording your location when it detects movement. WebTrack allows you to see all those movement points by hovering over the associated timestamp.
Fluxtream. You might know Fluxtream as Friend of QS and a great open-source data aggregation tool. They’ve set up a “Moves Connector” that allows you to import and visualize your Moves data. Because Fluxtream is set up as an aggregation and visualization tool you can also map other interesting data sets. Want to know where you were tweeting last week? Fluxtream will map it for you. (You can see me tweeting on a CalTrain ride between San Francisco and Palo Alto below.)
Zenobase. Another interesting data aggregation service here. Zenobase treats your Moves data bit differently. Rather than importing all the movement geolocation data it focuses on your place data and visualizes those locations. I like the high-level view it start with, but make sure to keep zooming in to see more specific place data.
Resvan Maps. This mapping application adds a unique twist to the typical mapping visualizations. It will plot your places, paths, and categorize paths depending on the activity (transport, walking, running, and cycling). Additionally, you can create “analysis cirlces” and have the application compute the time you spent in a certain location you bound (it aggregates to hours:minutes per day).
MMapper. This method for mapping your data, developed by Nicholas Felton, is by far the most technical, but it produces some really neat visualizations. You’ll have to download Processing and follow the instructions Nicholas provides on the Github repository page here. The great thing here is that the mapping and data access is all happening locally.
Move-O-Scope. Another great mapping application here from the folks at Halftone.co. They’ve probably completed the most thorough mapping and exploration tools for your Moves data. After linking your Moves account you can explore maps by activity type, day of the week, and custom data ranges. Additionally, they’ve implemented a neat feature for exploring place data. You can see how many times you’ve visited a specific place, where you’ve come from and where you go next, what days you typically visit, and your typical time of day at that place. (See this post for background on why they created this nifty tool.)
Map It Yourself!
If you don’t want to trust your data to a third party, but you still want to explore your movement maps there is really great option for you. Our friend and co-organizer of the QS LA Meetup, Eric Blue, recently published a method for easily exporting your data: the Moves CSV Exporter. You’ll have to login and use the Moves pin system in order to download your data, but Traqs isn’t storing your data, just providing a way for you to access it. The tool allows you to download and explore your activity, summary, tracks and place data. We’ll focus on the place data for creating maps. You can also use your full tracks history for mapping all the geolocation points Moves collects.
Because this data is based on latitude/longitude coordinates there are many different methods available that you can use to map your data. I’m going to focus on two here: Google Fusion Tables and CartoDB (if you know of others share them in the comments or our forum).
Google Fusion Tables
Fusion Tables are a new Google Drive tool that you can use to store, analyze, and visualize many different types of data. Once you download your Moves places.csv file you can upload it to a new Google Fusion Table. Once you upload your data, which takes about 2 minutes, you’ll see a menu bar and three tabs: Rows, Cards, Map of longitude. Just click on the “Map” tab and you’ll see your data already placed on a map. If you want to see a heatmap rather than a point map just navigate to Tools -> Change Map and you’ll see an option for a heatmap on the lefthand side. This is just the tip of iceberg for mapping fusion table data. You can learn more about different mapping methods and tricks here.
CartoDB is a visualization and analysis engine for geospatial data. I’ve been using it to play around with a few of the different geolocation datasets that I have (I actively keep three). Although it is paid service, they do offer a free plan for smaller datasets, which is perfect for your Moves data. Again, you’ll have to upload your places.csv file to a new table once you set up your account. Once the data is uploaded there are quite a few different map visualization wizards you can use to view your data in different ways. Pesonaly I like playing with the “Torque” visualization that gives you a real feeling of space-time to your data.
TileMill is an interactive map design tool from the folks over at Mapbox. If you’re looking to create custom maps with your data that you can format, style, and share then this is a wonderful tool to use. At first glance it’s a little daunting because it looks like a mashup of a CSS editor and map tool. That actually gives it the unique power to drive customization. Don’t be afraid, it’s not too hard to get started with. Mapbox has provided a great “crashcourse” to get you started with importing data, saving it as a new layer on your map, and then manipulating how it looks on your screen. If you want to go just a bit farther you can also add legends and informative popups to describe your data points. Mapbox also offers a free hosting plan if you want to share your interactive maps on a webpage. For example check out my MovesMap here, where I added a quick styling to manipulate the point size in relation to the time spent at a location.
Hopefully you’ve learned something new from this. If you map your Moves data (or any other geolocation data) we want to see it! Leave a link in the comments, post it in the location mapping thread on the QS Forum or get in touch on twitter!
We’ve all come face to face with tracking some aspect of our life only to realize that we’re not quite sure how to get started. Enrico Bertini encountered this roadblock when he began thinking about tracking the amount of time he spends engaging in “focused work.” As an information visualization researcher at NYU he decided on a simple rule that would give him the most accurate data that represented his interests: if it wasn’t tracked then it wasn’t focused work. In this talk, given at the New York QS meetup group, Enrico explains his process and shares his findings (including some great visualizations).
Slides available here.
(Editor’s Note: Enrico also co-hosts a great podcast on data visualization and information design called Data Stories. I highly recommend listening. If you’re looking for a place to start try Episode 17: Data Sculptures.)
As you may know, we get excited when someone in our community uses interesting data visualizations to help tell their self-tracking story. Jana Beck is no exception. As a woman living with Type 1 diabetes she’s constantly learning how to better understand what her Dexcom data is telling her. In this talk, Jana follows up on her previous show&tell presentation with some new visualization techniques she’s using. If you’re interested in Jana’s methods be sure to check out her Github repository and her work with Tidepool.org.
(Editor’s Note: I very interested in Jana’s use of Chernoff faces for multivariate data visualization. If you’re using this type of visualization for your own data I would love to see it. Get in touch.)
We thought it would be nice to post David’s 138 days of activities visualization here. Make sure to watch his talk from the 2013 QS Global Conference to learn how he created this and what he’s learned from tracking his time.
This is my life during the past six months. Each square = 15 minutes. Each column = 1 day. This picture represents 138 days or 3,000+ activities.
- David El Achkar