User Experience Insights from Data Lens Usability

At Socrata we love data, not just the data that lives in our products but also the data about our products and the data about the people who use our products. If you're interested in this data and our user experience research efforts at Socrata, read on.
Highlights
Don't have a lot of time to dive into the research? Here's the skinny...
Citizen’s first impressions of the Data Lens experience are very positive. Data Lens enables most citizens to successfully find trends and insights in large datasets such as crime data. In comparison, citizens are unable to find similar trends and insights using the tabular view. Furthermore, one of the most significant ways a publisher can increase success rates for citizens is by simplifying the Data Lens such that it has fewer cards.
Study Details
In August 2015 we conducted a remote unmoderated usability test which included qualitative video capture using a 'think-aloud' technique. Our goals were to assess the user experience of Data Lenses, relative to the data sets tabular view. Tasks were representative of constituent's real questions about crime and safety elicited from a previous study. 
Our target participants included citizens with an interest in the topic of the data (in this case crime), familiarity with the topic area (in this case Chicago or San Francisco), a high school education, and no prior experience with open data sets. We had 65 participants divided across 5 testing conditions. The conditions were:
A quick note about the methodology: In order to conduct valid and representative UX research we apply standardized qualitative research methods, we represent values as approximations, and we triangulate our insights across various methods and sources. More details here.
Citizen’s first impressions of the Data Lens experience are very high.
Approximately 10% of sentiments were negative.
  • Why were most impressions positive? Primarily, citizens are surprised (and even delighted) to know that this information exists. Also, citizens care most about local and relevant data, and visualizations such as choropleth maps and timelines support this. In some cases we even saw citizens become emotionally moved by relevant and local crime data.
  • Why were a few impressions negative? Because some citizens are easily overwhelmed by data and charts. They want it to be more digestible, holistic, and scannable.
  • In contrast, first impressions of a tabular view of the data set are quite polarizing, some citizens love it however most severely disliked it. As such, citizens are much less likely to abandon Data Lens than tabular views.
Data Lens enables most citizens to successfully find trends and insights in data sets such as crime data.
  • Approximately 95% of target citizens can detect basic trends instantly using a simple Data Lens in about a minute. And approximately 75% of our target citizens can detect more complex trends involving cross filters using a simple Data Lens in under two minutes.
  • In contrast, none of the target constituents can accurately detect any trends within 5 minutes of browsing a tabular view of data. Analysts are also unable to complete the task in under 5 minutes because they need to first download the dataset (up to 45 minutes) and then perform the analysis by using a specialized tool or by writing a script (5-10mins).
Notes: A basic trend is one that involves reading a chart. E.g. ‘What is the most common type of crime?’. A complex trend is one that involves a cross filter on one chart followed by reading a second chart. E.g. ‘What neighborhood has the least amount of crime due to drugs?’ A simple Data Lens has four visualization cards.
One of the most significant ways a publisher can increase success rates for citizens is by simplifying the Data Lens to answer a specific set of questions with fewer cards.
  • Success rates increase by 25% across all tasks when using a simple Data Lens in comparison to a complex Data Lens. In our study, a simple Data Lens had four visualization cards while a complex one had ten or more, requiring scrolling to see all cards.
Other ways to increase success include:
  • Provide a concise but meaningful data set title and description 
  • Ensure all data labels are meaningful 
  • Expand most valuable and relevant cards (e.g. primary categories, maps, timelines) so that more labels are shown by default
In summary...
If we want to improve the experiences for more constituents, we need to make the experience simpler, more relevant, and more inviting. There are actions that the publisher can take, as well as improvements we can make to the user experience. In addition to what's mentioned above, we are exploring improvements to search, annotations and emphasis, and simplifying the filtering experience.
We are eager to create solutions to better meet the needs of citizens. 
Tell us what surprised you! Send us your questions. And collaborate with us in future research. We look forward to learning more from a broader range of customers and a broader range of end users. You can reach me at Tira.Schwartz@Socrata.com.
-Tira Schwartz, Principal UX Researcher