Visual Storytelling for data science
Using storytelling to enhance science communication.
Storytelling is as old as humanity itself, much older than even writing. Early humans used storytelling as a way to share news, to keep historical records, and to impart their wisdom and values on the later generations. We know for a fact that the Australian aboriginal people have used rock art and painted symbols to help them remember their stories, yet images in caves have been found all over the world, so maybe it is not a far stretch of the imagination to suppose that ancient peoples everywhere were using their art for this purpose.
“A need to tell and hear stories is essential to the species Homo sapiens — second in necessity apparently after nourishment and before love and shelter. Millions survive without love or home, almost none in silence; the opposite of silence leads quickly to narrative, and the sound of story is the dominant sound of our lives, from the small accounts of our day’s events to the vast incommunicable constructs of psychopaths.”
— Price, Reynolds (1978). A Palpable God, New York: Atheneum, p.3.
The invention of writing changed the ephemeral nature of oral stories to a somewhat more fixed format, and stories were spread over much greater distances. Stories are known to have been written on almost any surface imaginable, including carvings in bone and rock, markings in clay and pottery, woven into textiles and even written on human skin in the form of tattoos.
In every human culture, storytelling is abundant. The medium (pun intended) may change, but the function is similar, to convey information in a relatable format.
In this article we explore some uses of ‘visual’ storytelling for data science being employed at the eScience center.
The visual aspect
What then, is visual storytelling? In my view, the visual aspect is supportive of the storyline, and does not necessarily have to replace text. Of course, some stories can be completely told by mime; let me give you an example:
While this particular visual story is likely not going to make a lot of sense a couple of hundred years from now, I’m willing to bet that everyone understands it at the time of writing, and might even feel a deep emotional connection to the narrative.
But many stories are best told accompanied by visual elements, not replaced by them completely, after all… one picture is worth a thousand words.
To dive into visual storytelling for movies and TV would go too far off topic, but let me just recommend the absolutely gorgeous YouTube channel Every frame a painting if you’d like to know more.
As a final thought to further demonstrate the power of visuals in storytelling; I recently acquired a set of Story Cubes, the idea of which is to roll the dice, and use the images that come up in an improv story. It’s entertaining and very effective, ideas for stories come so easily when prompted by just a few pictures.
Visual Storytelling in Data Science
Armed with this new knowledge of the power of visual storytelling, how do we use it for good … data science?
When conducting research, the data collected can in itself be exciting and meaningful, but when it is presented as spreadsheets or bullet points it can be too boring or difficult to understand for the intended audience. Telling a visual story with data is not trivial however, depending on the level of expertise of your readers, you should choose your storyline structure well.
The excellent paper Narrative Visualization: Telling Stories with Data by Edward Segel and Jeffrey Heer provides some good insights on this. The main takeaway from this paper I’d like to share is this:
Author-driven VS reader-driven
Consider if you want to tell a ‘fixed’ story, or if you want the user to discover new stories based on the data you are providing. If telling a fixed story is what you want, an interactive slideshow would be a good option. Interactively ‘playing’ with the data on a slide-by-slide basis could build the reader’s trust in the author’s analysis, but the author’s conclusions are leading here.
In a more reader-driven discovery scenario, more interaction is definitely needed, and a martini-glass-structure is likely a good option. Provide a ‘narrow’ authored storyline at first to explain a possible interaction with the data for tutorial purposes, and then open up options to let the user discover new stories on their own. This allows the user to learn about the possibilities without getting overwhelmed, but gives them the freedom to discover their own stories.
Depending on your intended audience, a drill-down story structure could also be an interesting option. You provide a theme for all of the stories you present, but the user can choose which one they are interested in at their leisure. If your intended audience has enough expertise in both the subject matter and the interactive methods to display your data, letting them examine all of the interactive options for themselves from the start can be beneficial. Providing multiple starting points like this can allow for variations in user’s interests, which could then provide an inlet into other stories they would normally be less interested in.
eScience application of Visual Storytelling
At the eScience center, we are actively working on visual storytelling applications for scientific purposes. We have three recent examples:
Climate Science
In the European H2020 funded RECEIPT (REmote Climate Effects and their Impact on European sustainability, Policy and Trade) project, we use storytelling as a framework for scientific discovery, modeling and communication. Many complex interactions between climate change in remote (non-EU) areas of the world and their impact on the EU cannot be easily communicated to policy makers and the public. Yet, a drought in South America or a flood in Asia can have major impacts on the European economy, or on the food supply that reaches our tables. The storytelling application we are developing allows the RECEIPT scientists to create storylines that link map locations and map-based data (displayed with TerriaJS) with text, images and custom graphs (created with Vega).
In the RECEIPT climate impact story application, we are using the drill-down-story structure. We Provide the common theme of climate changes in remote areas with impact on the EU, but we are letting the user decide what areas of the globe and/or what sectors they are most interested in, be that Agriculture, Finance, International Cooperation, Manufacturing or Coastal Infrastructure. This should provide an in for many different users, hopefully educating many potential users about the possible impacts of climate change.
Archeology and Art History
In the Via Appia: Revisited project, we are developing a storytelling application for 3d pointclouds (using potree.js). The goals of the project are to show the history of art and the (virtual) archeological reconstruction of monuments on the Via Appia Antica, an ancient Roman highway near Rome itself.
We will make two versions of this application with different storyline structures. The first instance will be a museum exhibit, where users will interact with physical forward and back buttons to interact with the application. We’ve chosen the interactive slideshow approach for the story structure to simplify the user interface and make sure nothing can go wrong, as well as to allow our authors (the researchers and artists responsible for the exhibit) to carefully craft the narratives.
The second instance of the application will be a public website, where users from the web can explore the Via Appia on their own. To guide the users towards interesting discoveries, we will adopt the martini-glass structure here. First, we will offer the same storylines as in the museum exhibit, but after these storylines have finished, we open up the interactivity and will let users explore the site on their own.
Medical Imaging
Another application area for visual storytelling we explored is medical imaging. The goal of this project was to see if we could improve the communication and support the building of trust between medical experts using Visual Storytelling techniques.
Our starting point was Vistories / CLUE, a library that uses interaction provenance as a basis for the authoring of Visual Storylines. The idea was that there could be an opportunity to increase the level of trust between a decision maker (the doctor planning treatment) and an expert doing analysis (a radiologist in the lab) by allowing the analysing expert to author a report directly from the tools used to analyse the images, with preservation of the links back to the data. This could then allow the decision maker to directly access the original analysis, and even directly try out alternate hypotheses or measurements. In this application, the martini-glass structure is also used to guide the users at first, but allow more freedom afterwards.
In the sister-blog to this one, we’d like to tell you more about the use of provenance for these (and other) purposes. Please read on here.
In conclusion
Combining data, narratives and visuals can bring powerful messages forward in a way that fact, numbers and data alone can not. Visual stories allow us to be more inclusive, to connect better to our audience, and most importantly, to be more memorable.
At the eScience Center, we do really like to be remembered by both the scientists that we work with, and by their intended audience as well. Visual storytelling is one of the ways in which we try to achieve this.