Visual Storytelling for data science

Maarten van Meersbergen
Netherlands eScience Center
8 min readJun 7, 2021

Using storytelling to enhance science communication.

Aboriginal Rock Art — X-Ray Style
Gabarnmung, Arnhem Land, Northern Australia

Storytelling is as old as humanity itself, much older than even writing. Early humans used storytelling as a way to share news, to keep historical records, and to impart their wisdom and values on the later generations. We know for a fact that the Australian aboriginal people have used rock art and painted symbols to help them remember their stories, yet images in caves have been found all over the world, so maybe it is not a far stretch of the imagination to suppose that ancient peoples everywhere were using their art for this purpose.

“A need to tell and hear stories is essential to the species Homo sapiens — second in necessity apparently after nourishment and before love and shelter. Millions survive without love or home, almost none in silence; the opposite of silence leads quickly to narrative, and the sound of story is the dominant sound of our lives, from the small accounts of our day’s events to the vast incommunicable constructs of psychopaths.”

— Price, Reynolds (1978). A Palpable God, New York: Atheneum, p.3.

Left: Probable Yuezhi soldier in red jacket and trousers, in the Sampul tapestry. Embroidered in Hellenistic style, with motif of a centaur, 1st century AD, Sampul, Ürümqi Xinjiang Region Museum. Right: The Ipuwer Papyrus (officially Papyrus Leiden I 344 recto) is an ancient Egyptian papyrus made during the Nineteenth Dynasty of Egypt. It contains the Admonitions of Ipuwer, an incomplete literary work whose original composition is dated no earlier than the late Twelfth Dynasty of Egypt (c.1991–1803 BCE) — in the Dutch National Museum of Antiquities in Leiden, Netherlands.
A section of the Bayeux Tapestry, an embroidered cloth nearly 70 metres (230 ft) long and 50 centimetres (20 in) tall that depicts the events leading up to the Norman conquest of England concerning William, Duke of Normandy, and Harold, Earl of Wessex, later King of England, and culminating in the Battle of Hastings.
St. Aemilian, ivory plaque with glass inlay by Master Engelram and his son Redolfo, Spanish (Castilian), 1060–80; in the Metropolitan Museum of Art, New York City.

The invention of writing changed the ephemeral nature of oral stories to a somewhat more fixed format, and stories were spread over much greater distances. Stories are known to have been written on almost any surface imaginable, including carvings in bone and rock, markings in clay and pottery, woven into textiles and even written on human skin in the form of tattoos.

In every human culture, storytelling is abundant. The medium (pun intended) may change, but the function is similar, to convey information in a relatable format.

In this article we explore some uses of ‘visual’ storytelling for data science being employed at the eScience center.

The visual aspect

What then, is visual storytelling? In my view, the visual aspect is supportive of the storyline, and does not necessarily have to replace text. Of course, some stories can be completely told by mime; let me give you an example:

https://xkcd.com/2459/

While this particular visual story is likely not going to make a lot of sense a couple of hundred years from now, I’m willing to bet that everyone understands it at the time of writing, and might even feel a deep emotional connection to the narrative.

But many stories are best told accompanied by visual elements, not replaced by them completely, after all… one picture is worth a thousand words.

A still from the Star Trek: Deep space 9 episode “Far beyond the stars”, which is based around science fiction writers in the 1950’s who write entire science fiction stories based on cover artwork. As an aside: this episode is great in so many ways, most prominently in its portrayal of racism in the United States, but the idea of stories being born from artwork has always stuck with me.

To dive into visual storytelling for movies and TV would go too far off topic, but let me just recommend the absolutely gorgeous YouTube channel Every frame a painting if you’d like to know more.

As a final thought to further demonstrate the power of visuals in storytelling; I recently acquired a set of Story Cubes, the idea of which is to roll the dice, and use the images that come up in an improv story. It’s entertaining and very effective, ideas for stories come so easily when prompted by just a few pictures.

Story Cubes

Visual Storytelling in Data Science

Armed with this new knowledge of the power of visual storytelling, how do we use it for good … data science?

When conducting research, the data collected can in itself be exciting and meaningful, but when it is presented as spreadsheets or bullet points it can be too boring or difficult to understand for the intended audience. Telling a visual story with data is not trivial however, depending on the level of expertise of your readers, you should choose your storyline structure well.

The excellent paper Narrative Visualization: Telling Stories with Data by Edward Segel and Jeffrey Heer provides some good insights on this. The main takeaway from this paper I’d like to share is this:

Author-driven VS reader-driven

Consider if you want to tell a ‘fixed’ story, or if you want the user to discover new stories based on the data you are providing. If telling a fixed story is what you want, an interactive slideshow would be a good option. Interactively ‘playing’ with the data on a slide-by-slide basis could build the reader’s trust in the author’s analysis, but the author’s conclusions are leading here.

Interactive slideshow — Martini glass structure — Drill down story, images from Narrative Visualization: Telling Stories with Data by Edward Segel and Jeffrey Heer

In a more reader-driven discovery scenario, more interaction is definitely needed, and a martini-glass-structure is likely a good option. Provide a ‘narrow’ authored storyline at first to explain a possible interaction with the data for tutorial purposes, and then open up options to let the user discover new stories on their own. This allows the user to learn about the possibilities without getting overwhelmed, but gives them the freedom to discover their own stories.

Depending on your intended audience, a drill-down story structure could also be an interesting option. You provide a theme for all of the stories you present, but the user can choose which one they are interested in at their leisure. If your intended audience has enough expertise in both the subject matter and the interactive methods to display your data, letting them examine all of the interactive options for themselves from the start can be beneficial. Providing multiple starting points like this can allow for variations in user’s interests, which could then provide an inlet into other stories they would normally be less interested in.

eScience application of Visual Storytelling

At the eScience center, we are actively working on visual storytelling applications for scientific purposes. We have three recent examples:

Climate Science

In the European H2020 funded RECEIPT (REmote Climate Effects and their Impact on European sustainability, Policy and Trade) project, we use storytelling as a framework for scientific discovery, modeling and communication. Many complex interactions between climate change in remote (non-EU) areas of the world and their impact on the EU cannot be easily communicated to policy makers and the public. Yet, a drought in South America or a flood in Asia can have major impacts on the European economy, or on the food supply that reaches our tables. The storytelling application we are developing allows the RECEIPT scientists to create storylines that link map locations and map-based data (displayed with TerriaJS) with text, images and custom graphs (created with Vega).

In the RECEIPT climate impact story application, we are using the drill-down-story structure. We Provide the common theme of climate changes in remote areas with impact on the EU, but we are letting the user decide what areas of the globe and/or what sectors they are most interested in, be that Agriculture, Finance, International Cooperation, Manufacturing or Coastal Infrastructure. This should provide an in for many different users, hopefully educating many potential users about the possible impacts of climate change.

A screenshot from the RECEIPT prototype visual storytelling interface.

Archeology and Art History

In the Via Appia: Revisited project, we are developing a storytelling application for 3d pointclouds (using potree.js). The goals of the project are to show the history of art and the (virtual) archeological reconstruction of monuments on the Via Appia Antica, an ancient Roman highway near Rome itself.

We will make two versions of this application with different storyline structures. The first instance will be a museum exhibit, where users will interact with physical forward and back buttons to interact with the application. We’ve chosen the interactive slideshow approach for the story structure to simplify the user interface and make sure nothing can go wrong, as well as to allow our authors (the researchers and artists responsible for the exhibit) to carefully craft the narratives.

The second instance of the application will be a public website, where users from the web can explore the Via Appia on their own. To guide the users towards interesting discoveries, we will adopt the martini-glass structure here. First, we will offer the same storylines as in the museum exhibit, but after these storylines have finished, we open up the interactivity and will let users explore the site on their own.

An screenshot of the Via Appia: Revisited (early alpha) application

Medical Imaging

Vistories / CLUE — Gapminder visualization

Another application area for visual storytelling we explored is medical imaging. The goal of this project was to see if we could improve the communication and support the building of trust between medical experts using Visual Storytelling techniques.

Our starting point was Vistories / CLUE, a library that uses interaction provenance as a basis for the authoring of Visual Storylines. The idea was that there could be an opportunity to increase the level of trust between a decision maker (the doctor planning treatment) and an expert doing analysis (a radiologist in the lab) by allowing the analysing expert to author a report directly from the tools used to analyse the images, with preservation of the links back to the data. This could then allow the decision maker to directly access the original analysis, and even directly try out alternate hypotheses or measurements. In this application, the martini-glass structure is also used to guide the users at first, but allow more freedom afterwards.

In the sister-blog to this one, we’d like to tell you more about the use of provenance for these (and other) purposes. Please read on here.

In conclusion

Combining data, narratives and visuals can bring powerful messages forward in a way that fact, numbers and data alone can not. Visual stories allow us to be more inclusive, to connect better to our audience, and most importantly, to be more memorable.

At the eScience Center, we do really like to be remembered by both the scientists that we work with, and by their intended audience as well. Visual storytelling is one of the ways in which we try to achieve this.

--

--

Published in Netherlands eScience Center

We’re an independent foundation with 80+ passionate people working together in the Netherlands’ national centre for academic research software.

Written by Maarten van Meersbergen

eScience Research Engineer at Netherlands eScience Center, focus on information- and scientific visualization

Responses (1)