Perspective Chapter: EnsembleDashVis Views and Volunteers – A Retrospective and Early History

Abstract

This paper offers a retrospective history of the early development stages of EnsembleDashVis, a visualization dashboard specifically crafted to support modelers in interpreting a simulation model utilized to forecast COVID-19 trends. The volunteer effort behind this dashboard was collaboratively contributed with the Scottish COVID-19 Response Consortium (SCRC), with the objective of enabling an enhanced comprehension of the complex dynamics of the pandemic through modeling of COVID-19 data collected by NHS Scotland during the first wave of the outbreak. This retrospective chronicles the design and development journey of the system, guided by feedback from domain experts, all taking place amidst the exceptional circumstances of an unprecedented pandemic. The outcome of this volunteer work is a streamlined relationship discovery process between sets of simulation input parameters and their respective outcomes, which leverages the power of information visualization and visual analytics (VIS). We hope that this retrospective will serve as an insightful resource for future effort, in VIS for pandemic and emergency responses and promote mutually beneficial engagement between scientific communities.

1. Introduction

The Scottish COVID-19 Response Consortium (SCRC) [1], in collaboration with the Royal Society’s call to action in March 2020, has taken a proactive approach to address the need for enhanced epidemiological models of COVID-19 transmission. This joint volunteer effort, known as Rapid Assistance in Modeling the Pandemic (RAMP) [2], aims to foster a deeper understanding of the consequences associated with various exit strategies from lockdown measures. Moreover, this consortium attracted the involvement of distinguished scientists and experts from diverse organizations both within the United Kingdom and abroad, thus augmenting the collective knowledge base and ensuring comprehensive expertise in specialized domains (Figure 1).

Figure 1.

A timeline of the events between mar 2020 and the end of our volunteer work on 19 may 2021. The upper section include policy changes during the time span, the lower section includes project developments and meetings. Milestone events are shown in red.

RAMPVis [3] is a group of researchers specialized in Data Visualization and Visual Analytics (abbreviated as VIS). The group voluntarily came forward to contribute its specialized skills and knowledge in order to provide valuable support to the SCRC modelers. The term modelers used here refers to the SCRC researchers who were actively engaged in the development of epidemiological models in SCRC. This target user group predominantly includes experts in domains such as mathematics, statistics, and epidemiology.

Serving as the volunteer team responsible for providing visualization support to one of the epidemiological models developed by the SCRC modelers [4], our main objective is to provide VIS researchers and practitioners with valuable insights gained from our research and development (R&D) activities conducted during the COVID-19 pandemic. In an effort to predict the potential impact of diverse interventions, modelers have actively utilized COVID-19 data, employing a method known as Uncertainty Quantification (UQ). This process seeks to measure uncertainties through the application of mathematical models and simulations. However, modelers are faced with significant challenges, including the aspects of expert elicitation and effective communication. In other words, there is a need for software engineering efforts coupled with visualization to provide support for validation and verification tests of models, and to create efficient workflows between modelers and researchers from other disciplines (Figure 2) [5].

Figure 2.

The organization of researchers from the SCRC and RAMPVis. The SCRC modeling team is responsible for developing the epidemiological models leveraging different modeling techniques. The RAMPVis team provides visualization support to the SCRC modeling team, by establishing four VIS volunteer teams who work on the actual development under the guidance of the RAMPVis team.

In addressing these hurdles, Data Visualization and Visual Analytics (VIS) emerge as a potent tool, offering the capacity to significantly enhance and streamline their collaborative workflows [6]. While our work may not have showcased the state-of-the-art VIS techniques, it effectively delivered rapid and practical VIS support to the modelers during an exceptional and demanding time.

Our contribution is an early history of our volunteer response from a software engineering and visualization perspective. We present the earliest stages of the visualization dashboard, EnsembleDashVis, developed during the pandemic, aiming to assist the modelers in interpreting an Approximate Bayesian Computation Sequential Monte Carlo (ABC-SMC) inference model that they have developed using COVID-19 data collected during the first wave of the outbreak in Scotland [7]. Much of this effort and the reasoning behind this volunteer work was never documented.

1.1 Unconventional software development

The visualization software created in this project was developed under unconventional and unprecedented circumstances. One of the distinctive features of this software project was the significant level of uncertainty encountered at the project’s inception. The following aspects were unknown at the project outset:

• An unknown a priori requirements specification: We did not know what the user requirements and expectations were.

• An unknown project team: The members of the project team were unknown and/or had no previous history of collaboration. We only knew the leader of the visualization team, Prof Min Chen. In addition, the project team was dynamic with new members joining throughout.

• Unknown data characteristics: We did not know what the simulation data was at the start of the project.

• An unfamiliar work environment: The collective work environment landscape shifted to a work-at-home model which was new to the team at the time.

While arguably, these characteristics could describe other software engineering projects, we believe that the uncertainty in this particular case was unusually high. All aspects of this project had the feel of “laying down the tracks as the train was running”.

2. Background and related work

VIS has been widely utilized in critical applications such as emergency responses and healthcare, assisting public officials and decision-makers in understanding intricate datasets and extracting useful, actionable insights from them [8]. VIS has also played a prominent role in disseminating COVID-19 information through various media channels. It has played a substantial role in enhancing public communication, making it more efficient and clear, thereby fostering a wider comprehension of the crisis [9].

In our work, our primary objective was to extend support through VIS to two distinct user groups. Firstly, the statisticians, who could significantly benefit from VIS in comprehending their models more effectively and fine-tuning them. Secondly, to the epidemiologists, whom VIS could assist in interpreting the outcomes of these computational models. Our outcomes are later included in multiple publications [4, 10, 11, 12]. The early stages functioned as the preliminary VIS prototype, shaping a portion of their respective studies. We refer the reader to Preim and Kai for an overview of VIS applied in the context of public health [13].

2.1 VIS for emergency response

Previously, we described related work that focuses on the use of VIS in emergency response. We refer the readers to the related work section in Chen et al. [4]. The aforementioned literature review laid the foundation and was conducted prior to the development of our study in 2020.

Maciejewski et al. [14] develop a VIS toolkit to analyze the effect of decision measures enforced during a simulated pandemic, the tool was later utilized by the Indiana State Department of Health during an outbreak of H1N1 (swine flu). Ribicic et al. [15] leverage VIS with the intention of delivering real-time feedback derived from flood simulations to non-expert users, while Konev et al. [16] use VIS to support decision-making in flooding scenarios.

Jeitler et al. [17] use VIS to analyze social media data to aid rescue teams, specifically in terms of optimal allocation of resources during emergency response situations. Similarly, Nguyen and Dang [18] harness social media data, paired with VIS, to facilitate and improve post-earthquake resource allocation and rescue effort.

In contrast to the majority of previous studies mentioned here that generally focus on preparing for future emergencies, our work was undertaken during the COVID-19 pandemic as a rapid response to a then current and ongoing emergency.

2.2 VIS for COVID-19 data modeling

In the rest of the section, we focus on the use of VIS to analyze the computational modeling of COVID-19 data. These studies were not published nor available to us during the development of the work we present here (from July 2020 to April 2021). In fact, the use of VIS in epidemiological modeling was rare, the modelers might have been unaware that they had such a potent instrument readily available [4].

He et al. [19] developed an SEIR (Susceptible, Exposed, Infected, and Recovered) model for spread prediction by leveraging COVID-19 data obtained from the Hubei province in China. They employed a variety of 2D plots to estimate the parameters of the model and interpret the results that the model yielded. Godio et al. [20] took the same approach in developing an SEIR model for the Lombardy region in Italy.

The IHME COVID-19 Forecasting Team [21] take the application of data visualization (VIS) a step further in their development of the SEIR model for accessing social distance mandates, they extend the use of VIS to include choropleth and violin plots, and small multiples for 2D plots.

Chinazzi et al. [22] develop a model to simulate the effectiveness of international travel restrictions in containing the spread of COVID-19. In addition to the use of 2D plots to refine their models, they also utilize a range of geospatial approaches. This enabled them to more effectively interpret the results generated by their models. The use of geospatial visualizations is also adopted by Alvarez Castro and Ford [23] in their development of a model for analyzing transmission in a university campus in the UK.

Studies have also been introduced which focus on the individual level, examining the transmission chain from person to person. Antweiler et al. [24] collaborated with public health departments in Germany and introduced a novel visual analytic method to identify clusters of COVID-19 infections in contact tracing networks. Meanwhile, Baumgart et al. [25] presented a visualization system designed to explore and analyze the pathways of pathogen transmission within hospitals. The system leverages linked views, including a transmission pathway view inspired by storyline visualization, aiming for efficient and intuitive contact tracing.

In contrast to these studies that highlight the efficacy of VIS in supporting the computational modeling of COVID-19 data with a primary focus on model development, as they are formulated by the modelers, our study takes a different approach. We focus our attention on exploring VIS as a potent tool that can significantly improve the computational modeling of COVID-19 data, all viewed through the unique lens of a VIS practitioner.

3. Data description

The data used in our work includes simulation parameters and outcomes from an <ac-ABC-SMC > inference model [26] developed by a group of modelers from Durham University, the University of Edinburgh, the University of Exeter, the University of Glasgow, and the London School of Hygiene & Tropical Medicine. The pandemic data used for the simulation was collected by NHS Scotland during the first wave of the outbreak in Scotland spanning a period of 59 days [7].

The model was built to analyze the pandemic data and infer the parameters of the model that best fit the data. The model accepts 16 input parameters (see Table 1), and a random seed facilitates the generation of 160 distinct sets of configurations for these input parameters. The model then employs these configurations as the initial input to perform 1000 simulation iterations. As the outcome of these simulations, 160 sets of predictions are generated (Figure 3), each containing 13 output parameters, as shown in Table 2.

Name	Description
T_lat	Mean latent period (days)
juvp_s	Probability of juvenile developing symptoms
T_inf	Mean asymptomatic period (days)
T_rec	Mean time to recovery if symptomatic (days)
T_sym	Mean symptomatic period prior to hospitalization (days)
T_hos	Mean hospitalization stay (days)
inf_asym	Reduction factor of infectiousness for asymptomatic infectious individuals
p_inf	Probability of Infection
p_hcw	Probability of Infection (Healthcare Worker)
c_hcw	Mean number of Healthcare Worker contacts per day
d	Proportion of population observing social distancing
q	Proportion of normal contact made by people self-isolating
p_s	Age-dependent probability of developing symptoms
rrd	Risk of death if not hospitalized
lambda	Background transmission rate
K	Hospital bed capacity

Table 1.

16 input parameters for the ABC-SMC inference model.

As constant parameters such as $K$ and $rrd$ do not affect the simulation results, they are not rendered in our visual designs.

Figure 3.

An illustration of the flow from the input parameters to the prediction results. 160 sets of input parameters are used to perform 1000 simulation iterations, resulting in 160 sets of prediction results.

Name	Description
iter	The simulation number.
day	The day number.
age_group	The age group of the population.
S	Number of susceptible individuals (not infected).
E	Number of infected individuals but not yet infectious (exposed).
E_t	Number of exposed individuals and tested positive.
I_p	Number of infected and infectious symptomatic individuals but at pre-clinical stage (show yet no symptoms).
I_t	Number of tested positive individuals that are infectious.
I	Number of infected and infectious asymptomatic individuals.
I_s	Number of infected and infectious symptomatic individuals.
H	Number of infected individuals that are hospitalized.
R	Number of infected individuals that have recovered from the infection.
D	Number of deceased individuals due to the disease.

Table 2.

13 output parameters from the simulation performed by the ABC-SMC inference model.

Upon receiving the data, we consulted the modelers to gain insights into the conventional workflow they employ for data processing, as well as the significance and the underlying meaning associated with each input and output parameter. As constant parameters such as $K$ and $rrd$ do not affect the simulation results, they are not rendered in our visual designs.

It is worth mentioning that after plotting the output data using a line chart, an error was immediately spotted, see Figure 4, where an unusual spike can be observed on day 20. The modelers were notified and the bug was fixed. However, the rectified output file was never made available to us.

Figure 4.

A line chart depicting the model outcomes. The x-axis of the chart corresponds to the number of days since the first date in the Scottish dataset, while the y-axis represents the population. To differentiate between different population categories, a color map was incorporated: Susceptible, exposed, hospitalized, recovered, death, asymptomatic, and symptomatic. The focus+context technique is used here to highlight the outcome of the current configuration, while the gray lines represent other outcomes. On day 20, there is an unusual spike which was later identified as caused by an error in the model.

4. EnsembleDashVis

This section presents the development of EnsembleDashVis from its technology and design and interaction techniques. We then present the history behind our fully virtual collaboration between volunteer researchers from multiple UK institutions. Being one of the four VIS volunteer teams, we received guidance from the RAMPVis team via regular virtual meetings. The RAMPVis team communicated with the SCRC modeling team regularly and provided us with important information and data. We chronicle the development of different views of the data, the order in which they were introduced, and the reasons and motivations at the time. In 2020 we were all in an unprecedented and unfamiliar situation, thus, some of our decisions were ad-hoc.

4.1 An unconventional software development cycle

A common agile software development life-cycle consists of five stages: (1) requirements specification, (2) software design, (3) implementation, (4) testing, (5) documentation [27]. And these five stages iterate repeatedly until the software project is finished. However, this project deviated significantly from the standard agile software engineering model.

4.1.1 Knowledge exchange

This project, as well as all of the other visualization projects we have collaborated on, start with a phase more aptly named knowledge exchange (KE). This is due the fact that the domain experts do not have a background in visualization, thus they do not know what the options are in terms of visual analysis. As a result of this absence of visualization expertise, the KE phase (which replaces the standard requirements specification phase) involves two sub-phases:

4.1.2 From domain experts to visualization team

The discussion starts with the visualization team asking a series of questions to the domain experts. These questions are typically: (1) What is the data you have collected? (2) Why did you collect this data? (3) What questions were you trying to answer with the (simulation in this case) data? (4) What information were you hoping to obtain as an outcome from your data collection process? (5) Can you describe the characteristics of your data in more detail? After the visualization team has gathered enough of the first round of knowledge the next phase of the KE process can begin.

4.1.3 From visualization team to domain experts

Since the domain experts, in this case the simulation experts, do not have a background in visualization they look to the visualization team to make recommendations to them in term of what visual analysis designs might make sense. Thus the visualization team typically discusses options in terms of graphical displays that might help the domain experts answer the questions posed in the previous sub-phase. In essence the KE process flows in the other direction. The visualization team essentially educates the domain experts on visual analysis options that they may not be familiar with. After this discussion, the actual next phase of the software engineering lifecycle can begin.

The software design and implementation phases are the same as in the typical agile model of software development.

4.1.4 Testing and evaluation

Instead of the conventional testing phase of a typical agile development model, this project and our other collaborative visualization projects, undergo a more aptly described testing and evaluation (TE) phase. Instead of the emphasis on extensive testing on a wide range of cases, our visualization software undergoes and extensive evaluation by the domain experts. Specifically, they carefully evaluate if and how the software can be used to answer their domain-specific questions or hypotheses. They will ask for a demonstration of precisely how it can be used for their specific application. Typically, when we demonstrate a version of the visualization software, the domain experts will ask several questions about how it works. And then, during the discussion new feature requests arise. Often these sessions are also characterized by feature creep [28]. The TE phase is usually fairly intense generating a lot of enthusiasm from the domain experts since they are seeing visualization software that they have never seen before and thus a large number of feature requests arise from the meetings in this phase.

After the TE phase the cycle repeats interactively. In the visualization software development lifecycle, the requirements specification phase is replaced by the KE phase and the testing phase is replaced by the TE phase. This is because an adequate knowledge transfer and evaluation cannot be completed in only one cycle. The cycle repeats until the project ends, typically constrained by a funding period.

4.2 Technology and design

The development of EnsembleDashVis was carried out using a combination of web technologies, including HTML, CSS, and JavaScript. The dashboard was designed to be a web-based application, enabling it to be accessed from any device with a web browser. The dashboard was built using D3.js [29], which is a powerful and flexible library for creating visual data representations in web applications. D3.js provides a wide range of tools for creating interactive graphics, including support for a wide range of data formats, and a large number of built-in visual designs. The dashboard was designed to be responsive, allowing it to adapt to different screen sizes and orientations, and to be accessible, allowing it to be used by people with disabilities.

The dashboard was then hosted on Netlify [30], which provided unlimited credits to websites that were dedicated to sharing information about COVID-19. This allowed the dashboard to be accessed by anyone with an Internet connection, which was crucial during the pandemic for virtual collaboration.

The dashboard was designed to be easy to use, with a simple and intuitive interface that enables users to quickly and easily explore the data. It employs a modular design, with each view of the data being rendered as a separate component, allowing the dashboard to be easily extended and modified. Data is preprocessed by utility functions and stored in separate CSV files, which is then loaded into the dashboard when it is accessed (Figure 5).

Figure 5.

The structure of the actual code. Components are organized into separate files, with each file containing the code for a single view. Utilities contain the code for the data preprocessing and calculations. Data contains the metadata and preprocessed output by utilities. The source code is publicly available on GitHub, https://github.com/thevisgroup/EnsembleVis [31].

4.3 Interaction

In this section, we describe the interaction techniques that were incorporated into the dashboard to enable the modelers to explore the data and identify interesting patterns. Here we follow the Visual Information Seeking Mantra [32]: “overview first, zoom and filter, then details-on-demand”.

4.3.1 Overview first

Figure 6 shows the overview of the dashboard. The dashboard consists of five views: Figure 6A a parallel coordinates plot [33, 34] for all input configurations, Figure 6B a table view with glyphs for all input configurations, Figure 6C a parallel coordinates plot for simulation outcomes, Figure 6D a line chart for model predictions, and Figure 6E a scatterplot for Principal Component Analysis (PCA) [35] outcomes.

Figure 6.

The overview of EnsembleDashVis. The dashboard consists of five views: (Figure 6A) a parallel coordinates plot for all input configurations, (Figure 6B) a table view with glyphs for all input configurations, (Figure 6C) a parallel coordinates plot for simulation outcomes, (Figure 6D) a line chart for model predictions, and (Figure 6E) a scatterplot for principal component analysis (PCA) outcomes. The views are coordinated with each other, enabling the modelers to observe relationships between input and outcome through interactions.

Each view provides an overview of the corresponding data, supporting the modelers to quickly identify interesting patterns and outliers.

4.3.2 Zoom and filter

The parallel coordinates plot in Figure 6A, C allows the modelers to select a subset of input parameters via brushing to focus on interesting configurations. The table view in Figure 6B enables the modelers to sort configurations by individual input parameters via sorting. The scatterplot in Figure 6E enables the modelers to reduce the dimensionality and identify key parameters via brushing.

These interactions enable the modelers to quickly adjust the focus of the views and drill down into the details.

4.3.3 Details-on-demand

These views in Figure 6 are coordinated with each other, e.g., brushing on the input parallel coordinates plot in Figure 6A highlights the corresponding input configurations in both the table view Figure 6B and scatterplot Figure 6E. Focusing on a specific row in the table view Figure 6B renders the corresponding output data in both the output parallel coordinates plot Figure 6C and line chart Figure 6D.

These coordinated interactions enable the modelers to quickly identify interesting configurations and observe relationships between input parameters and model outcomes.

4.4 Meetings and milestones

In this section, we provide a detailed history of meetings and development milestones. Table 3 shows the list of meetings held throughout the entire volunteering period, detailing each meeting’s date, the attendees, and the milestones accomplished.

Date	Attendees	Milestones
July 2020	Dylan Rees, Elif Firat, Hui Fang, Min Chen, Qiru Wang, Rita Borgo, Robert Laramee, Tom Torsney-Weir	Volunteer team established.
Nov 2020	Cagatay Turkay, Hui Fang, Qiru Wang, Rita Borgo, Robert Laramee, Tom Torsney-Weir	First prototype.
Nov 2020	Ben Swallow, Hui Fang, Qiru Wang, Rita Borgo, Robert Laramee, Tom Torsney-Weir	First prototype feedback
Nov 2020	Cagatay Turkay, Elif Firat, Hui Fang, Rita Borgo, Robert Laramee, Qiru Wang, Tom Torsney-Weir	6GB of simulation data received. Second prototype.
Nov 2020	Cagatay Turkay, Elif Firat, Hui Fang, Robert Laramee, Qiru Wang	Third prototype.
Dec 2020	Cagatay Turkay, Hui Fang, Robert Laramee, Qiru Wang	All views implemented.
Dec 2020	Ben Swallow, Cagatay Turkay, Hossein Mohammadi, Hui Fang, Janine Illian, Michael Dunne, Peter Challenor, Qiru Wang, Richard Reeve, Robert Laramee, Thibaud Porphyre	Presentation to modelers.
Mar 2021	Cagatay Turkay, Elif Firat, Hui Fang, Rita Borgo, Robert Laramee, Qiru Wang	Further feedback from modelers.
May 2021	Ben Swallow, Cagatay Turkay, Hossein Mohammadi, Hui Fang, Janine Illian, Michael Dunne, Peter Challenor, Qiru Wang, Richard Reeve, Robert Laramee, Thibaud Porphyre	Final presentation to modelers.

Table 3.

The table shows the list of meetings held throughout the entire volunteering period, detailing each meeting’s date, the attendees, and the milestones accomplished.

4.4.1 Meeting #1 - July 2020

On 27 July 2020, amid the UK’s first national lockdown and stricter measures imposed by local authorities, we convened the initial virtual meeting with VIS researchers from King’s College London, Loughborough University, Swansea University, University of Nottingham, University of Warwick, and University of Oxford.

During the meeting, we received an overview of the SCRC and the responsibilities of the visualization volunteer team. Our assigned task was to create visual interfaces for the model, for the purpose of enabling the modelers to analyze the outcomes of the model.

Following the initial meeting, we engaged in email correspondence with the modelers to delve into the visualization requirements. The modelers shared a comprehensive list of parameters and model outcomes, along with the corresponding outcome data [7].

4.4.2 Commit #1 - Sep 2020

We proceeded to create an initial prototype of the visualization, which was subsequently reviewed by the modelers. Incorporating their input, we refined the prototype during our weekly internal discussions. On 14 Sep 2020, England introduced the ‘rule of six’, which banned any gatherings above six. On the same day, we made our first commit to a GitHub repository, signifying the commencement of our development¹. At the same time, we began preprocessing the data. A week after the initial commit, the UK witnessed the implementation of additional restrictions, such as mandatory work from home and a 10 PM curfew.

4.4.3 Meeting #2, view #1 - Nov 2020

On 5 Nov 2020, the first day of the second national lockdown in the UK, we completed the first view of the simulated input parameters, a parallel coordinates plot. See Figure 7. We chose to use a parallel coordinates plot as it is a common technique for visualizing multivariate data, and is particularly useful to explore relationships and patterns across multiple input parameters. Each axis in the plot represents an input parameter, the y-axis represents the value of the parameter, and each polyline represents one input configuration. The plot supports brushing and linking, enabling modelers to select a subset of input parameters to focus on interesting configurations. This followed by the second meeting with the RAMPVis team from other institutions, where we received feedback on the first view, on 6 Nov 2020. The response from the modelers to the parallel coordinates view was, in general, very positive. They are very interested in multivariate analysis and had not seen this visual representation before. More details are provided in Section 5 on domain expert feedback.

Figure 7.

The first visual design, a parallel coordinates plot depicting all 160 input configurations of the model, was completed on 5 Nov 2020. Each axis represents an input parameter, the y-axis represents the value of the parameter, and each polyline represents one input configuration. The table below shows the configuration details.

4.4.4 Meeting #3, view #2 - Nov 2020

On 11 Nov 2020, the group convened for the third meeting, where we received further feedback from the RAMPVis team on the parallel coordinates plot. As per the modelers’ requests conveyed via email, we incorporated a line chart to depict the model outcomes. See Figure 4. The x-axis of the chart corresponds to the number of days since the first date in the Scottish dataset, while the y-axis represents the population. Line chart and other classic visual designs are widely used by the modelers, they are familiar with these designs and can easily interpret the results. The line chart is coordinated with the parallel coordinates plot, enabling the modelers to select a subset of the input parameters and quickly identify the corresponding model outcomes. A focus+context technique is used to highlight the selected subset of the input parameters in the parallel coordinates plot.

4.4.5 Meeting #4, view #3 - Nov 2020

On 25 Nov 2020, the group convened for the fourth meeting, held just a day after the announcement of the gathering rules for Christmas in the UK. During the meeting, we received feedback from the RAMPVis team on the new view of the input parameters, a table with glyphs. See Figure 8. We incorporated this table view featuring glyphs to depict all 160 input parameter configurations, following discussions with the modelers. Each row represents an input configuration, and each column represents an input parameter. The table view enables the modelers to sort configurations by individual input parameters. Each parameter value is symbolized by a bar glyph, the color and length correspond to its deviation from the average value of 160 predictions.

Figure 8.

The table view depicting all 160 input parameter configurations. The view enables the modelers to sort parameter values and identify interesting configurations. Each row represents an input configuration, and each column represents an input parameter. Upon clicking on a row, the line chart in Figure 4 is updated to display the corresponding model outcomes. Clicking on the column header sorts the table by the parameter values.

The table view provides the functionality to sort the parameters according to their values and can be dynamically updated by brushing the parallel coordinates plot for the input parameters in Figure 7. The line chart in Figure 4 can be quickly updated to display the corresponding model outcomes by clicking on the configuration index in the table view.

4.4.6 Meeting #5 - Dec 2020

On 9 Dec 2020, a week after the end of the second national lockdown in the UK, with England facing a stricter three-tier restriction policy, the group convened for the fifth meeting. At this point, we still had not met with the modelers, all communications and discussions took place via email. The RAMPVis team decided to organize a meeting with the modelers to present our prototype for feedback.

4.4.7 Meeting #6, views #4 & 5, feedback #1 - Dec 2020

On 10 Dec 2020, we finally met with modelers from Durham University, the University of Edinburgh, the University of Exeter, the University of Glasgow, the London School of Hygiene & Tropical Medicine, for the first time. In contrast to sharing screenshots via email and deploying a website with a live view of our development (which they might not have been proficient in using), we delivered a live presentation, fielding numerous questions. The modelers were pleased with the dashboard, and a list of ad-hoc requirements was provided. Furthermore, we collected insightful feedback that we elaborate on in detail in Section 5.

1. The modelers found that the parallel coordinates plot is useful in identifying outliers, and requested the incorporation of another one for the model outcomes. Given that the outcome data mirrors the input in a multivariate format, employing a parallel coordinates plot could potentially be useful. We implemented this as shown in Figure 9.

2. The modelers requested that all the simulation results be displayed in the line chart, with the current one highlighted. This resembles their usual workflow for analyzing multiple simulation outcomes. We implemented this as shown in Figure 4.

3. The modelers requested the incorporation of a scatterplot to visualize the model outcomes, specifically a Principal Component Analysis (PCA) result obtained from another VIS volunteer team. The motivation behind this is to reduce the dimensionality and identify key parameters. We implemented this as shown in Figure 10.

4. The modelers requested all views to be coordinated with each other, enabling observation of relationships between input parameters and model outcomes through interaction.

   1. Brushing on the input parallel coordinates plot (Figure 7) highlights the corresponding input configurations in both the table view (Figure 8) and scatterplot (Figure 10).

   2. Brushing on the scatterplot (Figure 10) for input configurations highlights the corresponding input configurations in both the table view (Figure 8) and input parallel coordinates plot (Figure 7).

   3. Clicking on a specific row in the table view (Figure 8) renders the corresponding output data in both the output parallel coordinates plot (Figure 9) and line chart (Figure 4).

Figure 9.

A parallel coordinates plot depicting the model outcomes by age group. As requested by the modelers, the mean value of 160 predictions generated by each input configuration, as well as for each age group, was computed and rendered here. Each axis represents one variable from the outcome and its value. Each age group is mapped to a color, the dashed red line - - -represents the average value, and the dotted red line .......... Represents the standard deviation.

Figure 10.

A scatterplot depicting the PCA outcome from another VIS volunteer group, was added upon request by the modelers. Upon brushing, the selected configurations are highlighted in the table view in Figure 8.

Furthermore, we received the exciting news that initial funding had been successfully secured [36], which led to the transition of our volunteer work to a team of paid developers, who would continue with further implementation of the project.

4.4.8 Meeting #7, feedback #2 - mar 2021

On 25 Mar 2021, the UK was in the process of cautiously lifting its third national lockdown, the ‘rule of two’ was still in place. The group convened for the seventh meeting, where we received further feedback from the modeling team on our implementation. We detail the feedback in Section 5.

4.4.9 Last commit - Apr 2021

By 28 Apr 2021, more restrictive measures were abolished, although the prohibition on mixing between households was still in effect. On this day, we made our last commit to our GitHub repository. This act signified the completion of our volunteer work, as we had smoothly transitioned all tasks to a team of paid developers.

During the entire development process, our meetings were exclusively conducted virtually, and our communication relied heavily on email correspondence. Despite the lack of in-person interactions, we successfully met the initial requirements of the modelers and delivered a VIS solution that received very positive feedback from the SCRC modeling team.

4.4.10 Meeting #8, feedback #3 - May 2021

On 19 May 2021, the UK was viewing the light at the end of the Covid tunnel, weddings and funerals were still restricted to 30 people, and indoor gatherings of more than two households were still banned. The group convened for the eighth and final volunteer meeting. During this final meeting, a modeler joined and gave us some in-depth feedback on the influence our work had on their modeling process, as well as suggesting potential improvements. We detail the feedback in Section 5.

5. Domain expert feedback

In this section, we share the invaluable feedback collected from the modelers. Meeting #6 and #7 were held prior to the conclusion of our development, serving as an iterative process of refinement aimed at validating and improving our visual designs while ensuring their relevance and utility to domain experts. Meeting #8 was held after the conclusion of our development, functioning as a means to gather feedback on our work and to identify potential future work. Three domain experts in statistics from Durham University, the University of Exeter, and the University of Glasgow, were invited to join these meetings.

5.1 Summary of feedback

In this section, we provide a summary of the feedback collected from the domain experts during our meetings.

5.1.1 Appreciation for interaction and visualization design

The experts commended the visual designs for effectively depicting the relative importance of input parameters on model predictions, highlighting the utility of interactive graphics in understanding the significance of different inputs. The ability to visually present the connection between input and output parameters was particularly appreciated, emphasizing the value of visual techniques in elucidating the relationships between variables.

5.1.2 Identification of ineffective parameter combinations

The linked visual designs were recognized for their potential to help identify ineffective parameter combinations, aiding in the optimization and calibration process by revealing which combinations may not be useful. The ability to filter out redundant input parameter configurations was seen as beneficial for focusing on the most influential parameters, thereby reducing the dimensionality of the problem.

5.1.3 Potential for identifying model discrepancy

There was interest in the potential of visual designs to aid in identifying model discrepancies when observational data becomes available, highlighting the importance of visualizing observational data alongside model predictions.

5.1.4 Overview of input parameters and distributions

The table view was praised for providing a clear overview of input parameters and their distributions, enabling quick identification of influential parameters and possible adjustments, as well as the elimination of unnecessary complexities.

5.2 Detailed feedback

In this section, we present some of the original quotes collected from the domain experts during these meetings.

5.2.1 Domain expert #1 - professor in statistics, Durham University

On 25 March 2021, Meetings #6 and #7, we presented the dashboard through screen-sharing demonstrations, the domain expert appreciated the interactions provided by the visual designs in depicting the relative importance of different input parameters on the model’s predictions. “The visualizations are able to show how important a particular input is for a particular output.”

In addition, the ability to visually present the link between the input and output parameters. “The real interesting game here is the connection techniques to understand the relations between the input and output.”

The linked visual designs also potentially enable the domain expert to identify ineffective parameter combinations. “The different configurations is the sort of history of calibration and by looking at those visualizations you can start saying certain combinations may not be useful.”

The inclusion of a PCA plot was seen as a significant step towards dealing with feature selection. The expert suggested adding two further plots depicting, MDS and possibly ICA, to support model calibration using history matching. “to perform history matching MDS is really what we use. The PCA plot is already very informative … t-sne like methods are ill suited for the task.”

Furthermore, the domain expert also expressed interest in the potential of our visual designs to aid in identifying model discrepancy, when the observational data becomes available. “The visualization would be helpful in identifying model discrepancies when we eventually plot the observational data.”

5.2.2 Domain expert #2 - professor in statistics, the University of Exeter

On the same date, during Meeting #6 and #7, the domain expert was pleased with the ability of the visual designs to provide the potential to filter redundant input parameter configurations, enabling users to concentrate on the most influential configurations. “For particular input configurations after filtering, the visualization shows that some of the input parameters can be ignored, which reduces the dimensionality of the problem, and we can focus on the important parameters.”

The domain expert also noted the usefulness of the PCA plot and suggested to replace the method with MPCA [37] to further support the process of detecting implausible input values “One approach is to look for inputs configurations which would produce implausible outputs, we work with a sort of implausibility statistical measure”.

5.2.3 Domain expert #3 - assistant professor in statistics, the University of Glasgow

On 19 May 2021, Meeting #8, the domain expert praised the visual designs’ ability to provide a clear overview of the input parameters and their distributions. This enables them to quickly identify possible adjustments they can make to their input parameters, as well as to identify the most influential parameters. “The table view is really useful in showing how close those input parameters are to the threshold, which is very useful to understand affordability.”

The domain expert also noted that some overlapping distributions can be ruled out quickly via the interactivity provided by our visual designs, this enables them to eliminate unnecessary complexities and increase the overall efficiency of their model. “It’s fairly obvious that some of the parameters can be ruled out quite quickly, including some overlapping distributions.”

An avenue for future work, as unanimously identified by all three domain experts, involves integrating new visual designs to render and compare observational data against model predictions effectively.

6. Limitations

Due to the impact of the pandemic, the project was conducted in a fully virtual manner, with all meetings and discussions taking place online, between a large group of researchers from different disciplines and different institutions. In total, 33 VIS researchers and 7 modelers were involved in this volunteer work. The development was ad-hoc in some ways due to the unprecedented nature of the pandemic. This resulted in a number of limitations, which we will discuss in this section.

6.1 Lack of novel and advanced visual designs

Operating under a time constraint, the primary objective of our project centered on offering immediate visual analysis assistance to the modelers. Thus, we were unable to explore the inclusion of innovative and advanced visual design approaches. Instead, we integrated a series of classic views, such as line charts and scatterplots. These are visual designs commonly leveraged by modelers in their day-to-day research. Interestingly, the modelers welcomed the introduction of a less conventional (to them) visualization technique: parallel coordinates. They had never before employed this, and its introduction proved beneficial to their research. Consequently, they expressed a desire for the incorporation of an additional parallel coordinates to assist in the visualization of model outcomes.

We believe that this is a testament to the effectiveness of advanced visual designs in enhancing the modelers’ understanding of their models, this signals the possibility for future inclusion of more sophisticated visual designs.

6.2 Lack of formal requirements gathering

We were unable to meet with the modelers until a particularly late stage. Instead, we had to rely on email correspondence, which was arguably not as effective as face-to-face or even virtual meetings. In a traditional software engineering project, requirements are gathered through a series of meetings and discussions with end users. This did not occur in our case.

This resulted in a lack of proper requirement gathering, which in turn led to a number of challenges during the development process. For example, the modelers made ad-hoc requests to incorporate different views at different stages of the project, resulting in unexpected changes on the development side. This could have been avoided if we had a better understanding of their requirements from the beginning.

6.3 Dynamic group membership

The group membership was dynamic, with researchers joining and leaving the group at different stages of the project. This introduced some lack of continuity, as newcomers had to spend time to familiarize themselves with the project. Furthermore, members came from different disciplines, with different levels of expertise in visualization. This has resulted in a lack of consistency in the development process, as different members have different ideas on how to implement the views. The responsibility of each member, apart from the only developer in the group, was not clearly defined.

6.4 Uncertain project direction

The exact direction of the project was not clearly defined from the outset. Numerous details remained unknown to us during the development process, such as the exact purpose of the visualization, the target audience, and the end product. Consequently, the final product suffered from suboptimal utilization of screen-space, as additional views were requested, the implementation of a multi-view display design or collapsible views became time-constrained and unattainable.

6.5 Other technical limitations

Some additional technical limitations include:

• Real-time updating: Coupling the simulation with the visual rendering directly would have been very beneficial to the project, e.g., computational steering.

• Standardization: Standardization of the data format would be beneficial to all participants.

• Interpretability: A more formal evaluation of how interpretable our visual representations are would be beneficial, e.g., presenting complex epidemiological concepts in a clear and understandable manner to a wider audience.

7. Conclusions

In this paper, we present the stories behind the development of EnsembleDashVis, an interactive dashboard designed to visualize the input parameters and outcomes of an <ac-ABC-SMC > inference model used to analyze COVID-19 data collected during the first wave of the outbreak in Scotland.

Given the multitude of uncertainties and challenges during this exceptional period, a considerable amount of information was unavailable to us during the development process. It was only through the Scottish COVID-19 Response Consortium Stakeholder Report [38], published in late 2021, and various publications [4, 10, 11, 12] that unveiled the remarkable endeavors undertaken by other volunteer teams, that we gained additional insight and details.

We hope that our experience serves as a valuable source of insight into how VIS research and techniques can play a crucial role in emergency response initiatives and aid in effectively preparing for future emergencies, serving as an inspiration to future volunteer efforts.

Acknowledgments

The authors gratefully acknowledge the valuable guidance and feedback provided by Cagatay Turkay, Hui Fang, Rita Borgo and Thomas Torsney-Weir on aspects of visualization. We also sincerely appreciate the insightful feedback from Ben Swallow, Thibaud Porphyre, Hossein Mohammadi, Janine Illian, Michael Dunne, Ian Vernon and Peter Challenor, which significantly strengthened our work. Special thanks are due to Min Chen and Richard Reeve for their tremendous coordination efforts on this project.

This work was funded by the grant EP/S010238/2 from the Engineering and Physical Sciences Research Council (EPSRC).

Conflict of interest

The authors participated in the volunteer effort to support the Scottish COVID-19 Response Consortium (SCRC) during the COVID-19 pandemic, which is a non-profit initiative without any commercial intents. The authors declare no conflict of interest.