The sub-national time series dataset hosted there is the first data feed I am using to test EbolaMapper against.

They have a very nice, interactive ebola dashboard. The story of how the dashboard came to be is impressive but I want to work to make that tortured path not have to be traveled for future outbreaks.

They do not have case mapped to locations over time. For that The New York Times is the best.

Sub-national time series data on Ebola cases and deaths in Guinea, Liberia, Sierra Leone, Nigeria, Senegal and Mali since March 2014

The dataset comes from The Humanitarian Data Exchange. Those HDX folks are doing great work. More on them later.

I came to this dataset via a long, convoluted hunt. The difficulty of this search has led me to understand that the problem I am working on is one of data, as well as of code. This will need to be addressed, with APIs and discoverability but for now it is time to move on to coding (finally).

After I concluded that the data was usable, I started poking around on its page on HDX a bit more. On the left side of the page there are links to various visualizations of the data. This is how I discovered Google’s Public Data Explorer which is quite a nice tool. Below is one view of the HDX data in the Explorer. Click through the image to interactively explore the data over at Google.
Also among the visualizations on the HDX page was, to my surprise, the NYTimes visualization. Low and behold that visualization credits their data source as the HDX:

Source: United Nations Office for the Coordination of Humanitarian Affairs, The Humanitarian Data Exchange

So, that is good enough for me: the data hunt over. It is time to code.

The Economist has a story, Off The Map, about the launch of MissingMaps.org which is a development out of OpenStreetMaps.

On November 7th a group of charities including MSF, Red Cross and HOT unveiled MissingMaps.org, a joint initiative to produce free, detailed maps of cities across the developing world—before humanitarian crises erupt, not during them.

I mention this here for multiple reasons.

1) OpenStreetMaps is a great resource and HOT has been active with the ebola response. OpenStreetMaps is the source for maps in the EbolaMapper project I am working on.

Although the current focus is the ebola outbreak, these open source tools that I am calling EbolaMapper can be easily repurposed for any future outbreaks, as they read their data via the generic <a href=”https://github.com/JohnTigue/EbolaMapper/wiki/Outbreak-Time-Series-Specification-Overview</a> APIs I am developing. Next time (and statistically that is likely to occur before 2020) there should be free, quality tools at the ready for people to quickly get started on outbreak monitoring without having to wait for large organizations to mobilize.

As I have gotten to know some of the folks who have been involved with responding to previous epidemic outbreaks, they sound like they are living through a nightmare version of Groundhog Day (Swine flu in 2009, SARS, etc.). Yet now this type of problem can be solved generic, mature, widely available Web technology i.e. it does not require complex novel technology that needs to be scaled massively. (On the other hand, we do need to be mindful that currently in Liberia “less than one percent of the population is connected to the internet.” [Vice News]).

With the current established culture of open source it would be shameful for this type of flatfooted, delayed response to occur again. We have the technology to enable local actors to immediately get started by themselves the next time there is an epidemic outbreak.

2) This is a perfect example of one way that funding in this weird space can be successful. Folks (private and public) trying to effectively allocate money can find open source and/or open data projects that are already working and then juice them with cash for scaling, which is always an aspect of the large success stories in open source.

This is a bizarre but exciting variant of the thinking of Steve Blank and the lean start-up folks as applied to open source business models. I say bizarre because the customers (those benefiting from public health and disaster relief projects like the ebola response) cannot pay and have no obvious monetizable value as users. Here the open-source community has found a successful model and now that it is proven out the funding organizations are providing the cash to accelerate tech development to scale, where normally that cash would come via a series A round with venture capitalists.

In many successful open source projects, tech companies are paying talent to produce code that will immediately be placed in essentially the public domain. The value of doing so is expertise status with paying customers and keeping that scarce talent in-house to service those customers.

As Marc Andreessen quipped on twitter:

“The best minds of my generation are thinking about how to make people buy support contracts for free software.” –Anonymous

But who is going to do the funding in disaster relief contexts, specifically for the maps and do so proactively? So, this is MissingMaps.org situations is great news.

3) This blog loves a good map visualization related to the ebola response. The Off The Map story has a neat one. In the above picture the red handle can be dragged left and right to see the before versus after map.

Rosling reported that in Liberia at the peak of the outbreak daily infections were about 75 per day and are now stuck at around 25 per day. He believes the current (second) stage of the outbreak could well be labeled as endemic , an intermediate level epidemic that will take some time to put out.

A statistic that he says is important is the reproductive number. At the peak of the outbreak it was almost 2.0; currently it is closer to 1.0. The point is that the reproductive number is a key stat that needs to be tracked.

Later, five minutes into the interview, he has a go at main stream media’s reportage, specifically the use of cumulative numbers:

It is a bad habit of media. Media just wants as many zeros as possible, you know. So, they would prefer them to tell that in Liberia we have had about 2,700 cases or 3,000 case. The important thing is that it was 28 yesterday. We have to follow cases per day. I can take Lofa province, for instance, that has had 365 cases cumulative and the last week it was zero, zero, zero, zero every day. That is really hopeful that we can see the first infected [county] is where we now have very low numbers because everyone is aware.

Notice that the NYTimes ebola viz uses cases-per-week. We can build out visualization tools which provide a similar level headed overview of a situation, which might even help to reduce anxiety in the public compared to cumulative-cases representations.

Take away: the two statistics he pointed out, cases-per-day and reproductive-number, will be visualized in this open source epidemic monitoring dashboard tool set being built out here.

That makes sense as Mike Bostock works at the Times. He is one the creators of D3.js which is the open source engine behind most of the gorgeous data-driven visualizations on the Web these days. If you have not yet seen it, the D3.js examples gallery is a whole lot of eye candy.

Take this outbreak visualization as confirmation that any open-source white label outbreak widget should be based on D3.js.

Another ebola factoid was reported by The Lancet:

During October, there were 21,037,331 tweets about Ebola in the USA, compared with 13,480 about Ebola in Guinea, Liberia, and Sierra Leone combined.

I want to produce both interactive and static viz, with hopefully the former being used to generate the latter. SVG is good for exporting static raster images to file or paper. D3.js uses SVG, so interactive D3.js-based visualizations should be able to export excellent static maps and charts (we will see). Some users of this information will be on limited machines so bandwidth-light static info should be readily available.

I have found very few highly interactive ebola visualizations. Please point out any that I have missed in the comments. The best three found so far are listed here.

The WHO’s ebola-data

PBS Frontline’s map: how the ebola outbreak spread through Africa

HealthMap’s Ebola timeline

All three’s features, pros, and cons are analyzed on the EbolaMapper wiki.

For example, The Economist is doing good work:

The curated links can be found on the EbolaMapper wiki.

Note: EbolaMapper is the working title for this project; really it is more like “Reusable Outbreak Monitoring Web Components for a Global Outbreak Monitoring Network Organization.” Right, so EbolaMapper is the working title until a better name comes along, if one did not just pass by a moment ago…

Please, in the comments, add any links you think should be considered.

That sort of thing is exactly what the world needs. As for technical implementation details, they made the right choice basing it on interactive JavaScript libraries rather than Flash. They planned to make the data available:

“Data will be made available for open access in the coming days. All data will be made available via open format downloads as well as through an open access API.”

Sadly the visualization’s development and, more importantly, its data seems to have stalled. The data was last updated on September 14, 2014 (almost three months old as I type this). The code is on github with the last commit on September 27th.

Currently, I cannot find anything more about the planned open access APIs. Unfortunately I could not find any licensing information in the repository. I want to move their vision forward but using open source and open data. Or let us think big: why do we even need to wait around for the WHO to come down from on high with the numbers? Surely there is a better way…

These guys obviously did good work. I will try to get in contact with them. I will report back in an future post.

Nonetheless, this thing (“ebola-data” is its name on github) is what inspired me. If only it were clearly licensed and the data was fresh (and licensed openly). If such a thing were widget-ized and made freely available the Web would have a many more eyeballs taking in quality visualizations of the crisis.

I am calling such a thing EbolaMapper. (The name is a bit misleading; the code is reusable in that there is nothing ebola specific about it but “Outbreak Visualization Widgets” is not a catchy at this time, even though that is what this is all about. I will get around to explaining all that in the long run.)

So, I started digging around and found out about the Ebola Open Data Jam that took place on October 18th. Well, that there is the open data problem getting worked on.

Next I found the Africa Open Data meetup. On the conference call of November 7th it became clear that data validation was the current step, blocking quality visualizations. What better way to validate the data than to run it through a visualization on a world map?

So, I’ve started a repository on github for EbolaMapper which will be a clean room re-implementation of the WHO’s ebola-data viz with a clear open source license (Apache 2.0). I will use some data found by the Ebola Open Data Jam. What is really needed is a standard for data interchange – APIs and such.

When EbolaMapper can do what the WHO ebola-data visualization does, and more, via data from a standard API, that will be the first major milestone in this project.

This thought has been bothering me for a while and now I’ve decided to try and help as I see a clear path forward on a specific tech issue: getting the numbers and dates out (lots more to come on this).

As the press has asked, Why Isn’t Silicon Valley Doing More to Fight Ebola? And, well, there have been some private financial donations big and small, the biggest being:

Making it rain is great and good on them. That money will go through the old and some new channels to buy “materials and services and provide swift action where it is needed”. What I see though is a need for better software (“have hammer, see nails” I guess) and new channels – Internet channels.

So, let’s see if the tao of open source can be brought to bear on this problem (a ludicrously outdated global outbreak monitoring infrastructure) and if some tech talent can be rallied to the cause.

• I am completely addicted to org-mode. The syntax org uses is kind of a markdown variant. It has become my native written English “dialect.” So, blogging in org syntax is a major goal. Org has excellent export capabilities. Perhaps exporting will not even come into play but it is a nice fallback to have in store.
• When architecting, I love to use static content as much as possible so a static site generator seems like the right type of machinery to choose for blogging.
• I’m all about that open source so GitHub is in the picture. GitHub uses Jekyll to host static pages.

So, Plan A is: org to Octopress to Jekyll, with Octostrap to make it pretty via Bootstratp. Here, Bootstrap is dailed in to look a lot like GitHub. So, that is Octopress looking like Github via Bootstrap. Let’s call it… octopussy in deference to Oxley’s original intent, more on that later.