Getting Open Data down from the cloud to create real impact on the ground

In order to transform our food systems and make the way we grow, sell and consume food more sustainable we need data. We need data that is open and online.

Photo by Neil Howard

That means not only ensuring new data is stored and updated in publicly accessible databases, but that older data repositories are harvested from their offline C drives and uploaded also.

But once open data makes it to the cloud, what’s next?

How do we ensure open data actually makes an impact on the ground to benefit those who need it most?

One of the key efforts of the CGIAR Platform for Big Data in Agriculture is to systematically aggregate CGIAR wide data from those hard drives to online data repositories such as AgTrials, Dataverse, and GBIF – to name a few.

However, we are now starting to realize that, in order for data to be accessed and used by the right people, more needs to be done than just unlocking it.

There is a lot of good intent and enough financial resources to enable the much-needed food systems transformation, but progress has been slow going.

It’s clear that only intent and money are not enough.

For open data to be effective, it must also be usable.

Decision and policy makers are key in the journey for the transformation of our food systems, and all this data that is being freed up needs to be accessible for them to easily connect with it.

In January earlier this year, CGIAR and Microsoft co-organized a training session that broached this issue. Speaking of the importance of data literacy and effective communication, Thomas Roca, a Microsoft Data Strategist and guest speaker at the event, said:

“It is sound data infrastructure design and visualization that put insights at the fingertips of policymakers and ultimately makes a data revolution tangible.”

As scientists, we tend to be extremely technical – often unnecessarily over-complicating things. In contrast, the majority of policymakers and other influential industry members are usually more pragmatic in their approach.

Using modern, intuitive data visualization (Data-Viz) frameworks can help bridge this gap, enabling us to tap into the potential of research outputs from CGIAR.

Without falling into the pitfall of information overload, interactive and regular data updates can really help attract and retain the attention of your audience.

Modern Data-Viz frameworks also give the power of data exploration to the data user. From dedicated software, such as PowerBi or Carto, to JavaScript libraries like Highcharts and Leaflets, data enthusiasts from all backgrounds and capabilities can produce appealing data narratives.

We can only imagine the potential impact easy data exploration can have; if it is made available at the fingertips of the right industry influencers and policymakers.

For this to happen, we scientists need to upgrade our skills and learn how to communicate and be in sync with how data flows online – and we can now say, first hand, just how easy it can be.

During January of this year, the Data Intelligence Hub of CIAT-Asia teamed up with Thomas Roca from Microsoft, to hold a workshop to introduce these new Data Visualization concepts and frameworks to both scientific and non-scientific staff from Asia.

A quick survey was done before the course to assess the abilities of the participants. It showed that most of them hated coding, finding it too complicated.

After the workshop, attendees were encouraged to participate in a Datathon during which they were made to produce Data Products. And guess what happened?

With just those three days of training, the workshop participants were able to produce intuitive data visualization products from static data sources and APIs, tell new data stories in an engaging and accessible way.

Here is one example from one of the participants.

With the advent of cloud computing, more and more data tools are now accessible in cloud infrastructure platforms like Microsoft’s Azure.

It has never been so easy to develop and deploy data application, automating data collection and connecting, seamlessly, live data streams to interactive visualization.

Even more traditional platforms like Excel have now embedded some cloud capabilities, allowing many more aspiring data ninjas to sharpen their skills.

To answer the question as to how to get Open Data down from the cloud to make the most impact – we just need good vehicles in the form of great data products and good storytelling tools.

And what is more impacting than a great story?


Mar 3, 2018

This article was co-authored with contributions from Burra Dharani Dhar; Data Scientist, Decision and Policy Analysis group; Nguyen Tri Kien, Statistics and Data Management CIAT-Asia’s Data Intelligence Hub in Hanoi, Vietnam; and Thomas Roca, Microsoft Economist and Data Strategist, CELA, in Brussels, Belgium

Blog Competition Entry

This article is published as a part of our publicly open big data blog competition. If you have enjoyed this reading this entry, you can vote by liking, commenting or sharing.