Earth Observation: Big Data

What is Big Data?

Big data is a term for data sets that are so large or complex that traditional data processing application softwares are inadequate to deal with them.

In an event this week in central London I heard big data described as “non-spreadsheet” data. I think Earth Observation (EO) data could fit into this definition of big data. What is nice about EO data is that is relatively homologous – a pixel has a DN number and you can do something with it (perhaps classify those pixels).


A version of the above diagram was shown. While much focus tends to be on the actions, the tricky part with the data is the journey from data to insight, from insight to action and from reaction back to data. An example was given about how supermarkets use machine learning to drive sales. When you shop online today, after you have completed your basket the supermarket will present you some options of items you may have missed. This data has been trained on previous shops and what was purchased the following day to drive the shopper to products he/she may still need.

Using the data (previous shopping habits and those habits the following few days) to drive insights (suggested list of products), to action (displayed to user at the end of a shop), then a record of the reaction by the consumer to purchase or not purchase, this all happens in the blink of an eye.

It was claimed that these additional insights drive a position reaction (ie added to basket) 25% of the time; that seems a pretty impressive reaction measure.

Can this workflow be applied to Earth Observation data?

Data – Insight – Action – Reaction.

If you are looking at building a terrain model in the Oilfield (for example), you can use EO data to drive insights into the surface. What kind of insight? Well perhaps the movement of vehicles, or planning a pipeline or building other oilfield infrastructure. Certainly EO derived models of the surface can provide insights.

There is a challenge to move from these insights to actions. Perhaps actions could include changing tyres, adjusting the cost routing for a pipeline or building on flatter more stable ground.

Reactions could be measured: what difference did these actions (if taken) actually achieve; did they save money, reduce risk or allow important construction deadlines to be met? It is these reactions that can then add another data layer to our existing EO data to again drive insights and actions.

EO data is just another source of data. The deluge of (very) high spatial resolution combined with daily coverage provide a reliable detailed measurement of the Earth’s surface. These are unquestionably beautiful images, but let’s remember it is data and data that can provide actionable insights into our world.