Big Data affects nearly all of us in NASA and it is exploding – the average annual growth rate is 60% and by end of 2012, the digital universe is estimated to be 2.7 zettabytes.  Goddard manages enormous volumes of data related to building and maintaining satellites, analytical simulations, and support functions.

So what is Big Data? In the IT industry, Big Data is defined by four V’s: volume, velocity, variety, and veracity. Volume is the sheer amount of data. Velocity is the speed with which new data is created and existing data modified. Variety is the management of various data formats and types. Veracity is a concern of many business leaders; it is estimated that one in three CEO’s don’t trust the information they use. Also, I’ll add a fifth V, for Value, or the considerable usefulness of Big Data. It allows us to see data patterns and anomalies and shifts our decision-making from being reactive to proactive. So how is NASA managing the many challenges of Big Data? The NASA Open Government Plan outlines many of our approaches such as: managing and processing; archiving and distribution; and sharing data.

On managing and processing, here’s an example. The Mission Data Processing and Control System (MPCS) was recently used by the Curiosity rover on Mars. MPCS interfaces with NASA’s deep-space network, and in turn the Mars Reconnaissance Orbiter, to relay data to and from Curiosity and process the raw data in real time, a process which previously took hours, if not days, to accomplish.

For archiving and distribution, consider the Atmospheric Science Data Center (ASDC) at Langley, which is processes, archives and distributes Earth science data, and the Planetary Data System (PDS), which contains considerable planetary science data. PDS offers access to over 100 TB of space images, telemetry, models, etc. associated with planetary missions from the past 30 years. 

NASA is a leader at sharing Big Data. The Earth Observing System Data and Information System (EOSDIS) manages and shares Earth science data from various sources – satellites, aircraft, field measurements, etc. The EOSDIS science operations are performed within 12 interconnected Distributed Active Archive Centers (DAACs), each with specific responsibilities for producing, archiving, and distributing Earth science data products.    

To enhance our ability to manage Big Data, I believe that the IT industry should adopt the Predictive Model Markup Language (PMML), an XML-based, vendor-agnostic markup language that provides an easier way to share predictive analytical model data. With PMML, proprietary issues and incompatibilities are no longer a barrier to the exchange of data and models between applications.  

One real world example of how NASA leverages its expertise in Big Data, and directly affects your life, is in the field of airline safety. NASA analyzes data from planes to study safety implications, which in turn helps to improve the maintenance procedures of commercial airlines and potentially prevent equipment failures. Using advanced algorithms, the agency helps sift through mountains of unstructured data to find key information that helps predict and prevent safety problems.