loading page

STARE for scalable unification of diverse data within Earth, Space, and Planetary Science
  • +6
  • Michael Rilee,
  • Kwo-Sen Kuo,
  • James Gallagher,
  • James Frew,
  • Niklas Griessbaum,
  • Kodi Neumiller,
  • Robert Wolfe,
  • Hongfeng Yu,
  • Pamela Clark
Michael Rilee
Rilee Systems Technologies LLC

Corresponding Author:mike@rilee.net

Author Profile
Kwo-Sen Kuo
Bayesics, LLC
Author Profile
James Gallagher
OPeNDAP, Inc.
Author Profile
James Frew
University of California, Santa Barbara
Author Profile
Niklas Griessbaum
University of California, Santa Barbara
Author Profile
Kodi Neumiller
OPeNDAP, Inc.
Author Profile
Robert Wolfe
National Aeronautics and Space Administration
Author Profile
Hongfeng Yu
University of Nebraska, Lincoln
Author Profile
Pamela Clark
Jet Propulsion Laboratory
Author Profile

Abstract

The variety of missions and observational campaigns in Earth and Space Science has led to a vast number of files containing low-level datasets with native, incompatible arrays. While higher-level datasets re-interpret this low-level data onto common, comparable arrays, this standardization moves scientists farther away from the observations, constraining the analysis and science that can be performed. SpatioTemporal Adaptive Resolution Encoding (STARE) provides a parallelizable, scalable index for co-aligning data with different native spatiotemporal formats efficiently across distributed computing resources. STARE is particularly useful for opening low-level datasets to intercomparison and integrative analysis by providing “array” indexes that carry spatiotemporal semantics, unifying datasets with previously incomparable native array indexing. We are developing STARE as a software library with both C++ and Python APIs and are integrating STARE indexing with existing data transfer tools (OPeNDAP). By organizing data in a hierarchical format and taking advantage of the Hierarchical Data Format’s (HDF) virtualization features, STARE may provide an end user with familiar HDF usability with STARE-enhanced performance and data unification on the back end. Furthermore, STARE’s spatial encoding can be used to index and integrate datasets associated with other planetary bodies, bringing scalability and unification of diverse data for planetary and space science as well.