Acquiring, processing, and analyzing open broadcast science data with
extremely low latency at scale with AWS Groundstation and cloud
processing.
Abstract
Many high value data products made available from agencies such as NASA,
NOAA, and USGS are broadcast in the clear directly from the satellite.
Traditional access patterns involve downlinking the data to an agency
ground segment, processing by a science team, and subsequent product
distribution through a program such as the NOAA Big Data Initiative or a
NASA Distributed Active Archive Center (DAAC). This talk demonstrates
the ability to acquire, process, and analyze open broadcast science data
entirely in the cloud. Leveraging the new AWS Elastic Ground Station
service, we demonstrate selecting and scheduling satellite acquisitions
for open science data then leveraging the scalable nature of the cloud
to execute a full processing stack. This talk demonstrates the ability
to acquire L0 data, leveraging open data processing algorithms from
agencies like NASA and NOAA, and a highly scalable, cost effective, and
secure processing architecture to produce near real-time data. By taking
advantage of direct satellite acquisition and cloud scale processing, we
demonstrate reduced latency from acquisition to data availability for
scenarios like disaster response, tipping and cueing, and location
monitoring. The deployed cloud stack leverages open source components
and demonstrates ingest and archival of L0 data, processing to L1
products and above, horizontally scaling data processing algorithms,
metadata generation, and search and discovery via APIs and a simple
serverless UI.