HIDDRA: a highly independent data distribution and retrieval architecture for space observation missions

Abstract

Institutions such as NASA, ESA or JAXA find solutions to distribute data from their missions to the scientific community, and their long term archives. This is a complex problem, as it includes a vast amount of data, several geographically distributed archives, heterogeneous architectures with heterogeneous networks, and users spread around the world. We propose a novel architecture that solves this problem aiming to fulfill the requirements of the final user. Our architecture is a modular system that provides a highly efficient parallel multiprotocol download engine, using a publisher/subscriber policy which helps the final user to obtain data of interest transparently. We have evaluated a first prototype, in collaboration with the ESAC centre in Villafranca del Castillo (Spain) that shows a high scalability and performance, opening a wide spectrum of opportunities.

A Federated Content Distribution System to Build Health Data Synchronization Services

“…In previous works, we developed different solutions that solve different challenges, such as availability, storage usage, efficiency in download engine, and sharing information across different sites to mention some. These solutions were evaluated in use cases, such as satellite imagery and geospatial data management, and they are HIDDRA [2], BRS [3], SkyCDS [4], and FedIDS [5]. HIDDRA provides a highly efficient parallel multi-protocol download engine, using a pub/sub policy that helps the final user to obtain data of interest transparently.…”

“…HIDDRA provides a highly efficient parallel multi-protocol download engine, using a pub/sub policy that helps the final user to obtain data of interest transparently. This solves the problem to distribute data from different institutions, such as NASA, ESA, or JAXA, to the scientific community aiming to fulfill the requirements of the final user [2]. FedIDS emerged as an improved version of HIDDRA.…”

“…In the past, we developed different solutions that accomplish the non-functional requirements required in use cases such as satellite imagery and geospatial data management. These solutions are HIDDRA [2], the Branch replication scheme (BRS) [3], SkyCDS [4], and FedIDS [5]. However, these Fig.…”