High-volume synchronized data collection

Abstract

Traditional methods of data collection typically rely on each instrument storing data locally during each data collect run with the files relayed to a central storage location at a later time. For moderate rate systems this is an acceptable paradigm. However, as ultra-high bandwidth instruments become available, this approach presents two significant limitations. First, the bandwidth required for the transfers can become unrealistic, and the transfer times are prohibitive. Second, the increasing complexity, speed, and breadth of instruments presents significant challenges in combining the data into a coherent data set for analysis. The Starfire Optical Range is in the process of implementing a centralized data storage system that provides multi-gigabyte per second transfer rates and allows each instrument to store directly to the primary data store. Additionally, the architecture provides for absolute synchronization of every data sample throughout all sensors. The result is a single data set with data from all instruments frame by frame synchronized.