POSIX and Object Distributed Storage Systems Performance Comparison Studies With Real-Life Scenarios in an Experimental Data Taking Context Leveraging OpenStack Swift & Ceph

Cloud archiving and data mining of High-Resolution Rapid Refresh forecast model output

“…Large network file systems or Redundant Array of Independent Disks (RAID) sets do not scale well as the number and size of drives increase, particularly since recovering and repairing after an error or disk corruption may require disks to be offline for many days. The CHPC selected Red Hat's Ceph object-based open source storage system (Maltzahn et al 2010) to address the shortcomings of both RAID and file systems based on published performance comparisons (e.g., Poat et al 2015) and testing over several years. Low-level operations, such as block or file level I/O, are managed by a software layer that manipulates objects for the user or administrator such that expensive RAID controllers are not necessary and archived objects can be replicated or made redundant according to configurable parameters.…”