Lack of I/O scalability is known to cause measurable slowdowns for large-scale scientific applications running on high end machines. This is prompting researchers to devise 'I/O staging' methods in which outputs are processed via online analysis and visualization methods to support desired science outcomes. Organized as online workflows and carried out in I/O pipelines, these analysis components run concurrently with science simulations, often using a smaller set of nodes on the high end machine termed 'staging areas'. This paper presents a new approach to dealing with several challenges arising for such online analytics, including: how to efficiently run multiple analytics components on staging area resources providing them with the levels of end-to-end performance they need and how to manage staging resources when analytics actions change due to user or data-dependent behavior. Our approach designs and implements middleware constructs that delineate and manage I/O pipeline resources called 'I/O Containers'. Experimental evaluations of containers with realistic scientific applications demonstrate the feasibility and utility of the approach.
Mean-variance relationship (MVR), nowadays agreed in power law form, is an important function.It is currently used by traffic matrix estimation as a basic statistical assumption. Because all the existing papers obtain MVR only through empirical ways, they cannot provide theoretical support to power law MVR or the definition of its power exponent. Furthermore, because of the lack of theoretical model, all traffic matrix estimation methods based on MVR have not been theoretically supported yet. By observing both our laboratory and campus network for more than one year, we find that such an empirical MVR is not sufficient to describe actual network traffic. In this paper, we derive a theoretical MVR from ON/OFF model. Then we prove that current empirical power law MVR is generally reasonable by the fact that it is an approximate form of theoretical MVR under specific precondition, which can theoretically support those traffic matrix estimation algorithms of using MVR. Through verifying our MVR by actual observation and public DECPKT traces, we verify that our theoretical MVR is valid and more capable of describing actual network traffic than power law MVR.traffic matrix, mean-variance relationship, self-similar
This paper uses computational I/O pipelines to integrate computations into the I/O path that perform data analytics on the data generated by scientific simulations. A novel attribute is the use of a pluggable execution environment in which analysis tools can be orchestrated into a multi-stage pipeline for processing simulation output data. Performance considerations are addressed through the use of a high performance data transport. The approach is evaluated with the end-to-end performance of a Magnetohydrodynamics application at large scale.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.