System S is a large-scale distributed streaming data analysis environment. Ultimately, we envision that there will be multiple sites running the System S software, each with their own administration and goals. However, cooperation between these sites can frequently be of mutual benefit. We are designing the framework to support numerous sites that can work both independently and in cooperative fashions, with a variety of interaction models such as peer-to-peer or federated. Depending on the degree of site autonomy and the relationships between any given pair of sites, the sites would be able to share data, perform processing on behalf of one another, or even take over tasks of a failed site. Interoperability is complicated by the degree of site autonomy as well as differences in execution environments and security policies. This paper surveys System S, describes its models for site interoperability, and discusses particular design issues such as site failover and heterogeneity.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.