High-throughput ingest of data provenance records into Accumulo

Moyer, Thomas; Gadepally, Vijay

doi:10.1109/hpec.2016.7761589

Cited by 11 publications

(10 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…to identify the subset of images from a specific event or location. It is implemented using the D4M paradigm with the Accumulo NoSql backend for storage [13]- [15] with Accumulo considered the one of the highest performing databases and widely used for government applications [16]. This paradigm supports sparse storage, and easy integration with analysis tools in python and matlab/octave.…”

Section: E Data Indexing With Accumulomentioning

confidence: 99%

Large Scale Organization and Inference of an Imagery Dataset for Public Safety

Liu

Strohschein

Samsi

et al. 2019

2019 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

Video applications and analytics are routinely projected as a stressing and significant service of the Nationwide Public Safety Broadband Network. As part of a NIST PSCR funded effort, the New Jersey Office of Homeland Security and Preparedness and MIT Lincoln Laboratory have been developing a computer vision dataset of operational and representative public safety scenarios. The scale and scope of this dataset necessitates a hierarchical organization approach for efficient compute and storage. We overview architectural considerations using the Lincoln Laboratory Supercomputing Cluster as a test architecture. We then describe how we intelligently organized the dataset across LLSC and evaluated it with large scale imagery inference across terabytes of data.

show abstract

Section: E Data Indexing With Accumulomentioning

confidence: 99%

Large Scale Organization and Inference of an Imagery Dataset for Public Safety

Liu

Strohschein

Samsi

et al. 2019

2019 IEEE High Performance Extreme Computing Conference (HPEC)

View full text Add to dashboard Cite

show abstract

“…Further, Moyer et al evaluated the storage requirements of provenance when used for security purposes in relatively modest distributed systems [21]. In such a context, several thousands of graph elements can be generated per second and per machine, resulting in a graph containing billions of nodes to represent system execution over several months.…”

Section: Where Does the Audit Live?mentioning

confidence: 99%

Personal data and the internet of things

2019

View full text Add to dashboard Cite

The Internet of Things promises a connected environment reacting to and addressing our every need, but based on the assumption that all of our movements and words can be recorded and analysed to achieve this end. Ubiquitous surveillance is also a precondition for most dystopian societies, both real and fictional. How our personal data is processed and consumed in an ever more connected world must imperatively be made transparent, and more effective technical solutions than those currently on offer, to manage personal data must urgently be investigated.

show abstract

“…One of the main hurdles of system-level provenance capture is the sheer amount of data generated [13,64]. One approach to this issue is to improve provenance ingest to storage [64] and provenance query [90]. A second approach is to reduce the amount of provenance data captured.…”

Section: Related Workmentioning

confidence: 99%

Practical whole-system provenance capture

Pasquier

Han

Goldstein

et al. 2017

Proceedings of the 2017 Symposium on Cloud Computing

Self Cite

View full text Add to dashboard Cite

Data provenance describes how data came to be in its present form. It includes data sources and the transformations that have been applied to them. Data provenance has many uses, from forensics and security to aiding the reproducibility of scientific experiments. We present CamFlow, a whole-system provenance capture mechanism that integrates easily into a PaaS offering. While there have been several prior whole-system provenance systems that captured a comprehensive, systemic and ubiquitous record of a system's behavior, none have been widely adopted. They either A) impose too much overhead, B) are designed for long-outdated kernel releases and are hard to port to current systems, C) generate too much data, or D) are designed for a single system. CamFlow addresses these shortcoming by: 1) leveraging the latest kernel design advances to achieve efficiency; 2) using a self-contained, easily maintainable implementation relying on a Linux Security Module, NetFilter, and other existing kernel facilities; 3) providing a mechanism to tailor the captured provenance data to the needs of the application; and 4) making it easy to integrate provenance across distributed systems. The provenance we capture is streamed and consumed by tenant-built auditor applications. We illustrate the usability of our implementation by describing three such applications: demonstrating compliance with data regulations; performing fault/intrusion detection; and implementing data loss prevention. We also show how CamFlow can be leveraged to capture meaningful provenance without modifying existing applications.

show abstract

High-throughput ingest of data provenance records into Accumulo

Cited by 11 publications

References 14 publications

Large Scale Organization and Inference of an Imagery Dataset for Public Safety

Large Scale Organization and Inference of an Imagery Dataset for Public Safety

Personal data and the internet of things

Practical whole-system provenance capture

Contact Info

Product

Resources

About