Columnar objects: improving the performance of analytical applications

Mattis, Toni; Henning, Johannes; Rein, Patrick; Hirschfeld, Robert; Appeltauer, Malte

doi:10.1145/2814228.2814230

Cited by 15 publications

(18 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proxies are Python classes that yield data on demand, using Python's property, __getitem__, and __getattr__ to emulate static members by fetching data (from memory, disk, or network) as necessary. These proxies most clostly resemble the work of Mattis et al in PyPy [11], except that PyPy is JIT compiled on the fly, making the proxy and code transformation approaches equivalent. However, Numpy and CPython provide access to much-needed scientific libraries.…”

Section: Methodsmentioning

confidence: 99%

“…We should also note that the code transformation technique described here is similar to that of Mattis et al [11], though we statically transform and compile Python functions, whereas Mattis et al implemented object proxies in PyPy and let PyPy's tracing JIT compiler dynamically optimize them. This technique can be viewed as a general alternative to object deserialization: when faced with user code that expects objects but the data are in another form, one could either transform the data to fit the code's expectation (deserialization) or transform the code to fit the data.…”

Section: Create Type Particle Frommentioning

confidence: 99%

See 1 more Smart Citation

Fast access to columnar, hierarchically nested data via code transformation

Pivarski

Elmer

Bockelman

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Big Data query systems represent data in a columnar format for fast, selective access, and in some cases (e.g. Apache Drill), perform calculations directly on the columnar data without row materialization, avoiding runtime costs.However, many analysis procedures cannot be easily or efficiently expressed as SQL. In High Energy Physics, the majority of data processing requires nested loops with complex dependencies. When faced with tasks like these, the conventional approach is to convert the columnar data back into an object form, usually with a performance price.This paper describes a new technique to transform procedural code so that it operates on hierarchically nested, columnar data natively, without row materialization. It can be viewed as a compiler pass on the typed abstract syntax tree, rewriting references to objects as columnar array lookups.We will also present performance comparisons between transformed code and conventional object-oriented code in a High Energy Physics context.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Create Type Particle Frommentioning

confidence: 99%

Fast access to columnar, hierarchically nested data via code transformation

Pivarski

Elmer

Bockelman

et al. 2017

2017 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…Our running example includes requirements similar to adaptations discussed in literature [12], [20], [16], [21], [22]. Generally, unanticipated scenarios involve adaptations of low-level structures such as the object representation in memory as well as behavioral features, e.g., to profile execution.…”

Section: Direct and Indirect Adaptationsmentioning

confidence: 99%

“…In relational databases, the overhead in data intensive applications is avoided by organizing the data in columns instead of rows [25], [26]. A direct adaptation approach for the scenario requiring fast field access is to organize the fields (of the class denoting the relational table) in a columnar manner, so that they are stored in subsequent memory cells [21].…”

Section: We Provide Detailed Implementationsmentioning

confidence: 99%

See 1 more Smart Citation

Fully Reflective Execution Environments: Virtual Machines for More Flexible Software

Chari

Garbervetsky

Marr

et al. 2019

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

VMs are complex pieces of software that implement programming language semantics in an efficient, portable, and secure way. Unfortunately, mainstream VMs provide applications with few mechanisms to alter execution semantics or memory management at run time. We argue that this limits the evolvability and maintainability of running systems for both, the application domain, e.g., to support unforeseen requirements, and the VM domain, e.g., to modify the organization of objects in memory. This work explores the idea of incorporating reflective capabilities into the VM domain and analyzes its impact in the context of software adaptation tasks. We characterize the notion of a fully reflective VM, a kind of VM that provides means for its own observability and modifiability at run time. This enables programming languages to adapt the underlying VM to changing requirements. We propose a reference architecture for such VMs and present TruffleMATE as a prototype for this architecture. We evaluate the mechanisms TruffleMATE provides to deal with unanticipated dynamic adaptation scenarios for security, optimization, and profiling aspects. In contrast to existing alternatives, we observe that TruffleMATE is able to handle all scenarios, using less than 50 lines of code for each, and without interfering with the application's logic.

show abstract

Reshape your layouts, not your programs: A safe language extension for better cache locality

Tasos

Franco

Drossopoulou

et al. 2020

Science of Computer Programming

View full text Add to dashboard Cite

The vast divide between the speed of CPU and RAM means that effective use of CPU caches is often a prerequisite for high performance on modern architectures. Hence, developers need to consider how to place data in memory so as to exploit spatial locality and achieve high memory bandwidth. Such manual memory optimisations are common in unmanaged languages (e.g. C, C++), but they sacrifice readability, maintainability, memory safety, and object abstraction. In managed languages, such as Java and C#, where the runtime abstracts away the memory from the developer, such optimisations are almost impossible.We present a language extension called SHAPES, which aims to offer developers more fine-grained control over the placement of data, without sacrificing memory safety or object abstraction. In SHAPES, programmers group related objects into pools, and specify how objects are laid out in these pools. Classes and types are annotated by pool parameters, which allow placement aspects to

show abstract

Columnar objects: improving the performance of analytical applications

Cited by 15 publications

References 11 publications

Fast access to columnar, hierarchically nested data via code transformation

Fast access to columnar, hierarchically nested data via code transformation

Fully Reflective Execution Environments: Virtual Machines for More Flexible Software

Reshape your layouts, not your programs: A safe language extension for better cache locality

Contact Info

Product

Resources

About