Bit-level binaries and generalized comprehensions in Erlang

Abstract

Binary (i.e., bit stream) data are omnipresent in computer and network applications but most functional programming languages currently do not provide sufficient support for them. Erlang is an exception since it does support direct manipulation of binary data, albeit currently restricted to byte streams, not bit streams. To ameliorate the situation, we extend Erlang's built-in binary datatype so that it becomes flexible enough to handle bit streams properly. To further simplify programming on bit streams we then show how binary comprehensions can be introduced in the language and how binary and list comprehensions can be extended to allow both binary and list generators.

Horn Binary Serialization Analysis

“…iii) The inference graph can be used to generate a parser for streams S( ). The axioms applied during the reasoning can be translated into actions, similarly to [18]: jumpRight o,s,i corresponds to buffering new data from the stream, and jumpLeft o,s,i or again jumpRight o,s,i to addressing within the buffer in case of already buffered data. join i , forward i and backward i correspond to consuming data and associating it to a field.…”

“…None of the following works uses explicitly, to my knowledge, any Horn clause representation of the parsing task. The ERLANG language [10] has a pattern-matching construct whose patterns can be binary comprehensions [18], similar to list comprehensions in functional programming languages. Given a set of bit patterns, the matcher is synthesized by constructing a labeled automaton and expressing the matching as a series of elementary actions: test the size of a field, read bits, test match.…”

Applications, Implementation and Performance Evaluation of Bit Stream Programming in Erlang

“…Another way to write code which iterates over a bit stream is to use a bit stream comprehension [6]. This is a construct analogous to a list comprehension [18], which in turn is an expression that is syntactic sugar for the combination of map, filter and concat on lists.…”

“…That is the pattern works as both a filter and a generator. To make this more explicit we can write a drop 0XX function which is equivalent with the previous one using an explicit filter in the following manner: 6 drop 0XX(BitStr) -> << X:3 || <<X:3>> <= BitStr, 2#100 =< X >>.…”

“…We have been heavily involved in this work and implemented a scheme for native code compilation of binaries and designed efficient algorithms for constructing deterministic pattern matching automata for byte-based binaries [7]. In last year's Erlang workshop we put forward a proposal [6] for lifting the restriction that Erlang binaries are sequences of bytes rather than bits and described the semantics of bit-level pattern matching on a bit-level binary (called bit stream). We have subsequently realized this proposal and describe its applications and implementation in this paper.…”