Themes in information-rich functional programming for internet-scale data sources

Syme, Donald; Battocchi, Keith; Takeda, Kenji; Malayeri, Donna; Petříček, Tomáš

doi:10.1145/2429376.2429378

Cited by 22 publications

(14 citation statements)

References 9 publications

(6 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Type checking of small programs is typically fast enough that no caching is necessary. However, The Gamma supports type providers [9,54], which can generate types based on an external file or a REST service call, e.g. [47].…”

Section: Type Checkingmentioning

confidence: 99%

Foundations of a live data exploration environment

Petříček

2020

Programming

Self Cite

View full text Add to dashboard Cite

A growing amount of code is written to explore and analyze data, often by data analysts who do not have a traditional background in programming, for example by journalists. The way such data anlysts write code is different from the way software engineers do so. They use few abstractions, work interactively and rely heavily on external libraries. We aim to capture this way of working and build a programming environment that makes data exploration easier by providing instant live feedback.We combine theoretical and applied approach. We present the data exploration calculus, a formal language that captures the structure of code written by data analysts. We then implement a data exploration environment that evaluates code instantly during editing and shows previews of the results. We formally describe an algorithm for providing instant previews for the data exploration calculus that allows the user to modify code in an unrestricted way in a text editor. Supporting interactive editing is tricky as any edit can change the structure of code and fully recomputing the output would be too expensive. We prove that our algorithm is correct and that it reuses previous results when updating previews after a number of common code edit operations. We also illustrate the practicality of our approach with an empirical evaluation and a case study.As data analysis becomes an ever more important use of programming, research on programming languages and tools needs to consider new kinds of programming workflows appropriate for those domains and conceive new kinds of tools that can support them. The present paper is one step in this important direction. ACM CCS 2012Human-centered computing → Interactive systems and tools; Information systems → Data mining;Software and its engineering → Compilers;Foundations of a live data exploration environment development and execution. Data analysts write small snippets of code, run them to see results immediately and then revise them.Notebooks are used by users ranging from scientists who implement complex models of physical systems to journalists who perform simple data aggregations and create visualizations. Our focus is on the simplest use cases. Making programmatic data exploration more spreadsheet-like should encourage users to choose programming tools over spreadsheets, resulting in more reproducible and transparent data analyses.Consider the Financial Times analysis of plastic waste [7,25]. It joins datasets from Eurostat, UN Comtrade and more, aggregates the data and builds a visualization comparing waste flows in 2017 and 2018. Figure 1 shows an excerpt from one notebook of the data analysis. The code has a number of important properties:There is no abstraction. The analysis uses lambda functions as arguments to library calls, but it does not define any custom functions. Code is parameterized by having a global variable material set to "plastics" and keeping other possible values in a comment. This lets the analyst run and check results of intermediate steps.The code relies on external lib...

show abstract

Section: Type Checkingmentioning

confidence: 99%

Foundations of a live data exploration environment

Petříček

2020

Programming

Self Cite

View full text Add to dashboard Cite

show abstract

“…Type checking can be used thus to check the valid use of the derived types in programs. This approach has been successfully used for SQL [28], XML [37,25,3], and more generally [24,35]. Naturally, mappings have been studied in a semantic data context, too.…”

Section: Related Workmentioning

confidence: 99%

The Essence of Functional Programming on Semantic Data

Leinberger

Lämmel

Staab

2017

Programming Languages and Systems

View full text Add to dashboard Cite

Abstract. Semantic data fuels many different applications, but is still lacking proper integration into programming languages. Untyped access is error-prone. Mapping approaches cannot fully capture the conceptualization of semantic data. In this paper, we present λDL, a typed λ-calculus with constructs for operating on semantic data. This is achieved by the integration of description logics into the λ-calculus for both typing and data access or querying. The language is centered around several key design principles, in particular: (1) the usage of semantic conceptualizations as types, (2) subtype inference for these types, and (3) type-checked query access to the data by both ensuring the satisfiability of queries as well as typing query results precisely. The paper motivates the use of a designated type system for semantic data and it provides the theoretic foundation for the integration of description logics as well as the core formal definition of λDL including a proof of type safety.

show abstract

“…The type provider mechanism has been introduced in F# [23,24], added to Idris [3] and used in areas such as semantic web [18]. The F# Data library has been developed as part of the early F# type provider research, but previous publications focused on the general mechanisms.…”

Section: Related and Future Workmentioning

confidence: 99%

Types from data: making structured data first-class citizens in F#

2016

Self Cite

View full text Add to dashboard Cite

Most modern applications interact with external services and access data in structured formats such as XML, JSON and CSV. Static type systems do not understand such formats, often making data access more cumbersome. Should we give up and leave the messy world of external data to dynamic typing and runtime checks? Of course, not! We present F# Data, a library that integrates external structured data into F#. As most real-world data does not come with an explicit schema, we develop a shape inference algorithm that infers a shape from representative sample documents. We then integrate the inferred shape into the F# type system using type providers. We formalize the process and prove a relative type soundness theorem.Our library significantly reduces the amount of data access code and it provides additional safety guarantees when contrasted with the widely used weakly typed techniques.

show abstract

Themes in information-rich functional programming for internet-scale data sources

Cited by 22 publications

References 9 publications

Foundations of a live data exploration environment

Foundations of a live data exploration environment

The Essence of Functional Programming on Semantic Data

Types from data: making structured data first-class citizens in F#

Contact Info

Product

Resources

About