Integrated Data Analysis Pipelines for Large-Scale Data Management, HPC, and Machine Learning

The infrastructure for data management is growing fast. For accurate predictions, modern data-driven applications leverage large, heterogeneous data collections to uncover interesting patterns. They also build robust machine-learning models for accurate predictions. As a result, new systems have been developed with traditional, high-performance computing and the architecture of underlying hardware clusters. There is also a trend toward complex data analysis pipelines that combine different systems. The EU-funded DAPHNE project will define an open and extensible systems infrastructure for integrated data analysis pipelines. It will build a reference implementation of language abstractions (APIs and a domain-specific language) and an intermediate representation, as well as compilation and runtime techniques.