POPPER: A Dataflow System for In-Flight Error Handling in Machine Learning Workflows

Read Paper: here

Watch Demo: here

Abstract

We present POPPER, a dataflow system for building Machine Learning (ML) workflows. A novel aspect of POPPER is its built-in support for in-flight error handling, which is crucial in developing effective ML workflows. POPPER provides a convenient API that allows users to create and execute complex workflows comprising traditional data processing operations (such as map, filter, and join) and user-defined error handlers. The latter enables inflight detection and correction of errors introduced by ML models in the workflows. Inside POPPER, we model the workflow as a reactive dataflow, a directed cyclic graph, to achieve efficient execution through pipeline parallelization. We demonstrate the in-flight error-handling capabilities of POPPER, for which we have built a graphical interface, allowing users to specify workflows, visualize and interact with its reactive dataflow, and delve into the internals of POPPER.

Reviewer summary:

The article introduces POPPER, a dataflow system designed to address the challenges of building and executing Machine Learning (ML) workflows. The system’s key innovation lies in its in-flight error handling capabilities, which allow for real-time detection and correction of errors introduced by ML models during workflow execution. POPPER achieves this through a reactive dataflow model, represented as a directed cyclic graph, enabling efficient pipeline-parallel execution and backward error correction. The system also provides a Python-based API (PyScope) for workflow specification and a graphical interface for visualization and interaction with workflows.

Encouraging reviewer comments:

1. Innovative Error Handling: POPPER’s in-flight error handling using reactive dataflows addresses a critical challenge of silent ML model errors in pipelines.
2. Interactive Interface: The graphical interface makes workflows tangible and accessible, enhancing usability during demonstrations.
3. Python API: The Python-based API integrates well with tools data scientists commonly use, promoting adoption.
4. Execution Optimization: Auxiliary graphs and pipeline parallelism aim to optimize workflow execution.
5. Effective Demo: Clear use cases (e.g., identity cards, traffic data) and an intuitive demo video illustrate the system’s potential.