A Temporal Attribution Explorer for Data Pipelines
Project Description
This project aims to design and implement a Temporal Attribution Explorer: a Python tool with an interactive dashboard that models a data processing pipeline as a network of operators exchanging quantities of data over time, and lets a user ask "how much of what came out of this pipeline at 2pm came from each source, and how has that changed since this morning?". It is ideal for UROP students who enjoy data analytics and visualisation and want practical exposure to data provenance and modern data pipelines. Everything is written in Python, at the level of a pipeline's logs rather than a database engine's internals, so the project has a gentle start and a clear path from a working first version to more ambitious extensions.
Background — What is Temporal Attribution?Data provenance (also called data lineage) is metadata that describes the origin and derivation of a piece of data: given an output, provenance tells you which inputs it came from and how it was produced. In databases, the classical flavours are where-provenance (which input value an output value literally comes from), why-provenance (which input rows justify an output row) and how-provenance (in what way those rows were combined). Provenance matters in practice for debugging ("why did this dashboard suddenly show a spike?"), auditing ("which source records influenced this report?"), data quality and reproducibility. These classical models, however, were designed for one-off queries over a static database. In a continuously running pipeline, such as an Apache Flink or Spark job, keeping track of every individual record's lineage becomes very expensive: the lineage graph keeps growing as data flows through the system, so it is impractical to keep it around long enough to study how a pipeline behaved over days or weeks.
Temporal attribution is a lighter-weight alternative. Instead of tracking individual records, it tracks how much each source contributed to an output, and how that contribution changes over time. The idea is to summarise the pipeline in fixed time intervals (say, one second at a time) and record, for each interval, simply how many records moved from one operator to the next. This turns the pipeline into a small graph of timestamped, quantified transfers, from which questions like "which sources fed this operator between 2pm and 3pm, and in what proportion?" can be answered cheaply, without storing anything per record. It is a natural fit whenever the individual records are not distinguishable anyway, for example once values have been summed inside a window, or when privacy rules prevent per-record tracking. What you lose is per-record detail; what you gain is the ability to monitor and compare a pipeline's behaviour over long periods. This project turns that idea into a working tool.
Background — What is Temporal Attribution?Data provenance (also called data lineage) is metadata that describes the origin and derivation of a piece of data: given an output, provenance tells you which inputs it came from and how it was produced. In databases, the classical flavours are where-provenance (which input value an output value literally comes from), why-provenance (which input rows justify an output row) and how-provenance (in what way those rows were combined). Provenance matters in practice for debugging ("why did this dashboard suddenly show a spike?"), auditing ("which source records influenced this report?"), data quality and reproducibility. These classical models, however, were designed for one-off queries over a static database. In a continuously running pipeline, such as an Apache Flink or Spark job, keeping track of every individual record's lineage becomes very expensive: the lineage graph keeps growing as data flows through the system, so it is impractical to keep it around long enough to study how a pipeline behaved over days or weeks.
Temporal attribution is a lighter-weight alternative. Instead of tracking individual records, it tracks how much each source contributed to an output, and how that contribution changes over time. The idea is to summarise the pipeline in fixed time intervals (say, one second at a time) and record, for each interval, simply how many records moved from one operator to the next. This turns the pipeline into a small graph of timestamped, quantified transfers, from which questions like "which sources fed this operator between 2pm and 3pm, and in what proportion?" can be answered cheaply, without storing anything per record. It is a natural fit whenever the individual records are not distinguishable anyway, for example once values have been summed inside a window, or when privacy rules prevent per-record tracking. What you lose is per-record detail; what you gain is the ability to monitor and compare a pipeline's behaviour over long periods. This project turns that idea into a working tool.
Supervisor
ZHOU, Xiaofang
Quota
2
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
You will build, entirely in Python, a tool that takes the execution log of a data pipeline and turns it into attribution information that can be queried and visualised. The work is deliberately staged so that there is a working system early on. You will begin by writing a small simulator: a script that plays out a simple pipeline with two or three sources, a couple of filtering and transformation steps, and one output, and writes a log of the form (from, to, time, number of records). Because the simulator generates the data, it can also record the true origin of every record, giving you ground truth to check your results against. Next, you will compute attribution over this log: for each operator and each time interval, work out what fraction of the records it holds came from each source, by pushing these proportions forward step by step through the pipeline. A filter that drops half the records passes on the same proportions; a step that splits records between two downstream operators divides its quantities between them. The result is a simple table of (operator, time interval, source, quantity) rows, which you can keep in pandas or DuckDB and look up by time. On top of this table you will implement the queries: the core three are backward attribution (which sources, and how much, are behind this operator's output at this time), forward attribution (where did the records this source produced end up), and temporal lineage (which sources contributed during a given time window). Two further query types, flow lineage (how much flowed from a source to the output through a particular intermediate operator) and versioning attribution (how did the attribution of an operator change between two times), are natural stretch goals once the first three work. You will then build the part that makes it useful: a dashboard, for example in Streamlit with Plotly, where the user picks an operator and a time and sees a diagram of the contributing sources, a chart of how each source's share evolves over the run, and a comparison between two points in time. Evaluation is mostly about correctness: check the attribution numbers against the ground truth from the simulator, and confirm that quantities add up along the pipeline. If time allows, there are several natural extensions: adding a windowed aggregation operator (which accumulates records and releases them when the window fires), measuring how much smaller the attribution table is than a per-record lineage log, or replacing the simulator with logs from a real Apache Flink job. The deliverables are: the Python tool and dashboard, the pipeline simulator, a test suite, and a written report describing the design, what the tool can and cannot answer, and possible extensions.
Applicant's Learning Objectives
Goals:
Review the basics of data provenance (where-, why- and how-provenance) and understand why per-record lineage becomes impractical in long-running data pipelines.
Write a small simulator that plays out a data pipeline and produces a log of timestamped, quantified transfers between operators, together with ground-truth record origins for checking your results.
Compute attribution by propagating source proportions forward through simple operators (filters, transformations, and steps that split records between several outputs).
Store the attribution as a table of (operator, time interval, source, quantity) rows in pandas or DuckDB, and make it possible to look up the state of any operator at any time.
Implement the three core query types: backward attribution, forward attribution, and temporal lineage over a time window.
Build an interactive dashboard that shows the contributing sources for a chosen operator and time, and how each source's share evolves over the run.
Validate the results against the ground truth produced by the simulator and write tests confirming that quantities are conserved along the pipeline.
Stretch goals: add flow lineage and versioning attribution queries, support a windowed aggregation operator, measure the storage saved compared with per-record lineage, or feed the tool with logs from a real Apache Flink job.
Document the design, the limitations of quantity-based attribution, and possible extensions in a final report.
Ideal Candidate:
This project is ideal for students who are curious about how data pipelines behave over time and who like turning an idea into a tool that someone can actually use. You should be comfortable writing Python and working with tables of data, enjoy building clear charts and interfaces, and be willing to check your own numbers carefully, since the attribution figures need to add up. Prior exposure to SQL is required; anything beyond that can be learnt along the way. No C++, no database engine internals, and no prior experience with stream processing systems are needed, as the tool works with pipeline logs rather than inside the engine. The project will be scoped to your background: the core version is a manageable, well-defined piece of work, and there are several clearly marked extensions for students who want to push further, either toward richer queries and indexing or toward the visualisation and interaction side.
Skills Required:
Programming: Python (pandas for the data handling, and a visualisation framework such as Streamlit or Plotly for the dashboard — both can be picked up during the project)
Basic understanding of SQL (SELECT, joins, GROUP BY, aggregates)
Comfort with tables of data and simple time-series plots
Comfort with a Unix shell and Git
Willingness to read a research paper and turn its main idea into working, tested code
Review the basics of data provenance (where-, why- and how-provenance) and understand why per-record lineage becomes impractical in long-running data pipelines.
Write a small simulator that plays out a data pipeline and produces a log of timestamped, quantified transfers between operators, together with ground-truth record origins for checking your results.
Compute attribution by propagating source proportions forward through simple operators (filters, transformations, and steps that split records between several outputs).
Store the attribution as a table of (operator, time interval, source, quantity) rows in pandas or DuckDB, and make it possible to look up the state of any operator at any time.
Implement the three core query types: backward attribution, forward attribution, and temporal lineage over a time window.
Build an interactive dashboard that shows the contributing sources for a chosen operator and time, and how each source's share evolves over the run.
Validate the results against the ground truth produced by the simulator and write tests confirming that quantities are conserved along the pipeline.
Stretch goals: add flow lineage and versioning attribution queries, support a windowed aggregation operator, measure the storage saved compared with per-record lineage, or feed the tool with logs from a real Apache Flink job.
Document the design, the limitations of quantity-based attribution, and possible extensions in a final report.
Ideal Candidate:
This project is ideal for students who are curious about how data pipelines behave over time and who like turning an idea into a tool that someone can actually use. You should be comfortable writing Python and working with tables of data, enjoy building clear charts and interfaces, and be willing to check your own numbers carefully, since the attribution figures need to add up. Prior exposure to SQL is required; anything beyond that can be learnt along the way. No C++, no database engine internals, and no prior experience with stream processing systems are needed, as the tool works with pipeline logs rather than inside the engine. The project will be scoped to your background: the core version is a manageable, well-defined piece of work, and there are several clearly marked extensions for students who want to push further, either toward richer queries and indexing or toward the visualisation and interaction side.
Skills Required:
Programming: Python (pandas for the data handling, and a visualisation framework such as Streamlit or Plotly for the dashboard — both can be picked up during the project)
Basic understanding of SQL (SELECT, joins, GROUP BY, aggregates)
Comfort with tables of data and simple time-series plots
Comfort with a Unix shell and Git
Willingness to read a research paper and turn its main idea into working, tested code
Complexity of the project
Moderate