Data Agents for Knowledge Self-Improvement
Project Description
Recent research on self-improving AI agents has largely focused on improving models, reasoning policies, prompts, tools, or other agent scaffolds through experience. However, the external knowledge on which agents rely is also a critical object of self-improvement: it can become incomplete, incorrect, redundant, or poorly linked over repeated use, thereby weakening the agents’ search and downstream answers. This project will investigate data agents for self-improvement: agents that use query and interaction histories to diagnose local knowledge defects, propose evidence-grounded edits, and incrementally improve a persistent knowledge base. Building on the query-conditioned refinement paradigm of DeepRefine, the project will develop a small-scale, auditable prototype with structured insert, delete, and replace operations, source verification, versioning, and rollback. The project will evaluate whether targeted knowledge refinement improves its effectiveness for agentic tasks and its quality more efficiently and safely than leaving the knowledge base unchanged or rebuilding it from scratch.
Supervisor
SONG Yangqiu
Quota
5
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
Working with the supervisor and PhD/MPhil student mentors, the applicant will conduct a focused literature review on self-improving agents, agent memory, and agentic knowledge refinement. The applicant will implement a query-driven data-agent pipeline, and design diagnostic signals for incompleteness, incorrectness, redundancy, and entity ambiguity. The applicant will implement structured knowledge-edit and an evidence-verification layer that accepts only supported, traceable edits while recording versions for rollback. The applicant will also establish baselines, run controlled experiments on simple and multi-hop QA tasks, analyze downstream quality, edit precision, cost, latency, and failure cases, and contribute to a reproducible open source project, and a research paper.
Applicant's Learning Objectives
1. Gain hands-on experience with retrieval-augmented generation, knowledge graphs, agent memory systems, vector retrieval, and Python-based agent pipelines.
2. Develop practical research in data quality management, including defect diagnosis, provenance tracking, structured knowledge editing, validation, version control, and rollback for persistent knowledge stores.
3. Learn to design rigorous evaluations for AI agents by comparing task performance, knowledge-edit precision, coverage, safety, latency, and computational cost.
Complexity of the project
Challenging