AI Agents Robustness
Project Description
Large language model (LLM) based AI agents have demonstrated powerful capabilities in sequential decision-making, tool usage, and complex task decomposition. However, the widespread adoption of model distillation, cascaded fine-tuning, and recursive knowledge transfer creates critical yet under-explored robustness risks for agent systems. In particular, multi-round recursive distillation pipelines — where student models are iteratively distilled from previously distilled teacher models — lead to progressive skill forgetting, tail knowledge erosion, and systemic knowledge collapse.
This summer UROP project focuses on investigating the robustness degradation mechanism of AI agent pipelines under recursive distillation settings. Unlike conventional model distillation research that primarily focuses on mainstream task performance retention, this project emphasizes tail knowledge preservation and specialized agent skill stability across multiple distillation generations. Tail knowledge refers to low-frequency, task-specialized, and reasoning-intensive agent capabilities that are critical for long-tail real-world tasks, yet highly vulnerable to recursive distillation noise and gradient alignment homogenization.
The project will consist of three core research components:
1. Benchmark Construction: Build a recursive distillation evaluation pipeline for AI agents, covering tool-use reasoning, multi-step planning, and long-tail decision tasks, to quantify performance drift across distillation generations.
2. Empirical Analysis of Knowledge Collapse: Systematically characterize how recursive distillation induces skill homogenization, tail capability vanishing, and agent robustness degradation, including ablation studies on distillation temperature, data filtering thresholds, and iteration depth.
3. Robust Distillation Strategy Exploration: Explore preliminary mitigation approaches to preserve tail knowledge and stabilize agent functional robustness in multi-round recursive distillation scenarios, including adaptive temperature scheduling, tail sample reweighting, and knowledge residual preservation mechanisms.
Ultimately, this project aims to establish a measurable understanding of agent robustness decay in iterative knowledge distillation and provide actionable insights for building stable, deployable agent systems in cascaded model training pipelines.
This summer UROP project focuses on investigating the robustness degradation mechanism of AI agent pipelines under recursive distillation settings. Unlike conventional model distillation research that primarily focuses on mainstream task performance retention, this project emphasizes tail knowledge preservation and specialized agent skill stability across multiple distillation generations. Tail knowledge refers to low-frequency, task-specialized, and reasoning-intensive agent capabilities that are critical for long-tail real-world tasks, yet highly vulnerable to recursive distillation noise and gradient alignment homogenization.
The project will consist of three core research components:
1. Benchmark Construction: Build a recursive distillation evaluation pipeline for AI agents, covering tool-use reasoning, multi-step planning, and long-tail decision tasks, to quantify performance drift across distillation generations.
2. Empirical Analysis of Knowledge Collapse: Systematically characterize how recursive distillation induces skill homogenization, tail capability vanishing, and agent robustness degradation, including ablation studies on distillation temperature, data filtering thresholds, and iteration depth.
3. Robust Distillation Strategy Exploration: Explore preliminary mitigation approaches to preserve tail knowledge and stabilize agent functional robustness in multi-round recursive distillation scenarios, including adaptive temperature scheduling, tail sample reweighting, and knowledge residual preservation mechanisms.
Ultimately, this project aims to establish a measurable understanding of agent robustness decay in iterative knowledge distillation and provide actionable insights for building stable, deployable agent systems in cascaded model training pipelines.
Supervisor
FUNG, May
Quota
2
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
The UROP student will serve as the core research contributor responsible for end-to-end experimental implementation, data analysis, and preliminary method validation throughout the summer project. The student’s key responsibilities include:
1. Reviewing state-of-the-art literature on LLM agent distillation, knowledge collapse, and recursive model training.
2. Implementing and configuring the recursive multi-generation distillation pipeline for open-source agent models.
3. Conducting quantitative and qualitative evaluations on agent skill retention, tail task performance, and robustness variation across distillation iterations.
4. Performing ablation experiments to isolate key factors causing tail knowledge collapse and agent robustness degradation.
5. Summarizing experimental findings, organizing result visualization, and assisting in drafting project reports and potential research write-ups.
6. Collaborating with the supervisor to refine mitigation strategies for robust agent distillation and validate preliminary improvement effects.
The student will independently drive experimental progress while receiving regular supervisor guidance on research direction, experimental design, and technical optimization.
1. Reviewing state-of-the-art literature on LLM agent distillation, knowledge collapse, and recursive model training.
2. Implementing and configuring the recursive multi-generation distillation pipeline for open-source agent models.
3. Conducting quantitative and qualitative evaluations on agent skill retention, tail task performance, and robustness variation across distillation iterations.
4. Performing ablation experiments to isolate key factors causing tail knowledge collapse and agent robustness degradation.
5. Summarizing experimental findings, organizing result visualization, and assisting in drafting project reports and potential research write-ups.
6. Collaborating with the supervisor to refine mitigation strategies for robust agent distillation and validate preliminary improvement effects.
The student will independently drive experimental progress while receiving regular supervisor guidance on research direction, experimental design, and technical optimization.
Applicant's Learning Objectives
By completing this summer UROP project, the student aims to achieve the following academic and technical learning outcomes:
1. Theoretical Knowledge Mastery: Develop a solid systematic understanding of LLM agent mechanisms, model distillation theories, recursive knowledge transfer characteristics, and the underlying causes of model knowledge collapse and capability homogenization.
2. Technical Engineering Capability: Master end-to-end pipeline construction for large model recursive distillation, including model fine-tuning, distributed training configuration, agent task benchmark evaluation, and large-scale experimental ablation design.
3. Research Analysis Ability: Learn to quantify subtle model capability degradation, systematically analyze long-tail knowledge erosion phenomena, and summarize robust experimental conclusions from high-dimensional agent evaluation data.
4. Cutting-edge Research Vision: Gain in-depth exposure to frontier research challenges in trustworthy AI agents, including robustness, stability, and knowledge preservation in iterative model deployment pipelines.
5. Independent Research Competence: Cultivate the ability to independently design experiments, troubleshoot technical problems, refine research hypotheses, and standardize academic research documentation, laying a solid foundation for future AI agent and trustworthy machine learning research.
1. Theoretical Knowledge Mastery: Develop a solid systematic understanding of LLM agent mechanisms, model distillation theories, recursive knowledge transfer characteristics, and the underlying causes of model knowledge collapse and capability homogenization.
2. Technical Engineering Capability: Master end-to-end pipeline construction for large model recursive distillation, including model fine-tuning, distributed training configuration, agent task benchmark evaluation, and large-scale experimental ablation design.
3. Research Analysis Ability: Learn to quantify subtle model capability degradation, systematically analyze long-tail knowledge erosion phenomena, and summarize robust experimental conclusions from high-dimensional agent evaluation data.
4. Cutting-edge Research Vision: Gain in-depth exposure to frontier research challenges in trustworthy AI agents, including robustness, stability, and knowledge preservation in iterative model deployment pipelines.
5. Independent Research Competence: Cultivate the ability to independently design experiments, troubleshoot technical problems, refine research hypotheses, and standardize academic research documentation, laying a solid foundation for future AI agent and trustworthy machine learning research.
Complexity of the project
Challenging