Advanced Confused Deputy Attack on the Context Compression Module of LLM Agent
Project Description
LLM agents have been widely used in our daily lives through Vibe Coding, Vibe Research, Deep Research, OpenClaw, and Hermes Agent. The complex agentic tasks bring motorically increasing LLM context (system prompt, user prompt, tool-use scheme, tool-use return) via an iterative reasoning loop. LLM context compression is, therefore, becoming an essential module in modern LLM agent frameworks. Our pioneering work (https://arxiv.org/pdf/2510.22963) firstly shows that the LLM context compression module can be tricked by an attacker into performing unintended yet malicious context overwriting, enabling attacks on real-world agents. This project dives into further investigation of an advanced confused duty attack on the real-world context compression module of LLM agents. The goal is to formalize attack theory and develop advanced attack algorithms on LLM context compression.
Supervisor
SHE, Dongdong
Quota
2
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
1. Conduct a comprehensive and in-depth survey on LLM agent, context compression.
2. Gain a deep theoretical and practical understanding of attack taxonomy.
3. Develop novel attack algorithms and implement a prototype.
4. Conduct an experiment, collect data, and analyze data.
5. Participate in top-tier academic conference paper writing.
Applicant's Learning Objectives
1. Deep understanding of ML security, LLM security. See my course COMP 4634, ML security, LLM security section for details.
2. Theoretical understanding of LLM compression and attack algorithm
3. Practical prototype/system development skill
4. Rigorous academic experimental setup, data collection, data analysis.
5. Professional academic paper writing skill
Complexity of the project
Moderate