Learning Compact Task-Conditioned Interfaces for Robot Manipulation
Project Description
Modern robot manipulation systems increasingly rely on multiple sources of information, including external cameras, wrist-mounted cameras, language instructions, and robot states. These sources provide complementary but often redundant information, while only a subset may be relevant to the robot's current task and manipulation stage. This project investigates how task-relevant information can be selectively distilled into a compact, fixed-size task-conditioned interface for downstream robot manipulation.
• Task-relevant information distillation: The project will investigate what global and contextual information is most useful for fine-grained manipulation and how such information can be effectively distilled into a compact task-conditioned interface with a fixed number of latent tokens. Candidate information includes target and goal semantics, spatial relationships, manipulation stages, interaction cues, and other action-relevant information, while possible approaches include feature-level distillation, semantic supervision, cross-view alignment, contrastive learning, and predictive objectives.
• Multi-view and multi-source information transfer: The project will study how complementary information from visual observations, language instructions, and robot states can be integrated into the compact interface. As a representative global-to-local setting, we will investigate how task information from main-view observations, such as target identity, goal location, spatial layout, and manipulation-stage cues, can support wrist-view perception and prediction.
• Downstream evaluation: The learned interface will be evaluated in both simulation and real-world settings on downstream tasks such as local visual grounding, future representation prediction, and action prediction.
• Task-relevant information distillation: The project will investigate what global and contextual information is most useful for fine-grained manipulation and how such information can be effectively distilled into a compact task-conditioned interface with a fixed number of latent tokens. Candidate information includes target and goal semantics, spatial relationships, manipulation stages, interaction cues, and other action-relevant information, while possible approaches include feature-level distillation, semantic supervision, cross-view alignment, contrastive learning, and predictive objectives.
• Multi-view and multi-source information transfer: The project will study how complementary information from visual observations, language instructions, and robot states can be integrated into the compact interface. As a representative global-to-local setting, we will investigate how task information from main-view observations, such as target identity, goal location, spatial layout, and manipulation-stage cues, can support wrist-view perception and prediction.
• Downstream evaluation: The learned interface will be evaluated in both simulation and real-world settings on downstream tasks such as local visual grounding, future representation prediction, and action prediction.
Supervisor
XU, Wenchao
Quota
1
Course type
UROP3100
UROP3200
UROP4100
Applicant's Roles
The applicant is expected to:
• Review relevant literature on robot manipulation, multi-view perception, vision-language models, representation learning, and knowledge distillation.
• Implement lightweight task-conditioned interfaces and information-distillation modules based on pretrained vision or vision-language models.
• Conduct controlled experiments comparing different information sources, distillation objectives, and interface sizes, and analyze their effects on downstream manipulation tasks.
• Evaluate and visualize the learned representations and summarize the findings in a final technical report or research presentation.
The project will involve both simulation and real-world robot experiments. Familiarity with Python and basic deep learning concepts would be helpful. Robot-operation safety training will be required for physical experiments.
• Review relevant literature on robot manipulation, multi-view perception, vision-language models, representation learning, and knowledge distillation.
• Implement lightweight task-conditioned interfaces and information-distillation modules based on pretrained vision or vision-language models.
• Conduct controlled experiments comparing different information sources, distillation objectives, and interface sizes, and analyze their effects on downstream manipulation tasks.
• Evaluate and visualize the learned representations and summarize the findings in a final technical report or research presentation.
The project will involve both simulation and real-world robot experiments. Familiarity with Python and basic deep learning concepts would be helpful. Robot-operation safety training will be required for physical experiments.
Applicant's Learning Objectives
Through this project, the applicant is expected to:
• Understand how different visual observations, language instructions, and robot states provide complementary information for robot manipulation.
• Learn the fundamentals of vision-language models, multi-view representation learning, knowledge distillation, and compact task-conditioned representations.
• Develop practical skills in PyTorch, pretrained models, robot-manipulation datasets, experimental design, ablation studies, and representation visualization.
• Gain experience in reading research papers, formulating research questions, implementing research ideas, analyzing experimental results, and communicating technical findings.
• Understand how different visual observations, language instructions, and robot states provide complementary information for robot manipulation.
• Learn the fundamentals of vision-language models, multi-view representation learning, knowledge distillation, and compact task-conditioned representations.
• Develop practical skills in PyTorch, pretrained models, robot-manipulation datasets, experimental design, ablation studies, and representation visualization.
• Gain experience in reading research papers, formulating research questions, implementing research ideas, analyzing experimental results, and communicating technical findings.
Complexity of the project
Moderate