Embodied Robotic Arm Systems with AI-based Control Policies
Project Description
Building on current advances in robotics and machine learning, this UROP project aims to develop an embodied robotic arm system (with one or two arms) featuring AI-based control policies. These policies may involve diffusion-based neural network, transformers, Vision-Language-Action models, reinforcement learning, and imitation learning. The student will work closely with PG students to integrate, refine, and potentially enhance existing methods to achieve:
1. Efficiency: Faster inference, reduced computational cost, simpler hardware.
2. Performance: Higher success rates on diverse manipulation tasks.
3. Generalization: Adaptiveness to various tasks/environments with minimal adjustments.
4. Error Handling: Resilient detection, management, and recovery from unexpected events.
5. User Interaction: More intuitive interfaces and user-friendly control.
Reference Projects:
[RDT (Robotic Diffusion Transformers)]
[OpenVLA (Vision-Language-Action Models)]
[Mobile ALOHA (Bimanual Manipulation)]
These references highlight state-of-the-art approaches in diffusion transformer policies, real-world embodiment, and advanced control in robotic manipulation tasks. The applicant will have the opportunity to gain practical, hands-on experience in cutting-edge robotics research, bridging hardware and software development.
1. Efficiency: Faster inference, reduced computational cost, simpler hardware.
2. Performance: Higher success rates on diverse manipulation tasks.
3. Generalization: Adaptiveness to various tasks/environments with minimal adjustments.
4. Error Handling: Resilient detection, management, and recovery from unexpected events.
5. User Interaction: More intuitive interfaces and user-friendly control.
Reference Projects:
[RDT (Robotic Diffusion Transformers)]
[OpenVLA (Vision-Language-Action Models)]
[Mobile ALOHA (Bimanual Manipulation)]
These references highlight state-of-the-art approaches in diffusion transformer policies, real-world embodiment, and advanced control in robotic manipulation tasks. The applicant will have the opportunity to gain practical, hands-on experience in cutting-edge robotics research, bridging hardware and software development.
Supervisor
SHAO, Qiming
Quota
2
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
Applicants are expected to review the related literature and work with the supervisor and PG students.
The applicant may assist in the following tasks:
1. Data collection and model training for embodied robotic tasks.
2. Model structure design and refinement, integration with advanced AI methods.
3. Intelligent control system design and implementation (motion planning, obstacle avoidance and actuation).
4. Maintenance or modification of the mechanical structure of the robotic arms.
Preferred Background & Skills:
1. Experience with robotic manipulation/mechatronics (course projects, competitions, etc.)
2. Familiarity with ML fundamentals (PyTorch, TensorFlow, etc.)
3. The ability to control and debug real-world robot using ROS (Robot Operating System)
4. Programming in Python/C++
5. Participation in Robotic Competitions Teams (RoboMaster, RoboCon) or equivalent hands-on experience is a plus
The applicant may assist in the following tasks:
1. Data collection and model training for embodied robotic tasks.
2. Model structure design and refinement, integration with advanced AI methods.
3. Intelligent control system design and implementation (motion planning, obstacle avoidance and actuation).
4. Maintenance or modification of the mechanical structure of the robotic arms.
Preferred Background & Skills:
1. Experience with robotic manipulation/mechatronics (course projects, competitions, etc.)
2. Familiarity with ML fundamentals (PyTorch, TensorFlow, etc.)
3. The ability to control and debug real-world robot using ROS (Robot Operating System)
4. Programming in Python/C++
5. Participation in Robotic Competitions Teams (RoboMaster, RoboCon) or equivalent hands-on experience is a plus
Applicant's Learning Objectives
This is a multi-semester project. Students will tackle high-level tasks that integrate cutting-edge AI control policies with complex robotic systems, requiring resourceful problem-solving and collaboration. This project provides an excellent opportunity to push the boundaries of robotics and machine learning research through hands-on experimentation.
The student is expected to achieve the following objectives throughout the entire project:
1. Deepen knowledge of robotic systems and AI-controlled robotics
2. Practical skills to operate, maintain, and integrate robotic arms with sensors/electronics
3. Ability to implement advanced control policies and evaluate performance
4. Develop error handling and recovery strategies in robotic manipulation
5. Improve collaboration, documentation, and presentation skills
6. Publication of research findings (if applicable)
The student is expected to achieve the following objectives throughout the entire project:
1. Deepen knowledge of robotic systems and AI-controlled robotics
2. Practical skills to operate, maintain, and integrate robotic arms with sensors/electronics
3. Ability to implement advanced control policies and evaluate performance
4. Develop error handling and recovery strategies in robotic manipulation
5. Improve collaboration, documentation, and presentation skills
6. Publication of research findings (if applicable)
Complexity of the project
Challenging