Physical AI for Smart Building Inspection: Robot Dogs, AI Agents and 3D Digital Twins
Project Description
Imagine asking a robot dog:
“Inspect this building, create a 3D map, identify anything suspicious, and decide where you should look next.”
This project explores Physical AI for next-generation building inspection, bringing together intelligent robots, multimodal sensing, 3D spatial mapping and modern AI foundation models.
Students will work with a real quadruped robotic platform equipped with cameras, LiDAR and other sensors to investigate how robots can autonomously explore buildings, understand their surroundings and identify potential defects or anomalies. Rather than simply collecting images for offline analysis, the project aims to develop an embodied AI system that can perceive, reason and act in the physical world.
The project will explore emerging technologies including vision-language models (VLMs), multimodal AI agents, open-vocabulary visual recognition, autonomous robot navigation, 3D mapping, and BIM/digital twins. For example, an AI agent may combine camera observations with 3D spatial information, reason about an uncertain defect, and decide whether the robot should move closer, change its viewpoint or collect additional evidence.
Depending on their interests and backgrounds, students may contribute to different parts of the system, including robot programming, autonomous navigation, computer vision, multimodal AI, 3D mapping, digital twins and human–robot interaction.
The long-term vision is to develop a human–AI–robot inspection team in which engineers contribute judgement and domain knowledge, AI provides perception and reasoning, and robots provide safe, repeatable and intelligent access to the physical environment.
“Inspect this building, create a 3D map, identify anything suspicious, and decide where you should look next.”
This project explores Physical AI for next-generation building inspection, bringing together intelligent robots, multimodal sensing, 3D spatial mapping and modern AI foundation models.
Students will work with a real quadruped robotic platform equipped with cameras, LiDAR and other sensors to investigate how robots can autonomously explore buildings, understand their surroundings and identify potential defects or anomalies. Rather than simply collecting images for offline analysis, the project aims to develop an embodied AI system that can perceive, reason and act in the physical world.
The project will explore emerging technologies including vision-language models (VLMs), multimodal AI agents, open-vocabulary visual recognition, autonomous robot navigation, 3D mapping, and BIM/digital twins. For example, an AI agent may combine camera observations with 3D spatial information, reason about an uncertain defect, and decide whether the robot should move closer, change its viewpoint or collect additional evidence.
Depending on their interests and backgrounds, students may contribute to different parts of the system, including robot programming, autonomous navigation, computer vision, multimodal AI, 3D mapping, digital twins and human–robot interaction.
The long-term vision is to develop a human–AI–robot inspection team in which engineers contribute judgement and domain knowledge, AI provides perception and reasoning, and robots provide safe, repeatable and intelligent access to the physical environment.
Supervisor
WANG, Yu-Hsing
Quota
5
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
Students will become part of a multidisciplinary Physical AI research team and take ownership of a focused research or development task. Depending on their interests and experience, they may work on one or more of the following areas:
1. Robot Intelligence
• Operate and program quadruped robots for building inspection.
• Work with cameras, LiDAR and other robotic sensors.
• Explore autonomous navigation, inspection-route planning and next-best-view strategies.
• Gain hands-on experience with robotic software such as ROS/ROS2.
2. Multimodal and Agentic AI
• Experiment with vision-language models and multimodal foundation models for building-scene understanding.
• Develop AI approaches for defect recognition, visual question answering and engineering reasoning.
• Explore AI agents that can combine observations, engineering knowledge and robotic tools to decide what information should be collected next.
3. 3D Spatial Intelligence and Digital Twins
• Process LiDAR and camera data to generate point clouds and 3D maps.
• Explore SLAM, spatial localization and robotic mapping.
• Investigate how robotic observations can be connected with BIM or 3D digital twins for building inspection.
4. Human–AI–Robot Collaboration
• Explore how engineers, AI systems and robots can work together during inspection.
• Investigate ways of incorporating human observations and engineering judgement into robotic inspection.
• Develop intuitive methods for people to interact with and guide intelligent robots.
5. Real-World Experimentation
• Design and conduct experiments in realistic building environments.
• Evaluate the performance, robustness and limitations of the developed system.
• Visualize and present research results.
• Contribute to prototype demonstrations and, where appropriate, research publications.
Students are not expected to have experience in all of these areas before joining. Each student can focus on a particular research track while learning from the broader team. Curiosity, creativity and willingness to learn are more important than prior expertise.
1. Robot Intelligence
• Operate and program quadruped robots for building inspection.
• Work with cameras, LiDAR and other robotic sensors.
• Explore autonomous navigation, inspection-route planning and next-best-view strategies.
• Gain hands-on experience with robotic software such as ROS/ROS2.
2. Multimodal and Agentic AI
• Experiment with vision-language models and multimodal foundation models for building-scene understanding.
• Develop AI approaches for defect recognition, visual question answering and engineering reasoning.
• Explore AI agents that can combine observations, engineering knowledge and robotic tools to decide what information should be collected next.
3. 3D Spatial Intelligence and Digital Twins
• Process LiDAR and camera data to generate point clouds and 3D maps.
• Explore SLAM, spatial localization and robotic mapping.
• Investigate how robotic observations can be connected with BIM or 3D digital twins for building inspection.
4. Human–AI–Robot Collaboration
• Explore how engineers, AI systems and robots can work together during inspection.
• Investigate ways of incorporating human observations and engineering judgement into robotic inspection.
• Develop intuitive methods for people to interact with and guide intelligent robots.
5. Real-World Experimentation
• Design and conduct experiments in realistic building environments.
• Evaluate the performance, robustness and limitations of the developed system.
• Visualize and present research results.
• Contribute to prototype demonstrations and, where appropriate, research publications.
Students are not expected to have experience in all of these areas before joining. Each student can focus on a particular research track while learning from the broader team. Curiosity, creativity and willingness to learn are more important than prior expertise.
Applicant's Learning Objectives
By participating in this project, students will gain hands-on experience at the intersection of AI, robotics and engineering. Students will learn to:
1. Understand Physical AI
• Understand how sensing, perception, reasoning and robotic actions are integrated into an intelligent physical system.
• Learn how modern AI moves beyond software-only applications to interact with the real world.
2. Work with Modern Robotic Systems
• Gain practical experience with quadruped robots, cameras, LiDAR and other sensors.
• Develop basic skills in robot programming, ROS/ROS2, localization, mapping and autonomous navigation.
3. Apply Modern Multimodal AI
• Explore vision-language and multimodal foundation models for visual understanding and engineering reasoning.
• Learn how AI agents can interact with robotic systems and make decisions based on multiple sources of information.
• Gain experience with open-vocabulary recognition and AI-assisted defect assessment.
4. Develop 3D Spatial Intelligence
• Learn the fundamentals of point clouds, SLAM and 3D reconstruction.
• Understand how robotic sensing can contribute to BIM and digital-twin applications.
5. Conduct Real-World AI and Robotics Research
• Design experiments and evaluate system accuracy, robustness and uncertainty.
• Learn that successful engineering AI involves more than simply running a model: it requires careful experimentation, validation and understanding of failure cases.
6. Build Interdisciplinary Research Skills
• Work collaboratively with researchers and students from engineering, AI and robotics.
• Develop problem-solving, communication and research skills.
• Potentially contribute to research demonstrations, student competitions, publications or follow-on research projects.
The project is particularly suitable for students interested in AI, robotics, computer vision, autonomous systems, smart buildings, digital twins, or the future of engineering.
1. Understand Physical AI
• Understand how sensing, perception, reasoning and robotic actions are integrated into an intelligent physical system.
• Learn how modern AI moves beyond software-only applications to interact with the real world.
2. Work with Modern Robotic Systems
• Gain practical experience with quadruped robots, cameras, LiDAR and other sensors.
• Develop basic skills in robot programming, ROS/ROS2, localization, mapping and autonomous navigation.
3. Apply Modern Multimodal AI
• Explore vision-language and multimodal foundation models for visual understanding and engineering reasoning.
• Learn how AI agents can interact with robotic systems and make decisions based on multiple sources of information.
• Gain experience with open-vocabulary recognition and AI-assisted defect assessment.
4. Develop 3D Spatial Intelligence
• Learn the fundamentals of point clouds, SLAM and 3D reconstruction.
• Understand how robotic sensing can contribute to BIM and digital-twin applications.
5. Conduct Real-World AI and Robotics Research
• Design experiments and evaluate system accuracy, robustness and uncertainty.
• Learn that successful engineering AI involves more than simply running a model: it requires careful experimentation, validation and understanding of failure cases.
6. Build Interdisciplinary Research Skills
• Work collaboratively with researchers and students from engineering, AI and robotics.
• Develop problem-solving, communication and research skills.
• Potentially contribute to research demonstrations, student competitions, publications or follow-on research projects.
The project is particularly suitable for students interested in AI, robotics, computer vision, autonomous systems, smart buildings, digital twins, or the future of engineering.
Complexity of the project
Challenging