Making Large Language Models (LLMs) Interact with Physical World
Project Description
The recent advancements in Large Language Models (LLMs) such as ChatGPT have paved the way for innovative applications that extend beyond conventional natural language processing tasks, such as robotic control. Our initiative introduces a novel concept termed "Penetrative AI," which aims at integrating LLMs with Internet of Things (IoT) technologies to interact with the physical world.
This project aims to leverage the principles of Penetrative AI to investigate and develop practical IoT-related applications using LLMs. These applications are designed to expand the functionalities of LLMs from purely digital interactions into tangible, physical-world applications.
For additional insights into Penetrative AI, please visit https://dapowan.github.io/wands_penetrative-ai/.
This project aims to leverage the principles of Penetrative AI to investigate and develop practical IoT-related applications using LLMs. These applications are designed to expand the functionalities of LLMs from purely digital interactions into tangible, physical-world applications.
For additional insights into Penetrative AI, please visit https://dapowan.github.io/wands_penetrative-ai/.
Supervisor
LI, Mo
Quota
3
Course type
UROP1100
UROP2100
Applicant's Roles
You are expected to work with researchers and/or senior PG students in learning "Penetrative AI" and its application, putting efforts in developing an AIoT system with LLM support. You may develop practical mobile applications on smartphones which may involve mobile clients across both Android and iOS platforms, coupled with a central server. These components collaboratively process sensor data harvested from smartphones to analyze and deduce user behavior using LLMs.
For students of strong research motivation, you may dive deeper into the research process and develop novel ideas of "Penetrative AI" and turning that into a technical report or research paper.
Requirements:
- Enthusiasm in research (exploring unknown or not yet in textbook studies).
- Comfortable with hands-on development of systems, confident with programming and experimentation skills.
- A basic understanding of LLMs.
- A solid understanding of deep learning and prior experience with deep neural network (DNN) projects, e.g., having completed relevant courses such as COMP 3211: Fundamentals of Artificial Intelligence, COMP 4211: Machine Learning, or COMP 4221: Introduction to Natural Language Processing would be a plus.
For students of strong research motivation, you may dive deeper into the research process and develop novel ideas of "Penetrative AI" and turning that into a technical report or research paper.
Requirements:
- Enthusiasm in research (exploring unknown or not yet in textbook studies).
- Comfortable with hands-on development of systems, confident with programming and experimentation skills.
- A basic understanding of LLMs.
- A solid understanding of deep learning and prior experience with deep neural network (DNN) projects, e.g., having completed relevant courses such as COMP 3211: Fundamentals of Artificial Intelligence, COMP 4211: Machine Learning, or COMP 4221: Introduction to Natural Language Processing would be a plus.
Applicant's Learning Objectives
Learning Outcomes:
- Develop fundamental research skills.
- Gain insights into the capabilities of LLMs and their integration into tangible, real-world applications.
- Develop fundamental research skills.
- Gain insights into the capabilities of LLMs and their integration into tangible, real-world applications.
Complexity of the project
Moderate