novel view generation from video inputs
Project Description
This project will develop an algorithm that takes a mobile phone video as input and outputs a neural representation that enables realtime rendering of novel viewpoints on a phone. The 3D reconstruction of input video has been largely solved. The focus of this project is on the derivation of the neural scene representation. We plan to explore different choices such as, multi-plane images, Gaussian Splatting, etc.
Supervisor
TAN, Ping
Quota
1
Course type
UROP1000
UROP1100
Applicant's Roles
The applicant is expected to train a neural network to generate the neural scene representation from some input images with known 3D camera poses. The applicant will work closely with an experienced senior PhD student on this task, which includes preparing training data, training the neural network, and testing on mobile devices.
Applicant's Learning Objectives
The applicant can learn the basics of 3D computer vision and computer graphics. In particular, this project will train a student with basic knowledge in camera matrices and 3D transformations, which are the foundation to understand 3D vision and robotics. It will also train a student with handson experiences in training neural networks.
Complexity of the project
Moderate