Underwater Scene Token Encoding for Low-Bandwidth 3D Reconstruction
Project Description
Underwater robots, with limited onboard computing power, must transmit large volumes of visual observations to a remote server for 3D scene reconstruction, analysis, and teleoperation. Sending the raw observations is infeasible over a typical acoustic link of only 64 bit/s.
This project asks three questions: How can large volumes of degraded underwater imagery be encoded into a compact representation that preserves 3D context? Can this representation be transmitted effectively over a 64 bit/s acoustic link? Can this representation be produced within the limited onboard computing capacity of an underwater robot?
These motivate three corresponding research directions:
1. Underwater Scene Encoding: Recent models [1,2] encode multi-view observations into 1,024 latent vectors (representation) for direct 3D recovery, but are trained primarily on in-air imagery. We will adapt them to underwater degradation by designs on data, adapters, and geometric supervision as we proposed in [3].
2. Discrete Scene Tokenization: Yet 1,024 768-dimensional FP16 vectors require 12.6 Mbit, requiring about 55 hours over a 64 bit/s link. We will jointly learn a 64-entry codebook [4], replacing each vector with a 6-bit index and reducing the representation to 6,144 bit, or 96 seconds for transmitting, while preserving 3D context.
3. Lightweight Scene Encoding: Existing scene encoders are too heavy for onboard deployment. We will distill the full encoder [5] into a compact model that produces the same scene representation within onboard compute and memory limits.
Related works include:
[1] Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale. https://arxiv.org/abs/2604.11331
[2] RayZer: A Self-supervised Large View Synthesis Model. https://arxiv.org/abs/2505.00702
[3] WAT3R: Feedforward Underwater 3D Reconstruction. https://arxiv.org/abs/2607.21023
[4] Neural Discrete Representation Learning. https://arxiv.org/abs/1711.00937
[5] Distilling the Knowledge in a Neural Network. https://arxiv.org/abs/1503.02531
This project asks three questions: How can large volumes of degraded underwater imagery be encoded into a compact representation that preserves 3D context? Can this representation be transmitted effectively over a 64 bit/s acoustic link? Can this representation be produced within the limited onboard computing capacity of an underwater robot?
These motivate three corresponding research directions:
1. Underwater Scene Encoding: Recent models [1,2] encode multi-view observations into 1,024 latent vectors (representation) for direct 3D recovery, but are trained primarily on in-air imagery. We will adapt them to underwater degradation by designs on data, adapters, and geometric supervision as we proposed in [3].
2. Discrete Scene Tokenization: Yet 1,024 768-dimensional FP16 vectors require 12.6 Mbit, requiring about 55 hours over a 64 bit/s link. We will jointly learn a 64-entry codebook [4], replacing each vector with a 6-bit index and reducing the representation to 6,144 bit, or 96 seconds for transmitting, while preserving 3D context.
3. Lightweight Scene Encoding: Existing scene encoders are too heavy for onboard deployment. We will distill the full encoder [5] into a compact model that produces the same scene representation within onboard compute and memory limits.
Related works include:
[1] Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale. https://arxiv.org/abs/2604.11331
[2] RayZer: A Self-supervised Large View Synthesis Model. https://arxiv.org/abs/2505.00702
[3] WAT3R: Feedforward Underwater 3D Reconstruction. https://arxiv.org/abs/2607.21023
[4] Neural Discrete Representation Learning. https://arxiv.org/abs/1711.00937
[5] Distilling the Knowledge in a Neural Network. https://arxiv.org/abs/1503.02531
Supervisor
LIU, Yuan
Co-Supervisor
YEUNG, Sai Kit
Quota
2
Course type
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
The project spans several tasks; each student will focus on a subset aligned with their background, interests, and the project's stage.
Responsibilities include:
1. Background and Baseline: Study the core methods, reproduce a pretrained scene encoding model, and record its reconstruction accuracy, representation size, and computational cost.
2. Underwater Scene Encoding: Prepare WAT3R data, integrate degradation-aware adaptation and geometric supervision to the above scene model, and evaluate underwater point map, depth, and pose accuracy.
3. Discrete Scene Tokenization: Add a vector-quantization bottleneck and jointly train the encoder, codebook, and decoder while tracking codebook use, quantization error, and 3D accuracy.
4. Lightweight Scene Encoding: Profile the full encoder, distill a compact student model, and evaluate its accuracy, latency, and memory on representative onboard hardware.
5. System Evaluation: Compare the baseline, underwater-adapted, quantized, and distilled models in reconstruction quality, bitrate, and transmission time.
6. Research Communication: Maintain reproducible code and records, analyze failures, and contribute to reports, demonstrations, and potential publications.
Responsibilities include:
1. Background and Baseline: Study the core methods, reproduce a pretrained scene encoding model, and record its reconstruction accuracy, representation size, and computational cost.
2. Underwater Scene Encoding: Prepare WAT3R data, integrate degradation-aware adaptation and geometric supervision to the above scene model, and evaluate underwater point map, depth, and pose accuracy.
3. Discrete Scene Tokenization: Add a vector-quantization bottleneck and jointly train the encoder, codebook, and decoder while tracking codebook use, quantization error, and 3D accuracy.
4. Lightweight Scene Encoding: Profile the full encoder, distill a compact student model, and evaluate its accuracy, latency, and memory on representative onboard hardware.
5. System Evaluation: Compare the baseline, underwater-adapted, quantized, and distilled models in reconstruction quality, bitrate, and transmission time.
6. Research Communication: Maintain reproducible code and records, analyze failures, and contribute to reports, demonstrations, and potential publications.
Applicant's Learning Objectives
By completing the project, students will:
1. Engage with Frontier Technologies: Gain hands-on experience in feedforward 3D reconstruction, quantized compression, and knowledge distillation, with broad applications across intelligent perception, robotics, and edge AI.
2. Build a Transferable Pipeline: Learn an encode-transmit-reconstruct pipeline that can extend beyond underwater robotics to mobile visual communication and other computation- or bandwidth-limited scenarios.
3. Develop Research Capability: Gain experience across the research lifecycle, from literature review and problem formulation to implementation, experimentation, critical analysis, and scientific communication.
1. Engage with Frontier Technologies: Gain hands-on experience in feedforward 3D reconstruction, quantized compression, and knowledge distillation, with broad applications across intelligent perception, robotics, and edge AI.
2. Build a Transferable Pipeline: Learn an encode-transmit-reconstruct pipeline that can extend beyond underwater robotics to mobile visual communication and other computation- or bandwidth-limited scenarios.
3. Develop Research Capability: Gain experience across the research lifecycle, from literature review and problem formulation to implementation, experimentation, critical analysis, and scientific communication.
Complexity of the project
Moderate