Efficient Optimization Methods for Large Language Model Training
Project Description
Training large language models (LLMs) requires substantial computational and memory resources, making optimization efficiency an important research problem in modern machine learning. This project will investigate efficient optimization methods for neural network and language model training, with a particular focus on the structure of gradient and parameter matrices and how such structures can be exploited to improve training efficiency.
The project will study and experimentally evaluate recent optimization approaches, including matrix-based optimization, gradient projection, and preconditioning techniques. Possible research directions include analyzing gradient and update structures during training, investigating low-rank or orthogonalized gradient updates, comparing different optimization algorithms in terms of convergence, memory consumption, and computational efficiency, and exploring modifications or combinations of existing methods.
Students will conduct experiments primarily on small- to medium-scale neural networks and language models, allowing the underlying optimization phenomena to be studied systematically under practical computational constraints. The exact research direction may be adjusted based on experimental findings and the student's progress.
No specific safety training is required for this computational research project.
The project will study and experimentally evaluate recent optimization approaches, including matrix-based optimization, gradient projection, and preconditioning techniques. Possible research directions include analyzing gradient and update structures during training, investigating low-rank or orthogonalized gradient updates, comparing different optimization algorithms in terms of convergence, memory consumption, and computational efficiency, and exploring modifications or combinations of existing methods.
Students will conduct experiments primarily on small- to medium-scale neural networks and language models, allowing the underlying optimization phenomena to be studied systematically under practical computational constraints. The exact research direction may be adjusted based on experimental findings and the student's progress.
No specific safety training is required for this computational research project.
Supervisor
ZHANG, Zihan
Quota
4
Course type
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
The student will participate in both the theoretical and experimental investigation of optimization methods for deep learning. Expected duties include:
1. Reviewing relevant literature on optimization algorithms for neural networks and large language models, including first-order, matrix-based, and memory-efficient optimization methods.
2. Implementing and reproducing selected optimization algorithms and experimental results using deep learning frameworks such as PyTorch.
3. Designing controlled experiments to compare optimizers in terms of training loss, convergence speed, memory usage, and computational efficiency.
4. Analyzing gradient, parameter, and optimizer-update statistics during training, including properties such as matrix norms, singular values, effective rank, and update magnitude.
5. Exploring potential modifications or extensions to existing optimization methods based on theoretical insights and experimental observations.
6. Documenting experimental results, producing plots and analyses, and discussing findings regularly with the supervisor.
1. Reviewing relevant literature on optimization algorithms for neural networks and large language models, including first-order, matrix-based, and memory-efficient optimization methods.
2. Implementing and reproducing selected optimization algorithms and experimental results using deep learning frameworks such as PyTorch.
3. Designing controlled experiments to compare optimizers in terms of training loss, convergence speed, memory usage, and computational efficiency.
4. Analyzing gradient, parameter, and optimizer-update statistics during training, including properties such as matrix norms, singular values, effective rank, and update magnitude.
5. Exploring potential modifications or extensions to existing optimization methods based on theoretical insights and experimental observations.
6. Documenting experimental results, producing plots and analyses, and discussing findings regularly with the supervisor.
Applicant's Learning Objectives
Upon completion of the project, the student is expected to:
1. Develop a solid understanding of modern optimization algorithms used in deep learning and large language model training.
2. Understand the mathematical principles behind gradient-based optimization, adaptive methods, matrix preconditioning, orthogonalization, and low-rank approximation.
3. Gain practical experience in implementing and evaluating optimization algorithms using modern deep learning frameworks.
4. Learn how to design controlled machine learning experiments and evaluate trade-offs among convergence, computational cost, and memory efficiency.
5. Develop skills in analyzing high-dimensional training dynamics through numerical and matrix-based diagnostics.
6. Gain experience in reading research papers critically, formulating research questions, interpreting experimental results, and communicating research findings.
1. Develop a solid understanding of modern optimization algorithms used in deep learning and large language model training.
2. Understand the mathematical principles behind gradient-based optimization, adaptive methods, matrix preconditioning, orthogonalization, and low-rank approximation.
3. Gain practical experience in implementing and evaluating optimization algorithms using modern deep learning frameworks.
4. Learn how to design controlled machine learning experiments and evaluate trade-offs among convergence, computational cost, and memory efficiency.
5. Develop skills in analyzing high-dimensional training dynamics through numerical and matrix-based diagnostics.
6. Gain experience in reading research papers critically, formulating research questions, interpreting experimental results, and communicating research findings.
Complexity of the project
Moderate