MatGenBench: A Unified Benchmark for Generative Materials Discovery
Project Description
MatGenBench is proposed as a unified benchmarking framework for state-of-the-art
crystal structure generation models. It would provide a single, reproducible CLI
(`run.py`) to **train, sample, and evaluate** multiple families of generative
materials models under consistent conditions, while respecting each upstream
repository's native configuration via Hydra-style overrides.
Models to be supported (integrated non-invasively through `third_party/` adapters):
- **CDVAE** — diffusion VAE for periodic crystals
- **DiffCSP-PP** — joint equivariant diffusion for crystal structure prediction
- **MatterGen** — diffusion model for de novo and property-conditioned generation
- **FlowMM** — Riemannian flow matching for materials
The core scientific problem: generative materials models are published with
inconsistent datasets, metrics, and training recipes, making fair comparison
almost impossible. A single model may look "SOTA" on one benchmark but fail on
another. MatGenBench would close this gap by standardizing data splits
(`mp_20`, `unique_structs_all`, `unique_structs_all_500`), evaluation metrics
(uniqueness, novelty, validity, structure matching, and optional
energy/stability via MatterSim MLFF), and reporting.
crystal structure generation models. It would provide a single, reproducible CLI
(`run.py`) to **train, sample, and evaluate** multiple families of generative
materials models under consistent conditions, while respecting each upstream
repository's native configuration via Hydra-style overrides.
Models to be supported (integrated non-invasively through `third_party/` adapters):
- **CDVAE** — diffusion VAE for periodic crystals
- **DiffCSP-PP** — joint equivariant diffusion for crystal structure prediction
- **MatterGen** — diffusion model for de novo and property-conditioned generation
- **FlowMM** — Riemannian flow matching for materials
The core scientific problem: generative materials models are published with
inconsistent datasets, metrics, and training recipes, making fair comparison
almost impossible. A single model may look "SOTA" on one benchmark but fail on
another. MatGenBench would close this gap by standardizing data splits
(`mp_20`, `unique_structs_all`, `unique_structs_all_500`), evaluation metrics
(uniqueness, novelty, validity, structure matching, and optional
energy/stability via MatterSim MLFF), and reporting.
Supervisor
CHENG, Lixue
Quota
2
Course type
UROP1000
UROP1100
UROP2100
UROP3100
UROP3200
UROP4100
Applicant's Roles
1. **Onboard the codebase.** Study `run.py`, the `matgenbench/` package, and the
`third_party/` adapters; reproduce the four models' training/generation/eval
pipelines on at least one dataset; pass the test suite.
2. **Extend evaluation.** Implement stability/property-aware metrics (MLFF
relaxation, energy/stability proxies, property distributions) and wire them
into the unified evaluator; document the additions.
3. **Run the comparative study.** Execute a controlled cross-model benchmark at
scale, manage compute (GPU batching, mixed precision), and aggregate results
into a leaderboard with uncertainty estimates.
4. **Drive the inverse-design case.** Select a CBE-relevant target, run
property-conditioned generation, curate candidates, and validate a shortlist
(DFT or MLFF) with the supervisor.
5. **Disseminate.** Write the report and a methods/manuscript; package the
extended benchmark for open release and present findings at a group seminar
or workshop.
`third_party/` adapters; reproduce the four models' training/generation/eval
pipelines on at least one dataset; pass the test suite.
2. **Extend evaluation.** Implement stability/property-aware metrics (MLFF
relaxation, energy/stability proxies, property distributions) and wire them
into the unified evaluator; document the additions.
3. **Run the comparative study.** Execute a controlled cross-model benchmark at
scale, manage compute (GPU batching, mixed precision), and aggregate results
into a leaderboard with uncertainty estimates.
4. **Drive the inverse-design case.** Select a CBE-relevant target, run
property-conditioned generation, curate candidates, and validate a shortlist
(DFT or MLFF) with the supervisor.
5. **Disseminate.** Write the report and a methods/manuscript; package the
extended benchmark for open release and present findings at a group seminar
or workshop.
Applicant's Learning Objectives
1. **Generative ML for materials** — diffusion models, flow matching, and
equivariant networks for periodic structures.
2. **Rigorous benchmarking science** — reproducible experiment design, metric
design, statistical comparison, and leaderboard integrity.
3. **CBE-domain bridging** — connecting generated structures to real
chemical-engineering problems (catalysis, energy materials, separations) and
validating them with quantum/ML simulations.
4. **Research autonomy** — scoping a multi-phase project, managing compute
budgets, writing a report, and communicating results to both ML and
domain audiences.
equivariant networks for periodic structures.
2. **Rigorous benchmarking science** — reproducible experiment design, metric
design, statistical comparison, and leaderboard integrity.
3. **CBE-domain bridging** — connecting generated structures to real
chemical-engineering problems (catalysis, energy materials, separations) and
validating them with quantum/ML simulations.
4. **Research autonomy** — scoping a multi-phase project, managing compute
budgets, writing a report, and communicating results to both ML and
domain audiences.
Complexity of the project
Challenging