Benchmarks

Explore what each benchmark measures, the data it uses, and how models compare.

Browse the dataset catalog for training datasets and repositories, access terms, licenses, and model usage. Keep benchmark test data separate from model training and tuning.

Downloading data

Public benchmark inputs and reference files are available from Figshare. The DataFiles registry downloads and caches individual files locally:

from matbench_discovery.enums import DataFiles

wbm_structures = DataFiles.wbm_initial_atoms.path
phonon_structures = DataFiles.phonondb_pbe_103_structures.path
md_trajectories = DataFiles.dynamat_v1_0_md_trajectories.path
diatomic_curves = DataFiles.diatomics_dft_reference.path

Each task’s test-set section links its public inputs and explains reference availability.

All public data files

1. all_mp_tasks (mp/2023-03-16-all-mp-tasks.json.gz)
   

   Complete copy of the Materials Project database on 2023-03-16 (14 GB), exported through now-retired direct database access and preserved on Figshare (release v2022.10.28).

2. mp_computed_structure_entries (mp/2023-02-07-mp-computed-structure-entries.json.gz)
   

   JSON-Serialized pymatgen ComputedStructureEntries containing all Materials Project DFT-relaxed structures and corresponding final energies as of 2023-02-07

3. mp_elemental_ref_entries (mp/2023-02-07-mp-elemental-reference-entries.json.gz)
   

   Minimum energy ComputedEntry for each element in MP

4. mp_energies (mp/2025-02-01-mp-energies.csv.gz)
   

   Materials Project formation energies and energies above convex hull in eV/atom as a fast-to-load CSV file.

5. mp_patched_phase_diagram (mp/2023-02-07-ppd-mp.pkl.gz)
   

   PatchedPhaseDiagram constructed from all MP pymatgen ComputedStructureEntries

6. mp_trj_original (mp/2023-11-22-mp-trj.extxyz.zip)
   

   1,580,395 pymatgen Structures along with 1,580,395 energies, 7,944,833 magnetic moments, 49,295,660 forces, and 14,223,555 stresses. Originally created to train CHGNet and later used by many other machine learning force fields.

7. mp_trj_extxyz (mp/2024-09-03-mp-trj.extxyz.zip)
   

   ~1.6 M Materials Project DFT relaxation frames (subsampled to reduce redundancy and faulty data) converted to ase-compatible extended XYZ format including energies (eV), forces (eV/Å) and stress (eV/ų).

8. wbm_computed_structure_entries (wbm/2022-10-19-wbm-computed-structure-entries.jsonl.gz)
   

   JSON-Serialized pymatgen ComputedStructureEntries containing all WBM DFT-relaxed structures and corresponding final energies

9. wbm_relaxed_atoms (wbm/2024-08-04-wbm-relaxed-atoms.extxyz.zip)
   

   WBM relaxed structures as ase Atoms in extended XYZ format

10. wbm_initial_structures (wbm/2022-10-19-wbm-init-structs.jsonl.gz)
    

    Unrelaxed WBM structures in pymatgen Structure format

11. wbm_initial_atoms (wbm/2024-08-04-wbm-initial-atoms.extxyz.zip)
    

    Unrelaxed WBM structures as ase Atoms in extended XYZ format

12. wbm_summary (wbm/2023-12-13-wbm-summary.csv.gz)
    

    Computed material properties only, no structures. Available properties are VASP energy, formation energy, energy above the convex hull, volume, band gap, number of sites per unit cell, and more. Exact regeneration of the MP2020-corrected formation energy and hull-distance columns depends on the pymatgen MaterialsProject2020Compatibility behavior used for anion-correction assignment; the released values are reproducible with v2023.5.10-era behavior (see https://github.com/janosh/matbench-discovery/issues/358).

13. wbm_dft_geo_opt_symprec_1e_2 (data/wbm/dft-geo-opt-symprec=1e-2-moyo=0.3.1.csv.gz)
    

    WBM DFT-relaxed crystal symmetries as determined by moyopy==0.3.1 with symprec=1e-2.

14. wbm_dft_geo_opt_symprec_1e_5 (data/wbm/dft-geo-opt-symprec=1e-5-moyo=0.3.1.csv.gz)
    

    WBM DFT-relaxed crystal symmetries as determined by moyopy==0.3.1 with symprec=1e-5.

15. phonondb_pbe_103_structures (phonons/2024-11-09-phononDB-PBE-103-structures.extxyz)
    

    103 phononDB structures run by Togo with PBE settings received in private communication. See https://github.com/atztogo/phonondb/blob/bba206/README.md#url-links-to-phono3py-finite-displacement-method-inputs-of-103-compounds-on-mdr-at-nims-pbe for details.

16. phonondb_pbe_103_kappa_no_nac (phonons/2024-11-09-kappas-phononDB-PBE-noNAC.json.gz)
    

    103 phononDB structures run by Togo with PBE settings received in private communication. See https://github.com/atztogo/phonondb/blob/bba206/README.md#url-links-to-phono3py-finite-displacement-method-inputs-of-103-compounds-on-mdr-at-nims-pbe for details.

17. dynamat_v1_0_md_trajectories (md/2026-06-29-dynamat-v1.0-reference-trajectories.h5)
    

    DynaMat v1.0, a 17-system ab-initio NVT molecular dynamics reference set courtesy of working groups 2+4 of the COST Action "Data-driven Applications towards the Engineering of Functional Materials". It spans pure metals, alloys, high-entropy alloys, transition-metal dichalcogenides, perovskites and molecular crystals at 293-1500 K, used as references for the MD benchmark task. Single HDF5 file with one group per system holding label-free trajectory arrays (positions, cells and for most systems stresses, but no energies or forces) plus dt_fs (saved-frame interval) and temperature_kelvin as group attributes.

18. diatomics_dft_reference (diatomics/diatomics-dft.json.gz)
    

    Homonuclear DFT diatomic reference curves for 92 elements (H–U) at PBE and r2SCAN, including distances, energies, forces, site-projected magnetic moments, and winning spin candidates. The diatomics benchmark uses the PBE curves for reference-relative metrics. Reference provenance.