Cancer3.AI › AI in Oncology › AI Models › ESM-2 / ESMFold
Foundation model Open weights Not applicable protein-structure

ESM-2 / ESMFold

Protein language models from 8 million to 15 billion parameters trained on UniRef sequences; embeddings power variant-effect and function prediction, and ESMFold predicts structure directly from a single sequence without MSAs.

🤗 facebook/esm2_t33_650M_UR50D GitHub Paper DOI { } JSON
1 303 383 downloads / month ♥ 89 updated 2023-03-21

At a glance

DeveloperMeta AI (FAIR)
Released2022-11-01
LicenceMIT
AvailabilityOpen weights
KindFoundation model
Regulatory statusNot applicable
Parameters650M (this checkpoint)

What it does

Masked-language modelling over amino acids; the 650M checkpoint is the practical default for embeddings, zero-shot variant scoring (log-likelihood ratios) and fine-tuning on oncology protein tasks.

Tasks, data types and cancers

TaskFeature extraction (embeddings)Variant effectProtein structure
Data typeProtein sequence
Clinical settingBasic researchDrug discovery
CancerPan-cancer
Inputamino-acid sequence (up to 1,024 tokens by default)
Outputper-residue embeddings (1,280-dim for 650M), masked-token logits

Architecture

FamilyTransformer encoder (BERT-style)
Backboneesm2_t33_650M (33 layers) — checkpoints from 8M to 15B
Parameters650M (this checkpoint)
Pre-trainingmasked language modelling on UniRef50/UniRef90

Training data

UniRef50 clusters (sampled from UniRef90 members), ~65 million unique sequences.

InstitutionsMeta AI; UniProt as data source

Evaluation

Benchmark / datasetMetricValueExternal validationSource
CAMEO / CASP14 (ESMFold) TM-score close to AlphaFold 2 on high-perplexity-free targets, faster by orders of magnitude (paper) yes Source

How to run

from transformers import AutoTokenizer, EsmModel
tok = AutoTokenizer.from_pretrained('facebook/esm2_t33_650M_UR50D')
model = EsmModel.from_pretrained('facebook/esm2_t33_650M_UR50D')
out = model(**tok('MKTAYIAKQRQISFVKSHFSRQ', return_tensors='pt'))
emb = out.last_hidden_state  # 1 x L x 1280
Librarytransformers / esm package

Regulatory status and intended use

Regulatory statusNot applicable
Intended useResearch tool.

Source →

Regulatory status is quoted from the source linked above and can change. Research-use-only models must not be used for clinical decisions.

Limitations and bias

  • Sequence-only: no ligands, no complexes.
  • Zero-shot variant scores are less accurate than supervised predictors for clinical classification.

Sources

  1. Lin Z et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 2023
  2. Hugging Face — facebook/esm2_t33_650M_UR50D

This page is educational — it is not medical advice and does not replace consultation with an oncologist. Diagnostic and treatment decisions are made solely by specialist physicians.