Cancer3.AI › AI in Oncology › AI Models › Geneformer
Foundation model Open weights Not applicable genomics

Geneformer

Transformer pretrained on about 30 million single-cell transcriptomes (Genecorpus-30M) that learns gene-network context from ranked expression, enabling few-shot prediction of gene dosage effects, cell states and candidate therapeutic targets.

🤗 ctheodoris/Geneformer Paper DOI { } JSON
1 698 downloads / month ♥ 311 updated 2026-05-26

At a glance

DeveloperTheodoris Lab (Gladstone Institutes / UCSF)
Released2023-05-31
Licenceapache-2.0
AvailabilityOpen weights
KindFoundation model
Regulatory statusNot applicable
ParametersV1 ~10M; V2 checkpoints of 104M and 316M parameters

What it does

Each cell becomes a sequence of genes ranked by expression; the model is fine-tuned for cell-type classification, disease-state prediction and in-silico perturbation (which gene knock-out shifts a malignant state).

Tasks, data types and cancers

TaskSingle-cell analysisGene expressionFeature extraction (embeddings)Drug discovery
Data typeSingle-cellTranscriptomics (RNA)
Clinical settingBasic researchDrug discovery
CancerPan-cancer
Inputrank-encoded single-cell expression — 2,048 genes in V1, 4,096 genes in V2
Outputcell and gene embeddings

Architecture

FamilyBERT-style transformer over rank-value gene encodings
ParametersV1 ~10M; V2 checkpoints of 104M and 316M parameters
Pre-trainingmasked gene prediction on Genecorpus-30M (V1) or Genecorpus-104M (V2); a separate V2 variant is continually trained on about 14 million cancer transcriptomes

Training data

V1 (June 2021 corpus, 2023 release): Genecorpus-30M — about 29.9 million human single-cell transcriptomes from public datasets across tissues, not restricted to cancer. V2 (December 2024): Genecorpus-104M — about 104 million human single-cell transcriptomes, with 104M- and 316M-parameter checkpoints and an input window of 4,096 genes. A separate V2 variant (Geneformer-V2-104M_CLcancer) is continually trained on about 14 million cancer transcriptomes. Corpus composition per the developers' model card; the underlying datasets are not published as a single downloadable collection.

Training set size~30M cells (V1) / ~104M cells (V2) / ~14M cancer cells (continual-learning variant)

Linked datasets

Evaluation

Benchmark / datasetMetricValueExternal validationSource
Corpus scaling and model quantization (Nature Computational Science, 27.03.2026) — resource use during fine-tuning fine-tuning time and GPU memory relative to the full-precision model quantization preserved the contextual gene and cell embedding space while requiring 15% of the time and 34% of the memory of the full model; the pretraining corpus was expanded to more than 100 million human single-cell transcriptomes no Source

How to run

# generic transformers loader — see the model card for the task-specific head and preprocessing
from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained('ctheodoris/Geneformer')
processor = AutoProcessor.from_pretrained('ctheodoris/Geneformer')
Librarytransformers + geneformer package

Install from the Hugging Face repo; tokenize with the provided gene-median dictionary. Fine-tuning on a few hundred labelled cells is the intended workflow.

Regulatory status and intended use

Regulatory statusNot applicable
Intended useResearch tool.

Source →

Regulatory status is quoted from the source linked above and can change. Research-use-only models must not be used for clinical decisions.

Limitations and bias

  • Rank encoding discards absolute expression; batch effects still leak in.
  • Pretraining corpus is not cancer-specific; tumour microenvironment states may need extra fine-tuning.

Sources

  1. Theodoris CV et al. Transfer learning enables predictions in network biology. Nature 2023
  2. Chen H, Venkatesh MS et al. Scaling and quantization of large-scale foundation model enables resource-efficient predictions in network biology. Nature Computational Science, 27.03.2026 (PMID 41896605)
  3. Hugging Face — ctheodoris/Geneformer (karta modelu: warianty V1/V2, Genecorpus-104M, wariant nowotworowy CLcancer)

This page is educational — it is not medical advice and does not replace consultation with an oncologist. Diagnostic and treatment decisions are made solely by specialist physicians.