CONCH CONCH (v1)

At a glance

What it does

CONCH maps tiles and free-text descriptions into one embedding space. You can classify a tile by comparing it with text prompts ('invasive ductal carcinoma' vs 'lobular carcinoma'), retrieve images for a sentence, or use the image tower as a strong feature extractor.

Notable uses in oncology

Evaluated on 14 benchmarks spanning subtyping, grading, retrieval and segmentation prompts; the text side lets small labs prototype classifiers from prompts before any annotation.

Tasks, data types and cancers

Architecture

Pairs came from the PubMed Central open-access subset and in-house educational material; captions were cleaned and split into pathology-relevant segments before training.

Training data

1.17 million image–caption pairs after filtering (figures from open-access biomedical literature plus institutional educational content), preceded by image-only self-supervised pretraining on institutional slides.

Linked datasets

• pretraining PubMed / PMC Open Access Subset Open download — figure–caption pairs from the PMC open-access subset
• evaluation TCGA — The Cancer Genome Atlas (via NCI Genomic Data Commons) Free registration
• training PubMed / PMC Open Access Subset Open download

Evaluation

How to run

# generic timm loader — check the model card for the exact init args and image normalisation
import timm, torch
from huggingface_hub import login
login()  # gated repos: accept the licence on huggingface.co first
model = timm.create_model('hf-hub:MahmoodLab/CONCH', pretrained=True)
model.eval()
cfg = timm.data.resolve_data_config({}, model=model)
transform = timm.data.create_transform(**cfg)
# emb = model(transform(tile).unsqueeze(0))  # 1 x D tile embedding

Install from the GitHub repo; weights are gated on Hugging Face. Zero-shot prompts should follow the phrasing templates in the repo — prompt wording moves accuracy by several points.

Regulatory status and intended use

Source →

Regulatory status is quoted from the source linked above and can change. Research-use-only models must not be used for clinical decisions.

Limitations and bias

• Caption data is literature-biased: rare entities and non-English terminology are under-represented.
• Zero-shot accuracy is prompt-sensitive; report the prompts you used.
• Non-commercial licence.

Sources

1. Lu MY et al. A visual-language foundation model for computational pathology. Nat Med 2024
2. Hugging Face — MahmoodLab/CONCH