Foundation model Open weights Research use only multimodal

BiomedCLIP

CLIP-style vision-language model pretrained on PMC-15M — 15 million figure–caption pairs from biomedical papers — with a PubMedBERT text tower and a ViT-B image tower; supports zero-shot classification and retrieval across pathology, radiology and more.

🤗 microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 GitHub Paper { } JSON

🤗 249 031 downloads / month ♥ 424 updated 2025-01-14

At a glance

DeveloperMicrosoft Research

Released2023-03-02

LicenceMIT

AvailabilityOpen weights

KindFoundation model

Regulatory statusResearch use only

Parameters~196M

What it does

Embeds biomedical images and text into one space; strong general baseline when a domain-specific foundation model (e.g. CONCH for pathology) is unavailable or too restrictive to license.

Tasks, data types and cancers

TaskImage–text retrieval Classification Feature extraction (embeddings)Question answering

Data typeMultimodal Histopathology (WSI)X-ray CT Biomedical literature

Clinical settingBasic research Education

CancerPan-cancer

Input224×224 image / text ≤ 256 tokens

Output512-dim joint embedding

Architecture

FamilyCLIP (contrastive image–text)

BackboneViT-B/16 (224) + PubMedBERT (256 tokens)

Parameters~196M

Pre-trainingcontrastive on PMC-15M

Training data

PMC-15M: 15 million figure–caption pairs from 4.4 million PubMed Central open-access articles, spanning 30+ image types.

Training set size15M image–caption pairs

Linked datasets

pretraining PubMed / PMC Open Access Subset Open download
evaluation PatchCamelyon (PCam) Open download
training PubMed / PMC Open Access Subset Open download

Evaluation

Benchmark / dataset	Metric	Value	External validation	Source
zero-shot classification on PCam, RSNA, and other public sets ↗	accuracy	above general CLIP and PubMedCLIP (paper)	yes	Source

How to run

import open_clip
model, preprocess = open_clip.create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
tokenizer = open_clip.get_tokenizer('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')
# image_features = model.encode_image(preprocess(img).unsqueeze(0))
# text_features = model.encode_text(tokenizer(['adenocarcinoma histopathology']))

Libraryopen_clip

Regulatory status and intended use

Regulatory statusResearch use only

Intended useResearch.

Source →

Regulatory status is quoted from the source linked above and can change. Research-use-only models must not be used for clinical decisions.

Limitations and bias

Trained on published figures, not clinical images: resolution, cropping and annotations differ from real slides and scans.
Captions carry publication bias.

Sources

At a glance What it does Tasks, data types and cancers Architecture Training data Evaluation How to run Regulatory status and intended use Limitations and bias Sources

Card provenance

Confidencehigh

Verified2026-09-05 · editorial

Updated2026-09-09

Hugging Facesnapshot from 2026-09-09

Facts on this card come only from the listed sources; the Hugging Face figures are a cached public snapshot.

Related models

This page is educational — it is not medical advice and does not replace consultation with an oncologist. Diagnostic and treatment decisions are made solely by specialist physicians.