Cancer3.AI › AI in Oncology › AI Models › BiomedBERT (PubMedBERT)
Foundation model Open weights Not applicable clinical-LLM

BiomedBERT (PubMedBERT)

BERT pretrained from scratch on PubMed abstracts and PMC full text with a biomedical vocabulary; the workhorse encoder for named-entity recognition, relation extraction and classification over oncology literature and reports.

🤗 microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext Paper DOI { } JSON
220 044 downloads / month ♥ 335 updated 2023-11-06

At a glance

DeveloperMicrosoft Research
Released2020-07-31
LicenceMIT
AvailabilityOpen weights
KindFoundation model
Regulatory statusNot applicable
Parameters~110M

What it does

Not yet documented on this card.

Tasks, data types and cancers

TaskInformation extractionClassificationFeature extraction (embeddings)
Data typeBiomedical literatureClinical text
Clinical settingBasic researchClinical trials
CancerPan-cancer
Inputtext ≤ 512 tokens
Outputtoken/sentence embeddings

Architecture

FamilyBERT-base
Parameters~110M
Pre-trainingmasked language modelling from scratch (domain vocabulary)

Training data

PubMed abstracts (14M) plus PMC full-text articles; domain-specific WordPiece vocabulary built from the corpus.

Linked datasets

Evaluation

Benchmark / datasetMetricValueExternal validationSource
BLURB (biomedical NLP benchmark) average score state of the art at publication yes Source

How to run

# generic transformers loader — see the model card for the task-specific head and preprocessing
from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained('microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext')
processor = AutoProcessor.from_pretrained('microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext')
Librarytransformers

Regulatory status and intended use

Regulatory statusNot applicable
Intended useResearch NLP encoder.

Source →

Regulatory status is quoted from the source linked above and can change. Research-use-only models must not be used for clinical decisions.

Limitations and bias

  • English literature only; clinical notes and Polish text need further adaptation.
  • 512-token window.

Sources

  1. Gu Y et al. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans Comput Healthc 2021
  2. Hugging Face — microsoft/BiomedNLP-BiomedBERT

This page is educational — it is not medical advice and does not replace consultation with an oncologist. Diagnostic and treatment decisions are made solely by specialist physicians.