{"architecture":{"backbone":"ViT-B/16 image encoder + transformer text encoder/decoder","family":"CoCa-style contrastive + captioning vision-language model","input":"448\u00d7448 tile (image tower) / free text up to the tokenizer limit","notes_en":"Pairs came from the PubMed Central open-access subset and in-house educational material; captions were cleaned and split into pathology-relevant segments before training.","notes_pl":"Pary pochodzi\u0142y z otwartego podzbioru PubMed Central i w\u0142asnych materia\u0142\u00f3w edukacyjnych; opisy oczyszczono i podzielono na segmenty istotne patomorfologicznie przed treningiem.","output":"512-dimensional joint embedding; text logits","params":"~395M total (image tower ~86M)","pretraining":"contrastive alignment + captioning on image\u2013caption pairs, after image-only pretraining"},"article":null,"cancer_slugs":["pan-cancer"],"category":"pathology","confidence":"high","datasets":[{"name":"PubMed / PMC Open Access Subset","note":"figure\u2013caption pairs from the PMC open-access subset","role":"pretraining","slug":"pubmed-pmc-oa"},{"name":"TCGA \u2014 The Cancer Genome Atlas (via NCI Genomic Data Commons)","note":"","role":"evaluation","slug":"tcga"},{"name":"PubMed / PMC Open Access Subset","note":"","role":"training","slug":"pubmed-pmc-oa"}],"developer":"Mahmood Lab, Brigham and Women's Hospital / Harvard Medical School","evaluation":[{"benchmark":"zero-shot subtyping (TCGA-derived tasks)","dataset_slug":"tcga","external":true,"metric":"balanced accuracy","source":"https://www.nature.com/articles/s41591-024-02856-4","value":"best among compared vision-language models"},{"benchmark":"image\u2013text retrieval (in-house and public sets)","external":true,"metric":"Recall@K","source":"https://www.nature.com/articles/s41591-024-02856-4","value":"reported in paper"}],"hf":{"downloads":58735,"fetched_at":"2026-09-09T21:33:09Z","gated":"auto","last_modified":"2024-05-05","library":"timm","license":"cc-by-nc-nd-4.0","likes":204,"pipeline_tag":"image-feature-extraction"},"kind":"foundation","license":"cc-by-nc-nd-4.0","limitations":"- Caption data is literature-biased: rare entities and non-English terminology are under-represented.\n- Zero-shot accuracy is prompt-sensitive; report the prompts you used.\n- Non-commercial licence.","links":{"demo":null,"docs":null,"doi":"10.1038/s41591-024-02856-4","github":"https://github.com/mahmoodlab/CONCH","huggingface":"https://huggingface.co/MahmoodLab/CONCH","paper":"https://www.nature.com/articles/s41591-024-02856-4","pmid":null},"modalities":["histopathology","clinical-text","multimodal"],"name":"CONCH","notable_uses":"Evaluated on 14 benchmarks spanning subtyping, grading, retrieval and segmentation prompts; the text side lets small labs prototype classifiers from prompts before any annotation.","openness":"gated-weights","regulatory":{"intended_use_en":"Research use; not a device.","intended_use_pl":"Do bada\u0144; nie jest wyrobem medycznym.","source_url":"https://huggingface.co/MahmoodLab/CONCH","status":"research-only"},"regulatory_status":"research-only","release_date":"2024-03-19","run_snippet":"# generic timm loader \u2014 check the model card for the exact init args and image normalisation\nimport timm, torch\nfrom huggingface_hub import login\nlogin()  # gated repos: accept the licence on huggingface.co first\nmodel = timm.create_model('hf-hub:MahmoodLab/CONCH', pretrained=True)\nmodel.eval()\ncfg = timm.data.resolve_data_config({}, model=model)\ntransform = timm.data.create_transform(**cfg)\n# emb = model(transform(tile).unsqueeze(0))  # 1 x D tile embedding\n","settings":["basic-research","diagnosis","education"],"slug":"conch","sources":[{"label":"Lu MY et al. A visual-language foundation model for computational pathology. Nat Med 2024","url":"https://www.nature.com/articles/s41591-024-02856-4"},{"label":"Hugging Face \u2014 MahmoodLab/CONCH","url":"https://huggingface.co/MahmoodLab/CONCH"}],"summary":"Vision-language foundation model for pathology: an image encoder and a text encoder trained together on 1.17 million histopathology image\u2013caption pairs, enabling zero-shot classification and image\u2013text retrieval without labelled slides.","tasks":["feature-extraction","classification","image-text-retrieval","segmentation"],"training":{"institutions":"BWH / MGH; PubMed Central OA figures","populations":"literature-derived \u2014 global but publication-biased","size":"1.17M image\u2013caption pairs","summary_en":"1.17 million image\u2013caption pairs after filtering (figures from open-access biomedical literature plus institutional educational content), preceded by image-only self-supervised pretraining on institutional slides.","summary_pl":"1,17 mln par obraz\u2013opis po filtrowaniu (ryciny z otwartej literatury biomedycznej plus instytucjonalne materia\u0142y edukacyjne), poprzedzone samonadzorowanym pretreningiem obrazowym na preparatach instytucjonalnych."},"updated_at":"2026-09-09T21:33:09.703721","url":"/ai-oncology/models/conch","usage":{"hardware":"single GPU for inference","library":"custom (conch package, PyTorch)","notes_en":"Install from the GitHub repo; weights are gated on Hugging Face. Zero-shot prompts should follow the phrasing templates in the repo \u2014 prompt wording moves accuracy by several points.","notes_pl":"Instalacja z repozytorium GitHub; wagi gated na Hugging Face. Podpowiedzi zero-shot warto budowa\u0107 wed\u0142ug szablon\u00f3w z repozytorium \u2014 sformu\u0142owanie zmienia dok\u0142adno\u015b\u0107 o kilka punkt\u00f3w."},"verified_at":"2026-09-05T22:26:01.107992","verified_by":"editorial","version":"CONCH (v1)","what_it_does":"CONCH maps tiles and free-text descriptions into one embedding space. You can classify a tile by comparing it with text prompts ('invasive ductal carcinoma' vs 'lobular carcinoma'), retrieve images for a sentence, or use the image tower as a strong feature extractor."}