A Foundation LAnguage Image model of the Retina (FLAIR):
Encoding expert knowledge in text supervision
📜 Published in Medical Image Analysis
Julio Silva-Rodríguez1,
Hadi Chakor2,
Riadh Kobbi2,
Jose Dolz1,
Ismail Ben Ayed1
1ÉTS Montréal, 2DIAGNOS Inc.
| Project | Journal | ArXiv | Code | Tutorials |
- Install in your enviroment a compatible torch version with your GPU. For example:
conda create -n flair_env python=3.11 -y
conda activate flair_env
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
- Install FLAIR library.
pip install git+https://github.com/jusiro/FLAIR.git
from PIL import Image
import numpy as np
# Import FLAIR
from flair import FLAIRModel
# Set model
model = FLAIRModel.from_pretrained("jusiro2/FLAIR")
# Load image and set target categories
# (if the repo is not cloned, download the image and change the path!)
image = np.array(Image.open("./documents/sample_macular_hole.png"))
text = ["normal", "healthy", "macular edema", "diabetic retinopathy", "glaucoma", "macular hole",
"lesion", "lesion in the macula"]
# Forward FLAIR model to compute similarities
probs, logits = model(image, text)
print("Image-Text similarities:")
print(logits.round(3)) # [[-0.186 -3.092 3.357 4.444 6.223 7.493 7.028 11.395]]
print("Probabilities:")
print(probs.round(3)) # [[0. 0. 0. 0.001 0.005 0.019 0.012 0.962]]
In the following, we present the scripts for model pre-training and transferability. To use them, we recommend cloning the whole repository.
git clone https://github.com/jusiro/FLAIR.git
cd FLAIR
pip install -r requirements.txt
-
Define the relative paths for datasets and dataframes in
./local_data/constants.py. -
Prepare the FUNDUS assembly dataset - check
./local_data/prepare_partitions.pyto prepare the dataframes.
- Vision-Language Pre-training.
python main_pretrain.py --augment_description True --balance True --epochs 15 --batch_size 128 --num_workers 6
- Define the relative paths for datasets and dataframes in
./local_data/constants.py. - Prepare the experiment setting for the target dataset - we used
./local_data/experiments.pyto store them.
if experiment == "02_MESSIDOR":
setting = {"dataframe": PATH_DATAFRAME_TRANSFERABILITY_CLASSIFICATION + "02_MESSIDOR.csv",
"task": "classification",
"targets": {"no diabetic retinopathy": 0,
"mild diabetic retinopathy": 1,
"moderate diabetic retinopathy": 2,
"severe diabetic retinopathy": 3,
"proliferative diabetic retinopathy": 4}}
- Zero-shot (no adaptation).
python main_transferability.py --experiment 02_MESSIDOR --method zero_shot --load_weights True --domain_knowledge True --shots_train 0% --shots_test 100% --project_features True --norm_features True --folds 1
- Linear Probing.
python main_transferability.py --experiment 02_MESSIDOR --method lp --load_weights True --shots_train 80% --shots_test 20% --project_features False --norm_features False --folds 5
If you find this repository useful, please consider citing this paper:
@article{FLAIR,
title = {A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision},
author = {Julio Silva-Rodríguez and Hadi Chakor and Riadh Kobbi and Jose Dolz and Ismail {Ben Ayed}},
journal = {Medical Image Analysis},
volume = {99},
pages = {103357},
year = {2025},
issn = {1361-8415},
}
- Code and Model Weights are released under Apache 2.0 license