Official implementation for the ICCV 2025 paper, Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator.

Sign language is a visual language that encompasses all linguistic features of natural languages and serves as the primary communication method for the deaf and hard-of-hearing communities. Although many studies have successfully adapted pretrained language models (LMs) for sign language translation (sign-to-text), the reverse task—sign language generation (text-to-sign)—remains largely unexplored. In this work, we introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs using a pretrained LM. To align sign language with the LM, we leverage a decoupled tokenizer that discretizes continuous signs into token sequences representing various body parts. During decoding, unlike existing approaches that flatten all part-wise tokens into a single sequence and predict one token at a time, we propose a multi-head decoding method capable of predicting multiple tokens simultaneously. This approach improves inference efficiency while maintaining effective information fusion across different body parts. To further ease the generation process, we propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs as auxiliary conditions, significantly improving the precision of generated signs.

Please run

conda create python=3.10 --name soke
conda activate soke
pip install -r requirements.txt

How2Sign: raw videos(Green Screen RGB clips (frontal view)) and split files.

CSL-Daily: raw videos and split files.

Phoenix-2014T: raw videos and split files.

SMPL-X Poses can be downloaded from the project homepage.

Please download human models (mano, smpl, smplh, and smplx) from here and unzip them into deps/smpl_models.

Download t2m evaluators via sh prepare/download_t2m_evaluators.sh.

Down t5 models via sh prepare/prepare_t5.sh. Note that this aims to avoid errors caused by the default config.

We use mBart-large-cc25, which can be downloaded here. Put the files into deps/mbart-h2s-csl-phoenix

python -m train --cfg configs/deto.yaml --nodebug

python -m test --cfg configs/deto.yaml --nodebug

We also provide the mean and the std of the SMPL-X poses. The checkpoint of the tokenizer is available here.

python -m get_motion_code --cfg configs/soke.yaml --nodebug
python -m train --cfg configs/soke.yaml --nodebug  #Note that please first update the path of the tokenizer's checkpoint.

python -m test --cfg configs/soke.yaml --task t2m  #you can set SAVE_PREDICTIONS in the config file to True if you want to save them.

Simple visualizations for meshes can be done by running

python -m vis_mesh --cfg=configs/soke.yaml --demo_dataset=csl

For colorful visualizations, please refer to the configurations of BlenderToolbox, and run

python vis_blender.py

We sincerely thank the open-sourced codes of these works where our code is based on: MotionGPT, ProgressiveTransformer, WiLoR, and OSX.

Please contact r.zuo@imperial.ac.uk for further questions.

@inproceedings{zuo2025soke,
    title={Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator},
    author={Zuo, Ronglai and Potamias, Rolandos Alexandros and Ververas, Evangelos and Deng, Jiankang and Zafeiriou, Stefanos},
    booktitle={ICCV},
    year={2025}
}