Code for LAGCN
Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition
Haojun Xu, Yan Gao, Zheng Hui, Jie Li, Xinbo Gao
- Upload NW-UCLA configs
- Add ensemble code
- Upload pretrained weights
- Add code of generating CPR graph (loss relevant)
- Add code of generating GPR graph (input data relevant)
- NTU RGB+D 60 Skeleton
- NTU RGB+D 120 Skeleton
- NW-UCLA
- Request dataset here: https://rose1.ntu.edu.sg/dataset/actionRecognition
- Download the skeleton-only datasets:
nturgbd_skeletons_s001_to_s017.zip(NTU RGB+D 60)nturgbd_skeletons_s018_to_s032.zip(NTU RGB+D 120)- Extract above files to
./data/nturgbd_raw
- Download dataset from here
- Move
all_sqeto./data/NW-UCLA
Put downloaded data into the following directory structure:
- data/
- NW-UCLA/
- all_sqe
... # raw data of NW-UCLA
- ntu/
- ntu120/
- nturgbd_raw/
- nturgb+d_skeletons/ # from `nturgbd_skeletons_s001_to_s017.zip`
...
- nturgb+d_skeletons120/ # from `nturgbd_skeletons_s018_to_s032.zip`
...
- Generate NTU RGB+D 60 or NTU RGB+D 120 dataset:
cd ./data/ntu # or cd ./data/ntu120
# Get skeleton of each performer
python get_raw_skes_data.py
# Remove the bad skeleton
python get_raw_denoised_data.py
# Transform the skeleton to the center of the first frame
python seq_transformation.py
Edit and run graph_gen/extract_cls_specific_embedding.py to obtain CPR & GPR graph.
- Change the config file depending on what you want.
# Example: training LAGCN on NTU RGB+D 120 cross subject with GPU 0
python main.py --config configs/ntu120-xsub/joint.yaml --work-dir work_dir/ntu120/csub/lagcn_joint --device 0
- To train your own model, put model file
your_model.pyunder./modeland run:
# Example: training your own model on NTU RGB+D 120 cross subject
python main.py --config config/ntu120-xsub/joint.yaml --model model.your_model.Model --work-dir work_dir/ntu120/xsub/your_model --device 0
- To test the trained models saved in <work_dir>, run the following command:
python main.py --config <work_dir>/config.yaml --work-dir <work_dir> --phase test --save-score True --weights <work_dir>/xxx.pt --device 0
- To ensemble the results of different modalities, run
# Example: ensemble four modalities of LAGCN on NTU RGB+D 120 cross subject
python ensemble.py --datasets ntu120/xsub --joint work_dir/ntu120/xsub/j.pkl --bone work_dir/ntu120/xsub/b.pkl --joint-motion work_dir/ntu120/xsub/jm.pkl --bone-motion work_dir/ntu120/xsub/bm.pkl
# Ensemble six modalities of LAGCN on NTU RGB+D 120 cross subject
python ensemble_6s.py --datasets ntu120/xsub --joint work_dir/ntu120/xsub/j.pkl --bone work_dir/ntu120/xsub/b.pkl --joint-motion work_dir/ntu120/xsub/jm.pkl --bone-motion work_dir/ntu120/xsub/bm.pkl --prompt work_dir/ntu120/xsub/p2.pkl --prompt2 work_dir/ntu120/xsub/p5.pkl
Pretrained weights and validation set inference results are provided in the link and link respectively.
The performance of NW-UCLA dataset is slightly different from the article table. The reason is that the MHA-GC used in NW-UCLA experiment is the numerical approximation version (Fig. 7b right). We will modify the relevant table of the article in the next version.
This repo is based on 2s-AGCN. The data processing is borrowed from SGN and HCN.
Thanks to the original authors for their work!
Please cite this work if you find it useful:
@article{xu2025language,
title={Language knowledge-assisted representation learning for skeleton-based action recognition},
author={Xu, Haojun and Gao, Yan and Hui, Zheng and Li, Jie and Gao, Xinbo},
journal={IEEE Transactions on Multimedia},
year={2025},
publisher={IEEE}
}
For any questions, feel free to contact: damnull@outlook.com