🎉 News: This survey has been ACCEPTED to the Lecture Style Tutorials Track of KDD 2025 as a HALF-DAY tutorial! 🎉

This is the official repository for "Multi-modal Time Series Analysis: A Tutorial and Survey". [Paper]

This repository is maintained by Yushan Jiang and Kanghui Ning from UConn DSIS.

Please consider citing our survey paper if you find it helpful :), and feel free to share this repository with others!

This survey aims to provide a unique and systematic perspective on effectively leveraging cross-modal interactions from relevant real-world contexts to advance multi-modal time series analysis, addressing both foundational principles and practical solutions. Our assessment is threefold:

• Reviewing multi-modal time series data
• Analyzing cross-modal interactions between time series and other modalities (Fusion, Alignment, Transference)
• Demonstrating revealing the impact of multi-modal time series analysis in applications across diverse domains.

Figure 1: The Framework of Our Survey	Figure 2: Categorization of cross-modal interaction methods and representative examples

We define three fundamental types of interactions between time series and other modalities, including Fusion, Alignment, and Transference, which occur at different stages within a framework --- Input, Intermediate (i.e., representations or intermediate outputs), and Output.

• Fusion refers to the process of integrating heterogeneous modalities in a way that captures complementary information across diverse sources to improve time series modeling.
• Alignment ensures that the relationships between different modalities are preserved and semantically coherent when integrated into a unified learning framework.
• Transference refers to the process of mapping between different modalities, which allows one modality to be inferred, translated, or synthesized from another.

Note:

• F: Fusion; A: Alignment; T: Transference

      title={Multi-modal Time Series Analysis: A Tutorial and Survey}, 
      author={Yushan Jiang and Kanghui Ning and Zijie Pan and Xuyang Shen and Jingchao Ni and Wenchao Yu and Anderson Schneider and Haifeng Chen and Yuriy Nevmyvaka and Dongjin Song},
      year={2025},
      eprint={2503.13709},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2503.13709}, 
}