Ashish Patel 🇮🇳’s Post

𝗗𝗮𝘆-𝟯𝟬𝟱 𝗖𝗼𝗺𝗽𝘂𝘁𝗲𝗿 𝗩𝗶𝘀𝗶𝗼𝗻 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 Researchers of National University of Singapore has Introduced ‘𝗖𝗼𝗹𝗼𝘀𝘀𝗮𝗹-𝗔𝗜 ’: A PyTorch-Based Deep Learning System For Large-Scale Parallel Training Follow me for a similar post: 🇮🇳 Ashish Patel ------------------------------------------------------------------- 𝗜𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗶𝗻𝗴 𝗙𝗮𝗰𝘁𝘀 : 🔸This paper published in arxiv2021. 🔸 Colossal-AI allows users to set up combinations of data, pipeline, sequence, and multiple tensor parallelism. A user can use tensor parallelism to make a distributed model. 🔸This is just like how they make a single-GPU model. In this work, the researchers separated the model building from how it is distributed. They support many types of models, including 2D, 2.5D, and 3D tensor parallelism, sequence parallelism, and activation checkpointing. ------------------------------------------------------------------- 𝗜𝗠𝗣𝗢𝗥𝗧𝗔𝗡𝗖𝗘 🔸 The Transformer architecture has improved the performance of deep learning models in domains such as Computer Vision and Natural Language Processing. Together with better performance come larger model sizes. 🔸This imposes challenges to the memory wall of the current accelerator hardware such as GPU. It is never ideal to train large models such as Vision Transformer, BERT, and GPT on a single GPU or a single machine. 🔸There is an urgent demand to train models in a distributed environment. However, distributed training, especially model parallelism, often requires domain expertise in computer systems and architecture. 🔸It remains a challenge for AI researchers to implement complex distributed training solutions for their models. 🔸In this paper, Researchers introduce Colossal-AI, which is a unified parallel training system designed to seamlessly integrate different paradigms of parallelization techniques including data parallelism, pipeline parallelism, multiple tensor parallelism, and sequence parallelism. 🔸Colossal-AI aims to support the AI community to write distributed models in the same way as how they write models normally. This allows them to focus on developing the model architecture and separates the concerns of distributed training from the development process. ------------------------------------------------------------------- #computervision #artificialintelligence #deeplearning -------------------------------------------------------------------