{"id":37736,"date":"2024-10-21T06:00:00","date_gmt":"2024-10-21T00:30:00","guid":{"rendered":"https:\/\/debuggercafe.com\/?p=37736"},"modified":"2024-10-21T06:02:15","modified_gmt":"2024-10-21T00:32:15","slug":"serving-llms-using-litserve","status":"publish","type":"post","link":"https:\/\/debuggercafe.com\/serving-llms-using-litserve\/","title":{"rendered":"Serving LLMs using LitServe"},"content":{"rendered":"\n

At one point or the other, we all have run LLMs locally, maybe through Hugging Face Transformers, Ollama, or any of the online tools and software available. However, for production-ready environments, we will need to serve LLMs using an API URL. It can be a locally hosted LLM accessible to the organization only or an exposed API for the customers to use. The serving part can be tricky. But with the myriad of libraries coming out, the process is slowly becoming simpler, even for deep learning engineers. One such library is LitServe<\/strong>. In this article, we will explore the serving of LLMs using LitServe<\/strong>.<\/p>\n\n\n\n

\n
Jump to Download Code<\/strong><\/a><\/div>\n<\/div>\n\n\n\n

<\/p>\n\n\n

\n
\"LitServe<\/a>
Figure 1. LitServe Lightning Studio inference – serving LLMs using LitServe.<\/figcaption><\/figure>\n<\/div>\n\n\n

Primarily, our goal is to explore different techniques for serving LLMs using LitServe, both locally and through an exposed API. After going through the article, we should be well-equipped to take our LLM projects one step further to real-world deployment. <\/p>\n\n\n\n

What are we going to cover in this article?<\/em><\/h2>\n\n\n\n