While many teams may not want a more intelligent general-purpose model, they probably do want one that’s familiar with their product, their help desk ticket issues, their documentation, their internal labeling, and their particular industry-specific choices.<\/p>\n\n\n\n
Prompting assists with instruction. RAG assists with external information. Fine-tuning assists with something else: behavior. Patterns for classification, tone, decision-making boundaries, output formats for structured outputs, and repeating tasks. None of these are solved by writing a better prompt or attaching a vector database. They’re solved by teaching the model how your specific industry operates.<\/p>\n\n\n\n
Until recently, that type of instruction required a research lab. That is no longer true.<\/p>\n\n\n\n
There were several obstacles. Costly graphics processing units. Difficult to manage training scripts. Fragile datasets. Unclear evaluation metrics. Difficult-to-understand model formats. A deployment method that seemed to be another development project entirely. Underneath all of the above was the worry of “destroying” an existing base model by training it incorrectly.<\/p>\n\n\n\n
This worry was valid. Fine-tuning does require extensive machine learning expertise, expensive compute resources, and a willingness to have the model perform in unpredictable manners which are difficult to debug.<\/p>\n\n\n\n
However, what changed is that the open source community began to encapsulate that complexity into simple, developer-friendly workflows. Unsloth<\/a> conducts fine-tuning on a completely free Google Colab T4 GPU at two times faster speeds than other methods and uses 70-80% less memory. Axolotl<\/a> manages the entire fine-tuning process via a YAML file. The barrier dropped much sooner than most developers realize.<\/p>\n\n\n\n The best way to perceive the current collection of tools is not as individual products but rather as layers in a process. Each layer addresses a distinct challenge.<\/p>\n\n\n\n Preparing your dataset<\/strong> is where most fine-tuning processes begin or end. Hugging Face Datasets<\/a> supplies the tools needed for extracting, filtering, and preparing your dataset for use during fine-tuning. <\/p>\n\n\n\n However, the majority of the work occurs at a domain level: cleaning your labeled examples, categorizing your labels in such a manner that makes sense, and dividing your labeled examples into training and testing sets based upon how you actually plan to use each set.<\/p>\n\n\n\n Efficient fine-tuning<\/strong> represents where the financial aspect became dramatically cheaper. LoRA<\/a> (Low-Rank Adaptation) freezes the base model weight values and trains adapter matrices to fit the new weight values instead, reducing trainable parameters by roughly 90% while maintaining strong performance.<\/p>\n\n\n\n QLoRA<\/a> expands upon LoRA and incorporates 4-bit quantization to enable training on a consumer-grade GPU that would normally require enterprise-class hardware. Hugging Face PEFT<\/a> is the canonical library for both, describing LoRA as a method that “drastically reduces the number of parameters that need to be fine-tuned.”<\/p>\n\n\n\n Training orchestration<\/strong> is where the developer experience has improved most. Axolotl<\/a> is a configurable tool available under the Apache 2.0 license, and currently includes LoRA, QLoRA, full fine-tuning, reward modeling, and as of March 2025, multimodal training involving both visual and auditory models.<\/p>\n\n\n\n Unsloth<\/a>\u00a0achieves faster results through custom Triton kernel code and introduced Unsloth Studio<\/a>\u00a0in March 2026 which included a no-code web-based user interface; Unsloth Studio generated enough buzz among developers that it appeared on the front page of Hacker News within one day of release.<\/p>\n\n\n\n LLaMA-Factory<\/a> contains over 68,000 GitHub stars and offers compatibility with the largest variety of models using its own web interface. TRL<\/a> is Hugging Face’s official library for conducting supervised fine-tuning, DPO, and RLHF. torchtune<\/a> is a native library to PyTorch which uses minimal abstractions in order to accommodate consumer-grade GPUs containing 24 GB of VRAM.<\/p>\n\n\n\n Read more: How to make your first meaningful open source contribution<\/a><\/strong><\/p>\n\n\n\n Scaling<\/strong> becomes a factor when your team outgrows a single GPU. DeepSpeed<\/a> and PyTorch’s FSDP both support distributed training across multiple GPUs; Axolotl enables seamless integration with both via its configuration layer.<\/p>\n\n\n\n Evaluation<\/strong> is possibly the least developed component of this stack. While lm-eval-harness<\/a> provides standard benchmarks, the primary evaluation methodology used by most teams will be a custom test set drawn from production-quality examples of usage. If you do not create an evaluation set before fine-tuning, you’ll have no way to measure whether it worked.<\/p>\n\n\n\n Deployment<\/strong> completes the loop. vLLM<\/a>, llama.cpp<\/a>, and Ollama<\/a> all provide functionality for deploying a base model with LoRA adapters, therefore allowing teams to deploy without merging weights or maintaining a duplicate of the base model.<\/p>\n\n\n\n Fine-tuning works well when you need your model to repeat a particular task over and over again. Examples include: categorizing customer service requests, writing corporate messages, finding structured fields in unstructured content, following a very narrow process in an industry, generating code for projects with different styles, or producing exactly the same results with minimal reliance on long, fragile prompts.<\/p>\n\n\n\n Use fine-tuning if the above describes your situation. Otherwise, there may be other methods available: use RAG for constantly changing information, better prompts for simpler tasks, or reconsider if you have low-quality datasets or limited resources to create a validation set. As Nova AI Ops<\/a> puts it, prompting, RAG, and fine-tuning “get more powerful and more expensive in that order. Most teams should try them in that order too.”<\/p>\n\n\n\n The emerging consensus in 2026<\/a> is that the best production systems combine these approaches: put volatile knowledge in retrieval, put stable behavior in fine-tuning, and stop trying to force one tool to do both jobs.<\/p>\n\n\n\n It really does matter which method you choose. All AI problems are not fine-tuning problems and therefore, teams will have to determine whether or not fine-tuning is the best option.<\/p>\n\n\n\n This is not a hypothetical scenario. Real-world case studies<\/a> show that fine-tuned small models can reach higher accuracy than generic frontier APIs on internal classification tasks while being roughly 50x cheaper to run in production. One team trained on 3,500 clean examples and classified 70,000 tickets<\/a> for a total cost of $500. Another used QLoRA on a 7B model with 5,000 hand-labeled examples<\/a> and achieved 98% accuracy at under $100 in training costs.<\/p>\n\n\n\n A large company receives thousands of support issues per month. These issues are categorized as either: billing, onboarding, bug reports, access to accounts, integration issues, or enterprise escalations. A general-purpose model can categorize these issues, but it performs poorly because it is inconsistent. Frequently, it incorrectly identifies integration issues as bug reports. Just as often, it misroutes enterprise escalations.<\/p>\n\n\n\n Step-by-step solution:<\/strong> Collect a representative sample of each type of issue. Label each example as one of the six types listed above. Clean and normalize the labels. Create two separate files based on these six categories. Create a small base model and then fine-tune it using LoRA or QLoRA with Axolotl or Unsloth. Test the accuracy of the model against a holdout set. Once satisfied, deploy the adapter. Monitor errors in production and retrain when changes occur in the error distribution.<\/p>\n\n\n\n The result is a model that is less expensive to operate than having the model query a remote API for every single issue received by the help desk. The model categorizes issues consistently and makes it easy to add new categories as they become necessary.<\/p>\n\n\n\n The adapter file is typically 2-5 MB, smaller than most .zip files.<\/p>\n\n\n\n Read more: The rise of small language models and why open source is winning<\/a><\/strong><\/p>\n\n\n\n Developers were able to access the weights associated with open source models. Now, developers have access to open source fine-tuning tools allowing them to take those weights and develop truly relevant solutions for their specific industries. As Hugging Face’s PEFT documentation<\/a> notes, parameter-efficient methods make it “more accessible to train and store large language models on consumer hardware,” and the ecosystem around these tools continues to grow rapidly.<\/p>\n\n\n\n Every part of the development cycle, from preparing a dataset for fine-tuning through training, testing, and finally deploying, can be done entirely with open source tools. According to a 2026 comparison of fine-tuning frameworks<\/a>, LLaMA-Factory, Axolotl, Unsloth, and torchtune each serve different hardware profiles and use cases, giving teams real options regardless of their GPU budget.<\/p>\n\n\n\n Clearly there is potential here for developers. The fine-tuning workflow still has rough edges. There are several opportunities for developers to contribute including developing better tools for creating datasets for fine-tuning, developing better frameworks for evaluating fine-tuned models, and developing easier-to-use tools for managing adapters.<\/p>\n\n\n\n In my previous article<\/a>, I made the case that small, open source models can handle the majority of production AI workloads. Fine-tuning is how you close the remaining gap. The base model gives you general intelligence. What you train it on gives you domain expertise.<\/p>\n\n\n\n <\/p>\n","protected":false},"template":"","class_list":["post-8975","articles","type-articles","status-publish","hentry"],"acf":[],"yoast_head":"\nThe open source LLM fine-tuning stack: Tools for every layer<\/strong><\/h2>\n\n\n\n
When to use fine-tuning vs RAG vs prompting<\/strong><\/h2>\n\n\n\n
Fine-tuning in practice: Support ticket categorization with LoRA and QLoRA<\/strong><\/h2>\n\n\n\n
What open source fine-tuning means for developers today<\/strong><\/h2>\n\n\n\n
More from We Love Open Source<\/h2>\n\n\n\n
\n