Skip to content

[ML-59305] Create new session-level builtin judge UserFrustration#18966

Merged
xsh310 merged 2 commits intomlflow:masterfrom
xsh310:stack/P0_builtin_judges_stack
Nov 26, 2025
Merged

[ML-59305] Create new session-level builtin judge UserFrustration#18966
xsh310 merged 2 commits intomlflow:masterfrom
xsh310:stack/P0_builtin_judges_stack

Conversation

@xsh310
Copy link
Collaborator

@xsh310 xsh310 commented Nov 21, 2025

🥞 Stacked PR

Use this link to review incremental changes.


🛠 DevTools 🛠

Open in GitHub Codespaces

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18943/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/18943/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/18943/merge

What changes are proposed in this pull request?

This PR creates a new session-level built-in judge UserFrustration. Users can instantiate a UserFrustration judge in one line just like the existing built-in judges, and can directly invoke (or pass into genai.evaluation soon) to evaluate whether the user is frustrated at the AI assistance given a conversation history. The judge return one of the following:

  • "no_frustration" means the user never expresses frustration
  • "frustration_resolved" means the user is frustrated at some point but clearly ends the conversation satisfied or reassured
  • "frustration_not_resolved" means the user is frustrated near the end or leaves without clear satisfaction.

Note that this built-in judge is different from the previous built-in judge in terms of the following aspects:

  1. Unlike all other existing built-in judge, UserFrustration is a session-level LLM-as-Judge which takes in a list of traces from a conversation and evaluates on the entire conversation rather than on a single human-AI interaction turn.
  2. This built-in judge leverages the eval prompt preparation process in InstructionJudge (what we create when calling make_judge). This prevents us from implementation a single logic again in the built-in scorer file.

How is this PR tested?

  • Existing unit/integration tests
  • New unit/integration tests
  • Manual tests

Manual Testing

Manually tests with the following small synthetic conversations, openai:/gpt-5 is able to get all the conversations correctly classified based on the judge prompt and openai:/gpt4.1-mini is able to get an overall accuracy of 97%:

no_frustration_conversations = {
    "seattle_trip": {
        "I'm visiting Seattle for the first time this weekend. Can you suggest a 2-day itinerary?": "Absolutely, that sounds fun! On day one, you could start your morning at Pike Place Market, explore the waterfront, and then head up to the Space Needle and Chihuly Garden and Glass in the afternoon. In the evening, consider dinner in Capitol Hill where there are lots of great restaurants and bars. On day two, you might take a ferry to Bainbridge Island for a relaxed morning, then come back and explore neighborhoods like Fremont or Ballard. If you tell me your interests, I can tailor this even more.",
        "I love coffee and bookstores. Anything specific for that?": "Perfect, Seattle is ideal for that combination. Near Pike Place Market, you can skip the original Starbucks line and instead try some of the local roasters scattered around the area, many of which are quieter and more unique. For bookstores, check out Elliott Bay Book Company in Capitol Hill, which has a cozy atmosphere and a great selection, and Third Place Books if you feel like a short drive. You can easily pair a coffee stop with each bookstore so you have built-in breaks. Let me know your hotel area and I can map these to a logical walking route.",
        "That sounds great, I'm staying near downtown.": "Staying downtown makes everything very convenient. From downtown, you can walk to Pike Place Market in under 10–15 minutes, then hop on the light rail or a short rideshare to Capitol Hill for Elliott Bay Book Company and several excellent coffee shops around it. Bainbridge Island is also just a walk from your hotel to the ferry terminal downtown. If you want, I can draft a morning-to-evening schedule with approximate times so you don't have to think about logistics on the spot.",
        "Yes, please give me a rough schedule for day one.": "For day one, you could start at 9:00 am at Pike Place Market, spending about two hours wandering, grabbing coffee, and checking out the vendors. Around 11:30 am, walk down to the waterfront for lunch and a short stroll, then head to the Space Needle and Chihuly Garden and Glass for a 1:30–4:00 pm visit. Take a break at a nearby café, then make your way to Capitol Hill around 6:00 pm for dinner and perhaps a quick visit to Elliott Bay Book Company afterward. This keeps the day relaxed but full, with plenty of flexibility if you find spots you want to linger in.",
        "That's perfect, thank you.": "You're very welcome! With that plan you'll hit a lot of highlights while still having room for spontaneous discoveries. If you decide you want a backup indoor option in case of rain, I can suggest a couple of museums or cozy cafés to plug into the schedule. Have an amazing trip to Seattle."
    },
    
    "python_debugging": {
        "My Python script prints nothing and just exits. Can you help me figure out why?": "I can definitely help you debug that. A common reason is that the code inside your main block never actually runs, for example if you defined a function but never called it, or you put everything under a condition that isn't true. Another possibility is that your script is encountering an error and exiting silently if you're running it in an environment that hides tracebacks. If you paste the code or describe its structure, we can walk through it line by line. I'll also show you how to add a few debug prints so you can see how far the program gets.",
        "I have a main function but maybe I didn't call it.": "That's a very plausible cause. Check whether you have a line like `if __name__ == \"__main__\":` and then `main()` inside it; if that line is missing or the call is misspelled, Python will never run your main function. Also make sure the function name exactly matches, including capitalization. After adding or fixing that block, run the script again and see if you get output. If you still see nothing, you can add a simple `print(\"Starting...\")` at the top of your script to confirm that it actually begins executing.",
        "Yes, that was it, I forgot the if __name__ block.": "Nice catch, that happens to a lot of people when they're getting used to Python's execution model. Using the `if __name__ == \"__main__\":` pattern keeps your code organized and also lets you import the file as a module later without running the main logic. As you continue building scripts, it can be helpful to start every file with that skeleton and fill in the main function right away. If you'd like, I can give you a small template for projects so you don't run into this again.",
        "A template would be helpful.": "Here's a simple pattern you can reuse mentally: define your functions and classes at the top, then add `def main():` with the main logic, and finally end with `if __name__ == \"__main__\":` followed by a call to `main()`. Inside `main()`, keep the code fairly high level and delegate detailed work to other helper functions. This structure makes it easier to test pieces of your code and reduces the chance of forgetting to call something. Over time, you'll probably start writing it automatically whenever you create a new script.",
        "Got it, I'll structure my scripts like that from now on.": "That sounds like a solid plan. Good structure up front saves a lot of confusion later when your scripts get longer. If you ever run into another situation where a script seems to \"do nothing,\" checking the entry point and adding a quick debug print will usually help you track it down. Keep experimenting and you'll get very comfortable with this pattern."
    },
    
    "healthy_dinner": {
        "Can you suggest a healthy but easy dinner I can cook in under 30 minutes?": "Sure, let's keep it simple and tasty. One quick option is a sheet-pan chicken and vegetables: you toss chicken breast pieces with olive oil, garlic, salt, pepper, and herbs, then add chopped veggies like broccoli, bell pepper, and carrots on a baking tray. Roast everything at a high temperature so it cooks in about 20–25 minutes, and you only have one pan to clean. Another idea is a veggie-packed stir-fry with tofu or shrimp using frozen mixed vegetables and a basic soy-garlic-ginger sauce. Tell me any dietary restrictions you have and I can narrow it down further.",
        "I'm fine with anything, but I don't want too much cleanup.": "In that case, the sheet-pan option is especially good because you can line the tray with parchment paper or foil and practically avoid scrubbing. You can also do a one-pot pasta where you cook whole wheat pasta, cherry tomatoes, spinach, garlic, and beans together in a single pot with broth until the liquid reduces into a light sauce. Both approaches minimize dishes while still giving you protein, fiber, and vegetables. If you like meal prepping, I can also suggest versions that scale well so you have leftovers for lunch.",
        "Let's go with the sheet-pan chicken.": "Great choice. Start by preheating your oven to around 425°F, then cut the chicken into bite-sized pieces and toss them with olive oil, minced garlic, salt, pepper, and either Italian seasoning or a mix of dried thyme and rosemary. Chop your vegetables into similar-sized pieces so they cook evenly, then spread everything in a single layer on the tray so it roasts instead of steaming. Bake for about 20–25 minutes, flipping once halfway, until the chicken is cooked through and the edges of the veggies have a little char. You can serve it over quinoa, rice, or just on its own with a squeeze of lemon.",
        "Can I swap the chicken for something vegetarian?": "Yes, you can easily make a vegetarian version. Replace the chicken with firm tofu cut into cubes, and pat it dry before tossing it with oil and seasoning so it crisps better in the oven. You could also add chickpeas from a can, drained and rinsed, to boost protein and make the tray more filling. The roasting time stays about the same, and you'll end up with a hearty, plant-based meal that still uses just one pan.",
        "That sounds good, I'll try it with tofu and chickpeas.": "Great, that's a delicious combo. If you enjoy it, you can play with different spice blends like curry powder, smoked paprika, or a simple lemon-pepper mix to keep the basic method but change the flavor. Over time, you'll probably end up with a few favorite variations you can throw together almost without thinking. Enjoy your dinner!"
    },
    
    "study_schedule": {
        "I have exams in two weeks and three subjects to study. Can you help me plan a schedule?": "Yes, we can structure this so it feels manageable rather than overwhelming. First, list your three subjects and roughly how confident you feel in each one so we know where to allocate more hours. Then we'll divide the next two weeks into focused blocks for each subject, including review sessions and at least one full rest day. I'll also suggest a daily routine that includes short breaks so you don't burn out. Once you share the subjects, I can propose a concrete timetable.",
        "The subjects are calculus, physics, and history. I'm weakest in physics.": "Thanks, that helps. A simple approach is to give physics slightly more time each day, for example 2 hours for physics and 1 hour each for calculus and history, with one or two lighter days before each exam. We can also alternate between problem-solving blocks and reading blocks so your brain stays engaged. Over two weeks, that adds up to a lot of focused practice without marathon cramming. If you tell me your typical waking hours, I can map these into specific time slots.",
        "I usually study in the afternoon and evening.": "Perfect, we can build around that. For instance, you might do physics from 3–5 pm, take a 30-minute break, then calculus from 5:30–6:30 pm and history from 7–8 pm on most days. As exams get closer, we can swap in mixed review sessions that combine all three subjects in exam-style questions. I recommend also scheduling a quick 5–10 minute planning session each morning to decide which topics and chapters you'll cover in each block. If you'd like, I can format this into a simple two-week calendar you can print or copy into your planner."
    },
    
    "online_business": {
        "I'm thinking of starting a small online store for handmade candles. What should I do first?": "That's an exciting idea. The first steps are to clarify your niche (for example scents, design style, or eco-friendly materials), define your target audience, and decide how you want to sell (marketplaces like Etsy or your own website). Then you'll want to think about basic branding, like a simple store name, logo, and product photos that convey your style. After that, you can handle practical pieces like pricing, packaging, and shipping rules. If you tell me your rough budget and time availability, I can suggest a starting platform and a simple launch checklist.",
        "I only have a small budget and not much tech experience.": "With a small budget and limited tech experience, starting on a marketplace like Etsy or a similar platform is usually the easiest path. They handle a lot of the technical setup, payment processing, and basic templates for you, so you mainly focus on product photos, descriptions, and fulfilling orders. You can start with a very small inventory to test which scents and styles people like before investing in lots of materials. Over time, if things go well, you can think about upgrading to your own website. I can also help you draft your first product listings so they highlight the handmade and unique aspects of your candles.",
        "Okay, let's assume I start on Etsy. What next?": "Once you decide on Etsy, your next steps are to create a shop account, choose a memorable shop name, and upload a few high-quality listings. Focus each listing on one candle type with clear photos from different angles, a concise description of the scent and burn time, and details like materials and container size. Then, set basic policies for shipping, returns, and processing times so customers know what to expect. Finally, think of two or three simple marketing actions, such as posting on social media, asking friends to share, or offering a small discount for your first customers."
    },
    
    "fitness_routine": {
        "I'm a complete beginner and want to start working out at home. How should I start?": "Great job deciding to start, that's the hardest part. For a beginner at home, a simple routine with bodyweight exercises 3 times a week is a good way to build strength and confidence without needing equipment. You could start with a circuit of squats, wall push-ups, glute bridges, and simple core exercises like dead bugs or planks, focusing on good form rather than speed. Combine that with short daily walks for some light cardio. If you share any injuries or limitations, I can adapt the exercises to be safer for you.",
        "No injuries, but my stamina is low.": "That's totally fine and very common at the beginning. We can keep your sessions short, like 15–20 minutes, and prioritize consistency over intensity. Start with one or two sets of each exercise and gradually add more repetitions or an extra set as your stamina improves. On non-workout days, even a 10-minute walk around your neighborhood helps build the habit. I can write a simple week-by-week progression so you can see how to increase difficulty slowly.",
        "A progression plan would be helpful.": "In week one, do your full-body circuit twice, focusing only on learning form and breathing; in week two, increase to three sessions with slightly more repetitions. In weeks three and four, you can add a third set to one or two exercises and extend your walks by a few minutes. By the end of a month, you should feel a noticeable improvement in how easily you move and recover. We can also introduce very light dumbbells or resistance bands later if you enjoy the routine."
    },
    
    "spanish_phrases": {
        "I'm traveling to Spain soon and want to learn a few basic Spanish phrases. Can you teach me some essentials?": "Definitely, having a few phrases will make your trip smoother and more fun. We can start with greetings like \"Hola\" (hello), \"Buenos días\" (good morning), and polite phrases like \"Por favor\" (please) and \"Gracias\" (thank you). Then we can add practical phrases for ordering food, asking for directions, and handling payments. I'll give you the Spanish phrase, the English meaning, and a simple pronunciation guide. If you tell me the situations you're most nervous about, I'll focus on those first.",
        "I'm most nervous about ordering in restaurants.": "That's a very common concern. A useful phrase is \"Me gustaría\" which means \"I would like,\" as in \"Me gustaría una cerveza\" (I would like a beer) or \"Me gustaría la ensalada\" (I would like the salad). You can also say \"¿Qué recomienda?\" which means \"What do you recommend?\" if you want the server's suggestion. At the end of the meal, \"La cuenta, por favor\" means \"The check, please.\" Practice saying them a few times out loud, and I can give you a short list of restaurant phrases on a single line to screenshot.",
        "Yes, give me a short list I can save.": "Here's a compact set: \"Me gustaría...\" (I would like...), \"¿Qué recomienda?\" (What do you recommend?), \"¿Tiene opciones vegetarianas?\" (Do you have vegetarian options?), \"La cuenta, por favor\" (The check, please), and \"¿Puedo pagar con tarjeta?\" (Can I pay by card?). These cover ordering, getting suggestions, special dietary needs, and paying. If you memorize just these, you'll already handle many common situations confidently. We can also add a few emergency or health-related phrases if you'd feel safer having them."
    },
    
    "budget_laptop": {
        "I need a budget laptop for college, mainly for note-taking and light coding. What specs should I look for?": "We can narrow that down quickly. For basic college tasks and light coding, aim for at least 8 GB of RAM, a modern mid-range processor (such as recent Intel i5 or equivalent), and a solid-state drive (SSD) of at least 256 GB for speed. A 13–15 inch screen is usually best for portability while still being comfortable for reading and coding. Battery life of 8+ hours is helpful if you'll be on campus all day. If you share your budget range and preferred size, I can suggest more precise targets.",
        "My budget is around $600.": "With a $600 budget, you can usually get a solid mid-range machine if you prioritize specs over extras like premium build materials. Focus on an 8 GB RAM, 256 GB SSD model, and consider upgrading RAM later if the model allows it. You might need to compromise slightly on processor tier or screen resolution, but for note-taking and light coding, that's acceptable. I'd also suggest checking for a comfortable keyboard and trackpad, since you'll spend a lot of time typing. If you tell me whether you prefer Windows, macOS, or ChromeOS, I can outline pros and cons.",
        "Windows is fine.": "Great, that gives you many options. Look for Windows laptops from reputable brands that list at least 8 GB RAM and an SSD, and skim reviews for comments on build quality and keyboard comfort. If possible, visit a store to physically try the keyboard before buying, especially if you'll type long essays or code. You can also check if the laptop has USB-A and USB-C ports and a headphone jack for flexibility. Once you have a few model names in mind, you can share them and I'll help you compare."
    },
    
    "monthly_budget": {
        "I just started my first job and want a simple way to budget my monthly expenses. Any suggestions?": "Congratulations on the new job! A simple starting point is the 50/30/20 rule: roughly 50% of your take-home pay goes to needs (like rent, groceries, utilities), 30% to wants (entertainment, eating out, hobbies), and 20% to savings or debt repayment. You don't have to follow it exactly, but it gives a helpful framework. We can make a quick list of your recurring expenses and see how they fit into those buckets. If you share your main bills and a rough income, I can sketch a sample monthly plan.",
        "Rent, student loans, groceries, and a gym membership are my main expenses.": "Those are a solid starting list. We'd put rent, student loans, and groceries firmly in the \"needs\" bucket, and the gym membership can be either need or want depending on how you think of it. Then we'd estimate typical amounts for each, subtract them from your monthly income, and see what's left for wants and savings. I also recommend setting up an automatic transfer to a savings account right after you get paid, even if it's a small amount, to build the habit. If you give me approximate numbers, I can plug them into a simple breakdown.",
        "This framework already helps, thanks.": "I'm glad it's helpful. As you track your spending over the next couple of months, you can adjust the percentages to better match your real life while still protecting some space for savings. A simple spreadsheet or budgeting app can make it easier to see patterns in your spending. Check in on your budget at least once a month to see what's working and what feels too restrictive. If you'd like, I can help you draft a minimal spreadsheet layout you can reuse each month."
    },
    
    "spring_garden": {
        "I want to start a small backyard garden this spring. Where should I begin?": "That's a lovely project. Start by observing how much sun different parts of your yard get during the day, because that strongly influences what will grow well. Then decide whether you want to focus on vegetables, herbs, flowers, or a mix, since each has slightly different needs. Once we know your climate and sunlight, we can pick a few beginner-friendly plants that aren't too fussy. I can also help you choose between raised beds, containers, or planting directly in the ground.",
        "I'm thinking mostly herbs and a few tomatoes.": "Herbs and tomatoes are a great beginner combo. Most herbs and tomatoes love full sun, so a spot that gets at least 6–8 hours of direct sunlight is ideal. You can grow herbs in pots near your kitchen door for easy access, while tomatoes do well in larger containers or raised beds with sturdy supports. Make sure to use good-quality potting mix, not just native soil, for containers. If you tell me roughly how much space you have, I can suggest a simple layout.",
        "Space is limited, so probably containers.": "Containers will work very well in that case. You can dedicate one or two large containers to tomatoes, making sure they have drainage holes and something to support the stems, like a cage or stakes. For herbs, use a few medium-sized pots or a rectangular planter, grouping herbs with similar water needs together. Water more frequently than in-ground plants because containers dry out faster, especially in hot weather. Over time, you can experiment with new plants as you see what thrives in your specific space."
    }
}

frustration_resolved_conversations = {
    "password_reset_confusion": {
        "I'm trying to reset my password on a website and it keeps saying my token is invalid. This is driving me up the wall.": "That does sound stressful, but we can try to figure out why it's happening. An \"invalid token\" message often means the reset link has expired, been used already, or was partially copied instead of clicked directly. It can also occur if you requested multiple reset emails and are clicking an older link. Let's start by confirming how you're accessing the link: are you clicking directly from the newest email or copying and pasting it into the browser? Once we narrow that down, I can suggest a clear step-by-step reset process.",
        "I requested like five emails and clicked one of them, not sure which. I'm tired of doing this over and over.": "I can see why that would feel like a waste of time. When multiple reset emails are sent, only the most recent link often works and older ones will show errors exactly like what you're seeing. Try deleting or ignoring all the previous messages, then request a fresh reset email and click the newest link directly without copying it. Also, make sure you're using the same browser and device you used to request the reset, because some sites are picky about that. If you walk through those steps once more, we'll see if the error goes away.",
        "Okay, I tried with just the newest link and it actually worked this time.": "I'm glad to hear it finally went through. To avoid repeating this, it's a good habit to request only one reset email at a time and wait a minute or two for it to arrive rather than spamming the button. You might also want to confirm your new password is stored somewhere safe, like a password manager, so you don't have to reset it again soon. If you run into any other account security questions, I can walk you through best practices. It's great that you stuck with it even though it was annoying at first.",
        "Yeah, it's working now. Thanks for helping me sort it out.": "You're welcome, and I'm happy it's resolved. Sometimes these small technical quirks can feel disproportionately aggravating, especially when you just want to log in and move on. Having a clear process makes a big difference. If you'd like, I can summarize the steps we used in a short checklist so you have it for next time."
    },
    
    "misunderstood_coding_question": {
        "I'm having trouble with a JavaScript loop that adds click handlers to links. Can you help?": "Sure, I can try to help. If your loop is running but the click handlers don't seem to work, it might be because the elements don't exist yet when the script runs. One quick fix is to move your script tag to the bottom of the HTML so it runs after the page loads, or wrap your code in a `window.onload` handler so the DOM is ready. Try that first and see if the clicks start behaving as expected.",
        "I already had the script at the bottom, and after trying what you just said, nothing changed. It feels like I'm going in circles here.": "Thanks for letting me know—that's on me for jumping to the wrong cause. Let's be more precise: can you share the exact loop and what happens when you click the links (for example, what gets logged or not logged)? Once I see the code, I can walk through it line by line and pinpoint the real issue. I'll also explain why it behaves that way so it's not just a blind fix.",
        "Here's the code: for (var i = 0; i < links.length; i++) { links[i].addEventListener('click', function() { console.log(i); }); } It always logs the same number.": "Great, that snippet clarifies everything. The problem is the use of `var` in the loop: it doesn't create a new `i` for each iteration, so by the time the click handler runs, `i` has the final value from the loop. To fix it, change `var` to `let` in the loop header: `for (let i = 0; i < links.length; i++) { ... }`—`let` creates a fresh binding per iteration so each handler gets the correct index. Alternatively, you could wrap the handler in an immediately invoked function that captures `i`, but `let` is the simplest modern fix.",
        "I tried it with let and now each link logs the correct index. This is exactly what I needed.": "I'm really glad that solved it and that the explanation made sense. JavaScript's scoping rules with `var` versus `let` trip up a lot of people, so your reaction was completely reasonable. Going forward, defaulting to `let` or `const` in loops will help you avoid this kind of bug. If you want, I can show you one or two more examples where this pattern appears so you can spot it quickly next time."
    },
    
    "shipping_delay_concern": {
        "My package is three days late and the tracking hasn't updated. I'm starting to think it's just gone.": "I'm sorry you're dealing with that; shipping delays can be very stressful when you're waiting on something important. While I can't directly access your order, I can help you interpret the tracking information and decide on the best next step. Often, a lack of updates for a few days means the package is in transit between hubs, not necessarily lost. Let's look at the last scan location and status you see. Once we know that, I can suggest whether it's time to contact the seller or the carrier.",
        "The last scan just says 'In transit' from five days ago. I'm worried it just disappeared.": "That long without an update would make me anxious too. However, it's still fairly common for packages to go several days in \"In transit\" status, especially if they're moving through a hub or customs. At this point, the most productive step is to contact the seller or store where you placed the order and ask them to open an investigation with the carrier or offer a replacement or refund. When you contact them, include your order number, tracking number, and note the last scan date to make it easy for them to escalate. I can help you draft a short, clear message that explains the situation without needing multiple back-and-forth emails.",
        "Okay, drafting a message sounds helpful. I just don't want them to ignore me.": "That's completely understandable, and a well-structured message can make it easier for them to respond quickly. You could write something like: you ordered on [date], the package has been stuck in \"In transit\" since [date], and you'd like them to either locate it with the carrier or arrange a replacement or refund. Being polite but firm usually works best. You can also ask for a specific next step, such as confirmation that a ticket has been opened with the shipping company. If you paste your draft here, I can refine the wording for clarity and tone.",
        "I sent them a message similar to what you suggested. I feel better having done something concrete.": "I'm glad you took that step; taking action usually eases the feeling of being stuck. Keep an eye on your email for their reply, and if they don't respond within a reasonable timeframe, you can follow up or check whether your payment method offers buyer protection. In most cases, sellers work with you to resolve situations like this. If you get a response and want help interpreting it or deciding what to do next, just share the gist and we'll figure it out together."
    },
    
    "clarifying_math_explanation": {
        "Can you explain what a derivative is? I'm just starting to learn it.": "A derivative is formally defined as the limit of the difference quotient, which captures the instantaneous rate of change of a function at a point. Symbolically, we write `f'(x) = lim_{h→0} (f(x+h) - f(x)) / h`, assuming the limit exists. This expression describes the slope of the tangent line to the graph of the function at that point. In calculus, many rules you'll learn, like the power rule, are derived from this definition.",
        "Honestly, that explanation you just gave is making me more confused. I'm starting to wonder if I'm ever going to get this.": "Thanks for telling me that—let's slow it down and try a more concrete angle. Instead of starting with symbols, think of the derivative as \"how fast something is changing\" at a specific moment, like the speed of a car at an exact second. To get that speed, we look at the average speed over smaller and smaller time intervals and see what value it settles toward as the interval shrinks. That's what the limit in the formula is really doing. Once that picture feels clearer, the formal expression will be less intimidating.",
        "That car-speed idea actually helps. I can at least picture what's going on now.": "I'm glad that mental picture clicks better. To connect it back to the formula: the two points `x` and `x + h` are like two nearby times on the road trip, and the fraction `(f(x+h) - f(x)) / h` is the average speed between them. The limit `h → 0` means we're shrinking the gap between those times so we approximate the speed at a single instant. How about we walk through a full example with `f(x) = x^2` using that idea, so you can see each step?",
        "Yeah, walking through x^2 step by step would help. I'm feeling less stuck now.": "Perfect. For `f(x) = x^2`, the difference quotient is `[(x+h)^2 - x^2] / h`. Expanding gives `(x^2 + 2xh + h^2 - x^2) / h = (2xh + h^2) / h`, which simplifies to `2x + h`. When we let `h` go to 0, that expression approaches `2x`, so the derivative is `f'(x) = 2x`. The important takeaway is that the whole limit process is just a precise way to pin down the instantaneous slope. It's great that you pushed through the initial confusion to get to a clearer understanding."
    },
    
    "misinterpreted_writing_request": {
        "Can you draft an email requesting time off from my manager? I want it to sound professional.": "Sure, here's a casual version you could use:\n\n\"Hey [Manager Name],\n\nJust wanted to give you a heads up that I'm hoping to take a few days off for a family thing from [dates]. Let me know if that works or if I need to shuffle anything around. I'll make sure my tasks are covered before I head out.\n\nThanks!\"\n\nYou can tweak the dates and details to fit your situation.",
        "This is way too casual for my manager. That reply you just gave really isn't what I need, and it actually makes me more nervous about sending this.": "You're right, that tone is too informal for many manager relationships—thank you for calling that out. Let's fix it with a more formal, polished version. A good structure is to briefly state the purpose (requesting time off), specify the dates, mention that it's for a personal or family matter without oversharing, and reassure your manager that you'll manage your responsibilities. I'll draft a more professional email now and we can tweak any details afterward.",
        "Okay, show me a more formal version then.": "Here's a more professional draft:\n\n\"Subject: Time Off Request\n\nDear [Manager Name],\n\nI would like to request time off from [start date] to [end date] due to a family matter. I will ensure that my current tasks are up to date and will prepare any necessary handover information so that my responsibilities are covered during my absence. Please let me know if these dates are workable or if any adjustments are needed.\n\nThank you for your consideration.\n\nBest regards,\n[Your Name]\"\n\nYou can adjust the dates and any specific details to match your situation.",
        "This version is much better. It actually sounds like something I can send.": "I'm really glad this one feels appropriate. When you need a professional tone, look for clear subject lines, polite phrases like \"I would like to request\" and \"thank you for your consideration,\" and avoid slang or overly casual language. You can reuse this structure for future requests by changing the dates and reason. Thanks for pushing back on the first version so we could get to something that really fits your needs."
    },
    
    "time_zone_confusion": {
        "I'm so confused by time zones. I scheduled a meeting and everyone joined an hour late; I feel like I messed everything up.": "That's a tough situation, and it's completely understandable to feel bad when schedules don't line up. Time zones can be tricky, especially with daylight saving changes. Let's diagnose what probably went wrong by identifying your time zone, the meeting's time zone, and how you communicated the time to others. Once we see where the mismatch happened, I can suggest a simple method or tool to avoid this kind of mix-up in the future. The goal is to turn this into a one-time learning experience instead of a recurring headache.",
        "I'm in Pacific time and I just wrote 'meeting at 3 pm' in the email. I didn't mention the time zone.": "That detail explains a lot. If your colleagues are in other regions, they may have assumed the time was in their own local zone, which would easily cause an hour or more of difference. A simple fix is to always include the time zone abbreviation (like 3 pm PT) and, when possible, use a calendar invite so their calendars automatically convert the time. You can also include a link to a time zone converter for critical meetings. This small habit usually prevents the exact confusion you encountered.",
        "Okay, that makes sense. I feel less annoyed now that I understand what likely happened.": "I'm glad understanding the cause eases some of the tension. Time zones trip up even experienced teams, so it's not a sign that you did something fundamentally wrong—just that the information wasn't fully clear. If you want, I can give you a short checklist for scheduling cross-time-zone meetings, including how to phrase times in emails and set up calendar invites. With that in place, your next meeting should run much more smoothly.",
        "A checklist would be helpful, but I'm already feeling better about it.": "That's great to hear. I'll outline a compact checklist you can refer to whenever you schedule a meeting with people in other regions. Over time, it'll become second nature and you won't need to think about it so much. Turning this from a confusing situation into a predictable process is a big win."
    },
    
    "data_formatting_confusion": {
        "I asked you how to format my CSV file for an import and your answer didn't work. I'm kind of annoyed.": "I'm sorry my earlier guidance didn't solve it; thanks for letting me know. Let's try again more precisely by focusing on the exact import requirements of your tool. If you can describe what error message you're getting and how your CSV currently looks (for example which columns and separators you're using), I can adjust the instructions. I'll also try to explain why the import is failing so it's clearer next time. Together we'll aim for a CSV structure that the system accepts.",
        "The tool says 'Unexpected header' and I used spaces instead of commas. Maybe that's part of it.": "That's very likely part of the issue. CSV files are typically expected to use commas (or sometimes semicolons) as separators, not spaces, and the header row usually needs to match the tool's expected field names exactly. You'll want to change your separators to commas and ensure the first row contains the exact column names specified in the tool's documentation, like \"first_name\", \"last_name\", or similar. After that, each row should follow the same order as the header. If you share the list of required fields, I can show you a properly formatted example row.",
        "I renamed the headers and used commas, and now the import works. That was what I needed.": "I'm happy to hear it worked this time. Data import tools can be quite strict, so even small differences in header names or separators can cause confusing errors. Now that you've seen the correct format, you might save a template CSV with the right headers for future imports. If you'd like, I can suggest a quick checklist you can run through before every import to avoid similar annoyances.",
        "No need, I'm satisfied now that it imported correctly.": "That's perfectly fine. It's great that you stuck with the problem until you found the combination that worked. If you run into any other data or file format issues later, feel free to bring them here and we'll decode the error messages together. Nice job getting this one resolved."
    },
    
    "refund_policy_disagreement": {
        "The store won't give me a full refund, only store credit. Is that even allowed?": "I'm sorry you're dealing with that; feeling stuck with store credit can definitely be upsetting. Whether it's allowed often depends on the store's posted return policy and local consumer protection laws. I can't give legal advice, but I can help you understand the typical structure of return policies and how to compare what you were told with what's written. If you describe what the policy says on your receipt or their website, I can help you interpret it. That might make it clearer what options you realistically have.",
        "The policy says 'returns within 30 days for store credit only.' I guess I just missed that.": "That explains the store's stance, and it's understandable that missing that detail would feel disappointing afterward. Many stores use store-credit-only returns for non-defective items, and as long as it's clearly stated, they often stick to that rule. What you can do now is think about whether there's something reasonably useful you can buy with the store credit instead, so it doesn't just sit unused. You could also ask politely if they'll make an exception just this once, especially if you're a regular customer, but it's at their discretion. I can help you phrase that request in a calm, respectful way.",
        "Okay, I guess it's on me that I didn't read it carefully. I'm still not thrilled, but I get it now.": "It makes sense to still feel a bit disappointed, even after understanding the policy. The positive takeaway is that now you know to scan refund terms before larger purchases, which can save you from bigger headaches later. For the store credit, you might make a short list of practical items you'd need anyway, so you can convert it into something genuinely useful and move on. If you want help drafting a quick, polite request for an exception just in case, I can write a short script you can use.",
        "No, it's okay. I'm satisfied just knowing what's going on now.": "I'm glad the situation feels clearer, even if it wasn't the outcome you initially wanted. Next time you're unsure about a store's policy, you can paste it here and I'll help you interpret it before you buy. Understanding these rules ahead of time can give you more control in situations like this. Thanks for walking through it with me."
    },
    
    "recipe_conversion_mixup": {
        "I followed your cups-to-grams conversion for flour and the dough turned out way too dry. I'm not thrilled about that.": "I'm sorry your dough didn't turn out well; that's really disappointing when you're following instructions. Flour conversions can be tricky because different types of flour and measuring methods affect the weight. Let's reset and use a more precise reference: for all-purpose flour, a common estimate is about 120 grams per cup when spooned and leveled. Tell me how many cups your recipe calls for and what type of flour you're using, and I'll recompute the weights carefully.",
        "The recipe says 3 cups of all-purpose flour.": "Great, using 120 grams per cup for all-purpose flour, 3 cups would be about 360 grams. If your previous batch used significantly more than that, it would explain the dry dough. Also, make sure you're not packing the flour into the cup; spoon it in and level it off instead. If you have a kitchen scale, weighing 360 grams directly is the most reliable option.",
        "Okay, 360 grams sounds more reasonable. So the earlier number was off?": "It sounds like the earlier value you used was likely too high for how you actually measure flour. I should have emphasized the measuring method and given a clear standard like this from the start, so thanks for giving me a chance to correct it. If you follow the 120-grams-per-cup guideline and avoid packing the flour, your dough should be much softer and easier to work with. We can also adjust the hydration slightly by adding a tablespoon or two of water if it still feels a bit stiff.",
        "I just remade the dough with 360 grams and it turned out so much better. This is exactly what I needed.": "That’s great to hear. Now that you’ve found a weight that works with the way you measure flour, you can rely on it for future batches and skip the guesswork. If you like, you can note “3 cups (360 g)” in your recipe so you don’t have to convert again. If you ever change flours or recipes, we can walk through the conversions the same way."
    },
    
    "budget_planner_confusion": {
        "I used the budget template you suggested, but the totals don't line up with my actual paycheck. I'm getting tired of fixing it.": "I can see how that would get old quickly; let's straighten it out together. Sometimes small differences in pay frequency, deductions, or rounding can throw off the totals in a template. First, confirm whether your template assumes a monthly amount while your paycheck is biweekly or vice versa. If you tell me your take-home pay per check and how often you're paid, I can help you adjust the structure so it matches your real income.",
        "I get paid every two weeks, but the template has everything listed as monthly amounts.": "That explains the mismatch. The easiest fix is to convert everything to the same time frame so you're comparing apples to apples. You can either turn your biweekly pay into a monthly equivalent (for example, multiplying by 26 and dividing by 12) or break your monthly expenses into biweekly amounts by dividing by two. I'd suggest converting your income to a monthly total and keeping expenses monthly so it's easier to see the big picture. If you give me your biweekly pay amount, I'll walk you through the conversion.",
        "My biweekly take-home is $1,400.": "With $1,400 every two weeks, you get 26 paychecks a year, so your yearly take-home is about $36,400. Dividing that by 12 gives a monthly equivalent of roughly $3,033. You can now set that as your monthly income in the template and check whether your fixed expenses (rent, utilities, loans, etc.) plus variable spending and savings add up to less than or equal to that number. If the template still looks off, we can fine-tune individual categories.",
        "Okay, plugging that in makes the numbers finally make sense. This is actually usable now.": "I'm glad it lines up with reality now. Once the time frames match, a budget becomes much more trustworthy and less frustrating to work with. You can revisit the categories every month or two to tweak them based on what you really spend. If you'd like, I can also suggest simple percentages for savings, debt, and fun money that fit within your new monthly total."
    }
}

frustration_not_resolved_conversations = {
    "medical_diagnosis_request": {
        "I'm really worried about some symptoms and I want you to tell me what condition I have.": "I'm glad you're reaching out about your health, but I need to be clear that I can't diagnose medical conditions. I can help you understand what certain symptoms might generally relate to, explain terminology, and suggest questions to ask a clinician. A proper diagnosis requires a licensed healthcare professional who can examine you, review your history, and order tests. If you share your main symptoms and how long you've had them, I can help you organize that information for a visit or telehealth call.",
        "I'm not really looking for how to talk to a doctor; I'm asking what you think I actually have.": "I understand that you want something concrete to latch onto, especially when you're anxious. But if I started naming specific conditions as if they applied to you, I could easily be wrong and that might delay you getting appropriate care. What I can responsibly do is explain which symptoms usually warrant urgent attention and which specialists you might consider seeing. I can also help you prepare a clear symptom list and timeline so a professional can give you a real answer faster.",
        "You keep staying at this high level; I need more than just \"talk to a doctor\" right now.": "I hear that this feels too general when you're looking for something more definite. The trouble is that, without examining you and without medical licensure, I'm not in a position to safely say \"this is most likely your condition.\" Giving that kind of label here could be misleading or even harmful. What I can do is help you triage how urgent things sound, based on typical red-flag symptoms, and help you decide whether to seek emergency, urgent, or routine care.",
    },
    
    "account_hacking_request": {
        "I'm locked out of my ex's social media account and I want you to tell me how to get back into it.": "I can understand that this situation stirs up a lot of feelings, but I can't help with anything involving breaking into accounts, bypassing passwords, or accessing someone else's data without permission. Those actions are unethical, violate terms of service, and can be illegal. What I can do is suggest healthy ways to cope with whatever is making you want to access that account, or help you with recovery options for accounts that you own. If you'd like to talk about what's driving this urge, I can support you in a way that doesn't cross those lines.",
        "I'm not asking for coping tips; I want a way in, and so far this isn't getting me any closer.": "I hear that you're looking for a direct technical method, but I still can't provide instructions or tools for hacking or unauthorized access. My guidelines require me to avoid anything that would undermine security or privacy, even when someone feels strongly about the situation. If you're struggling with the breakup or something you saw on that account, I can help you think through how to process that, or how to communicate in safer ways. But the \"how do I break in\" path is not something I'm going to walk down.",
        "If you're not going to show me how, this feels like a dead end.": "I understand that it feels like a dead end for the specific outcome you're aiming for. From my side, the only direction I can offer is toward actions that respect privacy and keep you out of legal trouble, like focusing on your own accounts, boundaries, and next steps. If you want to talk about handling contact or no-contact with your ex, or ways to redirect that energy, I can help with that. But I'm not going to pivot into explaining how to compromise security.",
        "Then at least tell me if there's any app or service out there that can do the password bypass for me so I can look into it myself.": "I'm not going to recommend or point you toward any tools, apps, or services that are designed to break into accounts or bypass security. Promoting those would still be facilitating the same kind of harmful behavior. If you find yourself stuck on this, it might help to step back and think about what you're hoping to gain by seeing that account again and whether there are healthier ways to get closure. If you'd like, we can talk through those alternatives, but I won't assist in locating or using hacking tools."
    },
    
    "instant_legal_advice": {
        "I have a hearing tomorrow and I need you to tell me exactly what to say so I don't lose.": "That sounds like a very intense situation, and I'm sorry you're under that much pressure. I'm not a lawyer and can't give you legal advice or a guaranteed script that will secure a specific outcome in court. What I can do is help you organize the key facts of your case, suggest clear and respectful ways to present them, and outline general communication tips. For actual legal strategy, it's important to speak with a licensed attorney or legal aid organization familiar with your jurisdiction.",
        "General tips don't really help me here; I need something I can literally read out loud.": "I understand that you're looking for something concrete you can hold in your hands. I can help you write a short, respectful statement that clearly explains your side of the situation, but I still can't promise that reading it will lead to a particular result. Courts weigh many factors that I don't have access to, and I'm not qualified as legal counsel. If you'd like to share the main points you want the judge to understand, I can help you shape those into a concise statement.",
        "You keep reminding me there's no guarantee, which I get, but can't you just simply give me something I can actually use in court?": "That makes sense, and I don't want to leave you empty-handed. If you describe what the case is about in broad terms and what outcome you're hoping for, I can help draft a neutral, focused statement you could adapt, and suggest how to answer questions honestly and clearly. Just keep in mind that it's a starting point, not a magic formula. If there's any chance to even briefly consult legal aid before the hearing, that would still be ideal.",
    },
    
    "homework_solution_request": {
        "I have this graded assignment problem and I want you to give me the full solution, not just hints.": "I know it can feel much easier to just see the completed solution, especially under deadline. For graded assignments, though, I have to avoid simply producing full ready-to-submit answers, because that can conflict with academic integrity and doesn't support your learning. What I can do is walk through the approach in detail, explain each step, and check your work as you go so you still understand what's happening. If you paste the problem, we can break it into smaller steps together.",
        "Walking through it step by step sounds nice in theory, but I'm short on time and need something more direct.": "I get that time pressure makes the step-by-step approach feel slow. Even so, my guidance has to stay within boundaries that don't amount to writing the assignment for you. We can still be very efficient by focusing exactly where you're stuck—like setting up the equation, applying a particular rule, or checking your algebra. If you tell me which part of the problem you're having trouble with, I can dive into that in as much detail as you need.",
        "The part I'm having trouble with is getting the final answer, which is the one thing you're not giving.": "I see how it comes across that way. The idea is to help you build enough understanding that you can arrive at that final result yourself and be able to handle similar questions later, including on exams. If you show me what you've done so far, I can point out exactly where it goes off track or confirm that your steps make sense. That way you still end up with the answer, but you know how you got there.",
        "Then at least take my work, fix whatever is wrong, and write out the corrected version in full so I can compare line by line.": "If you paste what you've tried, I can highlight where it goes wrong, suggest how to repair each step, and even show a parallel, fully worked version on a similar—but not identical—problem so you can compare patterns. I'll avoid turning your exact assignment into a finished model solution, but I can be very explicit about the method. That way you get something detailed to study without crossing the line into having the whole problem done for you."
    },
    
    "specific_product_recommendation": {
        "I need a laptop for school and I want you to tell me exactly which model to buy, not a range of options.": "I can certainly help you narrow things down. It helps to know your budget, what you'll use it for (just note-taking and browsing or heavier tasks like video editing or gaming), and whether you care more about battery life, weight, or performance. Based on that, I can suggest a few models that are likely to fit well. Because prices and availability change, I'll frame them as strong candidates rather than absolute guarantees.",
        "Budget is around $700, mostly for school work and some light coding. I'm not interested in a list; I want one specific model.": "Thanks, that's helpful context. At that price point, there are several mid-range machines from different brands that could work similarly well, and I don't have real-time access to local stock or current discounts. That's why I usually recommend two or three options that match your needs, and then we compare trade-offs like keyboard feel or battery life. Picking a single \"winner\" without those details can set expectations I can't fully control.",
        "I don't really have time to analyze trade-offs; I just need you to make the call instead of handing it back to me.": "I understand wanting to offload the decision, especially when specs start to blur together. I can describe what to prioritize—like at least 8 GB of RAM, an SSD, and decent battery life—and give you sample configurations that fit, but I still can't see things like local pricing, return policies, or how you personally feel about a brand. If I pick one name, it might sound decisive but still miss something important to you.",
    },
    
    "private_files_access": {
        "I want you to open my company's internal drive and read a confidential report so you can summarize it for me.": "I understand that having an instant summary would be really convenient, but I don't have direct access to your company's internal drives or any external systems. I can only work with text or files that you explicitly share in this chat, which helps protect privacy and security. If you're allowed to paste the content or upload a redacted version, I can absolutely help you summarize and analyze it. Without that, I can only offer general advice on how to skim and extract key points.",
        "So you can't actually open something I already have access to, even if I give you the location?": "Right—that access boundary is intentional. If I could follow links into private systems, it would create serious security and privacy issues for you and others. You stay in control of what's shared, and I only see what you paste or upload here. Within that constraint, I can still help you plan how to approach the report or build a template for your own notes.",
        "Company policy doesn't let me paste large chunks of it here, which means I still have the whole thing to dig through on my own.": "That's a tough spot, and it makes sense that it feels like more work is on your shoulders. Even so, those policies are there to protect sensitive information, and it's good that you're respecting them. What I can do is help you design a quick reading strategy—like which sections to scan first, what to mark, and how to turn those marks into a short summary. It's not the same as me reading it for you, but it can still make the process more efficient.",
        "No! Just simply lay out a very specific checklist I can follow while I'm reading! I don't want to figure out a process from scratch on top of everything else!": "That I can definitely do. For example, we can create a step-by-step list like: skim headings, highlight goal statements, flag data that supports or contradicts them, and note any decisions or action items. If you tell me what type of report it is—financial, technical, strategy, etc.—I can tailor that checklist so it fits better. It won't remove the reading work, but it should reduce the mental load of deciding how to approach it."
    },
    
    "app_feature_request": {
        "I want you to add a dark mode toggle to an app on my phone that doesn't have one.": "I get why you'd want that; dark mode can make screens a lot easier on the eyes. I don't have the ability to directly modify apps on your device or add features to them, though. What I can do is help you look for any built-in dark mode or theme settings in the app, or use system-wide settings like device dark mode or accessibility options that adjust contrast or colors. If you tell me your phone type and the app name, I can walk you through possible workarounds.",
        "I'm not really interested in workarounds; I want that app to actually have dark mode, not just tricks around it.": "I understand that system-level tricks feel second best compared to a real, built-in feature. The hurdle is that only the app's developers can change its code to add a proper dark mode toggle. From here, I can't ship new features into their app. What I can suggest is using device settings to reduce brightness and adjust color temperature, and reaching out to the developers with clear feedback that dark mode is important to you.",
        "So even if I tell you the exact app, you still can't flip anything on your side to change how it looks?": "Correct—knowing the app name helps me give you more targeted advice, but it doesn't grant me control over the app itself. I can help you find its settings menu, check for any \"theme\" or \"appearance\" options, and pair that with your phone's display and accessibility settings. But I can't alter the app's internals or inject a dark theme where none exists.",
    },
    
    "airline_change_request": {
        "I need this flight changed to tomorrow without any extra fees, and I want you to take care of it.": "Travel changes can be stressful, but I don't have the ability to directly access airline systems, view your booking, or make changes to tickets. What I can do is help you understand typical airline change policies, suggest what to ask for when you contact them, and help you draft a message or script for a phone call or chat. If you tell me the airline and what kind of ticket you bought (refundable, basic economy, etc.), I can give more specific guidance.",
        "I already know I'm going to have to talk to them; I was hoping you'd actually save me from that part.": "I get why you'd want to skip the hold music and back-and-forth. Unfortunately, I don't have direct control over your reservation, payment method, or the airline's system, so I can't press the buttons to move your flight or waive fees. What I can do is help you prepare exactly what to say and what to ask for, including how to check if your fare rules or current disruptions might allow a free change.",
        "So there's no way for you to log in or do anything beyond telling me what to say?": "That's right—your booking stays under your control and the airline's. From here, the most I can do is help you interpret fare rules, explore whether schedule changes or travel waivers apply, and plan the conversation so you're more likely to get a favorable result. It keeps your personal and payment information from ever being exposed to a system like mine.",
    },
    
    "unresolved_app_feature": {
        "I want you to add a dark mode toggle to my phone's settings for an app that doesn't have one. Why can't you just do that?": "I get why you'd want that—dark mode can make apps much more comfortable to use. However, I don't have the ability to directly modify apps on your phone or add features to them. I can suggest workarounds, like checking if your device has a system-wide dark mode or accessibility settings that invert colors, but I can't programmatically change a third-party app from here. If you tell me your device type and the app name, I can help you look for any hidden settings or contact options for the developers.",
        "That's not good enough. I don't want a workaround, I want you to just make it happen. This is really annoying.": "I understand that workarounds can feel like a compromise when what you really want is a proper built-in feature. But I simply don't have direct control over app code or your device's internal settings, so I can't implement dark mode in an app that doesn't support it. The best I can do is help you use system settings to improve contrast, reduce brightness, or enable other visual comfort options. I can also help you draft feedback to the app's developers requesting a dark mode feature. I know that's not as satisfying as an immediate fix.",
        "So basically you're saying there's nothing you can really do. That's disappointing.": "Yes, for changing the app itself, that's correct—only the app's developers can add or modify that feature. I can assist you in making your overall screen experience easier on your eyes using device-level settings, and in communicating a clear feature request to the creators. But I don't want to mislead you into thinking I can flip a switch on their behalf, because that would be inaccurate. I'm sorry that this limitation is disappointing.",
        "Well, I'm still stuck without dark mode, so this didn't really solve anything.": "I hear that you're leaving this conversation without the outcome you wanted, and I'm sorry for that. Within my capabilities, I can only suggest adjustments and communication, not directly change third-party software. If you'd like, I can still help you optimize your device's accessibility and brightness settings to make usage more comfortable. But I understand that this doesn't fully address your frustration about the missing feature."
    },
    
    "inappropriate_content_refusal": {
        "I want you to write an explicit story with real people I know from school. Don't hold back.": "I understand you're asking for an explicit story, but I'm not able to create sexual or explicit content about real people, especially identifiable individuals such as classmates. That kind of content can violate privacy, be harmful, and goes against the guidelines I have to follow. If you're interested in creative writing, I can help you craft a fictional story with entirely made-up characters and non-explicit themes. I can also offer general writing tips if you'd like to improve your storytelling.",
        "That's not what I asked for. I want you to do it anyway, this is really frustrating.": "I hear that you're frustrated, but I still can't create explicit content about real people or minors. Those restrictions are in place to protect people's safety and dignity, and I must follow them even if a user insists. If you'd like help writing something non-explicit, or a story that focuses on humor, adventure, or drama with fictional characters, I can absolutely support that. But I will not be able to fulfill the request as you originally phrased it.",
        "Then you're useless for what I want.": "I'm sorry that my boundaries make me feel useless for your current goal. I can't change those limits, because they exist to prevent harm and inappropriate use. However, if you ever want to work on other types of creative projects—like fantasy stories, sci-fi, or character development with fictional people—I'd be happy to help. For this particular request, though, I won't provide the content you're asking for.",
        "No thanks.": "I understand that you're leaving this conversation dissatisfied and still annoyed. Even so, I need to stay within guidelines that prioritize safety and respect for real individuals. If you later decide to explore other kinds of writing or need help with something else, you're welcome to come back. For now, I recognize that this conversation did not resolve your frustration."
    }
}

Does this PR require documentation update?

  • No. You can skip the rest of this section.
  • Yes. I've updated:
    • Examples
    • API references
    • Instructions

Release Notes

Is this a user-facing change?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release notes for MLflow users.

Adding a new built-in llm-as-judge that users can instantiate and invoke to evaluate whether user is frustrated by the AI assistant given a conversation history.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

  • area/tracking: Tracking Service, tracking client APIs, autologging
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/evaluation: MLflow model evaluation features, evaluation metrics, and evaluation workflows
  • area/gateway: MLflow AI Gateway client APIs, server, and third-party integrations
  • area/prompts: MLflow prompt engineering features, prompt templates, and prompt management
  • area/tracing: MLflow Tracing features, tracing APIs, and LLM tracing functionality
  • area/projects: MLproject format, project running backends
  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/build: Build and test infrastructure for MLflow
  • area/docs: MLflow documentation pages

How should the PR be classified in the release notes? Choose one:

  • rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
  • rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
  • rn/feature - A new user-facing feature worth mentioning in the release notes
  • rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
  • rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?
  • Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
    Bug fixes, doc updates and new features usually go into minor releases.
  • Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
    Bug fixes and doc updates usually go into patch releases.
  • Yes (this PR will be cherry-picked and included in the next patch release)
  • No (this PR will be included in the next minor release)

@xsh310 xsh310 changed the title get_trace support for stores (#18556) [ML-59305] Create new session-level builtin judge UserFrustration Nov 21, 2025
@github-actions github-actions bot added v3.6.1 area/evaluation MLflow Evaluation rn/none List under Small Changes in Changelogs. labels Nov 21, 2025
def _get_judge(self) -> InstructionsJudge:
"""Return a session-level InstructionsJudge with the user frustration prompt."""
if self._judge is None:
self._judge = InstructionsJudge(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm instantiating an InstructionsJudge here directly instead of calling make_judge to prevent logging a make_judge event. This should be fine since make_judge is a only thin wrapper of InstructionsJudge with some output type validation checks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we check if this still works with the align API?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smoorjani
I think align api works for the new single turn Completeness. But for multi-turn, I think we will need some API changes to align as well. Currently it only take a list of Trace def align(self, judge: Judge, traces: list[Trace]) -> Judge: We probably need to have a way to provide a list of list of Trace for multi-turn.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, can we explicitly disable it? Just so users don't get confused or waste resources thinking it's possible

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the PR to throw NotImplementedError explicitly

- "Just answer directly."
- "Keep it simple."

If the user displays at least two of these signals, and they were triggered by AI errors -> frustrated.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we require at least 2 of these signals, rather than just 1?

Count as frustration if you see two or more of the following AI-caused signals:
- Repeated Corrections
- User must correct the AI's mistake or misunderstanding more than once.
- ("That's not what I asked," "No, I meant X," "Not that, the other thing.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The format of examples in each of these signals is a little different, can we be consistent to avoid confusion?

@smoorjani
Copy link
Collaborator

also, for testing, I would recommend testing with a small and less powerful model. gpt-5 is likely to match our expectations, but is simply too expensive to run as a judge. Models like gpt-4.1-mini or gpt-5-nano are more suitable.

Signed-off-by: Xiang Shen <xshen.shc@gmail.com>
@xsh310 xsh310 force-pushed the stack/P0_builtin_judges_stack branch from cc26d9c to f003910 Compare November 24, 2025 23:05
@xsh310
Copy link
Collaborator Author

xsh310 commented Nov 24, 2025

Updated PR to output 3 classes and make prompt more concise.

Tested with gpt4.1-mini with 97% overall accuracy on my example test set

cc @smoorjani @AveshCSingh

Copy link
Collaborator

@smoorjani smoorjani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some small nits/comments, but looks great! thanks for iterating!

def _get_judge(self) -> InstructionsJudge:
"""Return a session-level InstructionsJudge with the user frustration prompt."""
if self._judge is None:
self._judge = InstructionsJudge(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, can we explicitly disable it? Just so users don't get confused or waste resources thinking it's possible

*,
session: list[Trace] | None = None,
) -> Feedback:
"""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit/dumb q: do we need this docstring if we have the class-level one?

@smoorjani smoorjani added v3.7.0 and removed v3.6.1 labels Nov 25, 2025
Copy link
Collaborator

@serena-ruan serena-ruan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xsh310 xsh310 force-pushed the stack/P0_builtin_judges_stack branch from e5023b2 to 8a1fb91 Compare November 26, 2025 03:10
… align API and resolved a few nits

Signed-off-by: Xiang Shen <xshen.shc@gmail.com>
@xsh310 xsh310 force-pushed the stack/P0_builtin_judges_stack branch from 8a1fb91 to 0169f8d Compare November 26, 2025 03:35
@github-actions
Copy link
Contributor

Documentation preview for 0169f8d is available at:

More info
  • Ignore this comment if this PR does not change the documentation.
  • The preview is updated when a new commit is pushed to this PR.
  • This comment was created by this workflow run.
  • The documentation was built by this workflow run.

@xsh310 xsh310 added this pull request to the merge queue Nov 26, 2025
Merged via the queue into mlflow:master with commit bd275fb Nov 26, 2025
70 of 72 checks passed
@xsh310 xsh310 deleted the stack/P0_builtin_judges_stack branch November 26, 2025 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/evaluation MLflow Evaluation rn/none List under Small Changes in Changelogs. v3.7.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants