Would very much appreciate this PR landing!

My app had a model swapper for quite a while now, through old api (by sending through /api/predict), then I recently updated to the method through /sdapi/v1/options.
What would be the use case of sending via override_settings? How does it differ from the other two methods? Just wondering 🤔

@Kilvoctu The use case would be to let a single POST request do the settings and can revert back to the original one after its done. You can see #4301 (comment) for the use case
I made this also because I run discord bot with this and found it more convenient as I don't have to take care of model switching back and forth

For me, I have multiple clients talking to the same SD API, each wanting to use their own model; they can happily wait, but at present they end up stepping over each other because one client sets the model and the second client issues a request, and neither party receives what they expect.

The SD API internally queues its requests, so it can switch model only when required for the specific request it's servicing.

Hm yea, that is making sense. Now that I think about it, I even talk about the use of override_settings vs /options in my API documentation actually 😅. Since my Discord bot has its own queuing system, in my particular case the previous method is doable. Of course more options is better, and may as well use webui's internal queue if possible.

So if I understand override_settings correctly, let's say the webui has checkpoint a.
Then user sends payload with override_settings for checkpoint b.
webui will swap to checkpoint b for the request, then back to checkpoint a afterward.

So if I understand override_settings correctly, let's say the webui has checkpoint a.
Then user sends payload with override_settings for checkpoint b.
webui will swap to checkpoint b for the request, then back to checkpoint a afterward.

Yes

The only thing I'm not sure about is the swapping back: that seems like it could result in a lot of unnecessary swapping if you have multiple requests all using the same model. Maybe make that an option as well?

The only thing I'm not sure about is the swapping back: that seems like it could result in a lot of unnecessary swapping if you have multiple requests all using the same model. Maybe make that an option as well?

I think in that case its better to just use the options API. If you cache them in RAM, the swapping back takes less than a second. And it is the core function of such a 'temporary' setting thing.

The problem with using the options API is the issue with the queuing; a request can be serviced much after the initial request, so that if you have 5 requests with Model A queued up, and then request options = Model B followed by your model B generation request, the Model A requests will end up using Model B as the options API applies immediately.

To solve that, I'd either need:

1. a way to switch options that respects the queue (i.e. "set settings at the end of the queue, not now")
   or
2. have an option to preserve the model switch in the settings override, so that I can either continue to generate with Model B at no cost or pay the cost to switch to Model A (or whatever other model) on demand.

I have quite a few models, so keeping them in RAM isn't super feasible :(

I'm leaning towards 1), just because I think that's generally useful anyway (it doesn't make much sense to have the settings override break things for queued generations)

Any updates if this is close to being merged?

why not just call the callback at this point?

I get it now. The call_queue.queue_lock would already locked and the callback would try to lock it one more time. I don't like the way this is duplicating code from webui.py, but it seems currently there's no easy better option.