amazing one thanks. can i somehow subscribe this and get notified when merged?

the ratio thingy should be in prompt itself not as command arg, restarting ui to change the ratio will be unconvenient
I used your fork and tested it, had 2.4 sec per 512 img now i have 1.77sec, is that correct ?

When I tried it (with the extension, not this PR) I couldn't use LORAs with it enabled. Does this PR fix that bug?

the ratio thingy should be in prompt itself not as command arg, restarting ui to change the ratio will be unconvenient

I will see what everyone thinks. There are also other options from tomesd that could be included. I think it probably fits best as a separate set of options within the settings tab. Changing options for tomesd isn't an instant change like with say the batch size slider.

I used your fork and tested it, had 2.4 sec per 512 img now i have 1.77sec, is that correct ?

It's in the ballpark. According to table 4 in the paper that I linked, the author achieves a ~46% speedup using a ratio of 0.5. The effect is dependent upon the amount of tokens used, and it is noted that it is less effective on smaller resolutions in section 5. Not knowing your exact generation settings, I think that your speedup probably does fall within expectations.

When I tried it (with the extension, not this PR) I couldn't use LORAs with it enabled. Does this PR fix that bug?

Is this the extension you're referring to? I hadn't seen this extension before making this PR but, at least for the loras I have tried, they seem to load like normal.

with an AMD RX 6900 XT, torch: 2.1.0.dev20230317+rocm5.4.2, using --opt-sdp-attention, generating 768x768 image, euler a, 20 steps

I went from 3.34 it/s to 5.44it/s

Loras seem to work fine, vram usage seems halved so I can do 1024x1024 without OOM issues now.

tomesd is now available on Pypi https://pypi.org/project/tomesd/

Would it be better to use pip install rather than cloning the repo?

tomesd is now available on Pypi https://pypi.org/project/tomesd/

Would it be better to use pip install rather than cloning the repo?

Yes, I will change that.

the ratio thingy should be in prompt itself not as command arg, restarting ui to change the ratio will be unconvenient I used your fork and tested it, had 2.4 sec per 512 img now i have 1.77sec, is that correct ?

It probably can't be part of the prompt because the logic is applied during model loading.

However, personally I think it should be a setting rather than a command arg so that like @2blackbar said, we can change it without restarting. Thoughts?

the ratio thingy should be in prompt itself not as command arg, restarting ui to change the ratio will be unconvenient I used your fork and tested it, had 2.4 sec per 512 img now i have 1.77sec, is that correct ?

It probably can't be part of the prompt because the logic is applied during model loading.

However, personally I think it should be a setting rather than a command arg so that like @2blackbar said, we can change it without restarting. Thoughts?

I think I will probably add it to the settings area and have the ratio setting there just override the command line argument for it default ratio of 0.5.

I think I will probably add it to the settings area and have the ratio setting there just override the command line argument for it.

i'd actually suggest the opposite - cmdline flags typically override settings. so use settings as default and override from cmdline (although i don't think cmdline is necessary at all if we have settings).

I think I will probably add it to the settings area and have the ratio setting there just override the command line argument for it.

i'd actually suggest the opposite - cmdline flags typically override settings. so use settings as default and override from cmdline (although i don't think cmdline is necessary at all if we have settings).

Sorry, I worded that poorly. I meant that I will override the default ratio setting of 0.5.

I commented in the discussion thread, but reposting it here for visibility:

Hi, original author here.

Some quick suggestions:

• With xformers, tome for small images doesn't really increase speed much (see the benchmark some else did here: Token Merging (ToMe) for Stable Diffusion huggingface/diffusers#2940). So, I'd suggest having an option to only apply ToMe during the highres fix.
• Most options other than the ratio don't really need to be shown to the user, since there's not much point in changing them (both in terms of quality and speed). Idk what the best solution for this is though, since some people might still want to tinker with them.

Note that the tome patch can be applied right before generation just fine, it doesn't need to be on model load. Applying the patch is free (it just sets some class variables) and it can be applied any number of times to the same model without adverse effects (e.g., to change the parameters). The way I implemented it for testing was in the txt2img processing function, right before sampling.

I think the best way to implement it would be to add one of those boxes under the generation parameters, like the controlnet plugin.

I think the best way to implement it would be to add one of those boxes under the generation parameters, like the controlnet plugin.

I haven't used them before, but another option might be to use a code component? I think if we put an 'advanced tweaking' toggle in the token merging settings section for it, that might be a good compromise. It would clean things so that only the ratio slider would need to be primarily displayed. Then, the box could be toggled visible for those that want it to be. For those that do want to tinker with tomesd, it offers them the full palette of options and should transparently allow them to tinker with any options you add upstream in the future.

I haven't used them before, but another option might be to use a code component?

Hmm that could work. Another option is if I add a "settings string" option to the apply_patch function that takes the settings in a compressed string format rather than python arguments. This is what I did during development, but I figured python arguments would be more interpretable, so I didn't include it in the release but I could add it back.

For instance, the default settings would be: 50-rand2x2-a-1 for 50% ratio, 2x2 with random perturbations, only attention, and max_downsample=1. Turning every possible option to the max would be 75-rand2x2-acm-8. I could also expand the string (in a backward-compatible way) for future options.

what's the reason for installer from launch.py instead of requirements_versions.txt? webui parses that file on each startup. anyhow. hard-coding installer in launch.py is typically reserved for core repos.
not saying it cannot be done, but it should be a last resort.

also, why note "compatible with xformers"? note something that is not working as expected, not general positives.
and while on that topic, is it compatible with sdp? it is picking up on usage quickly since its much simpler to use as its built-in into torch 2.0.

what's the reason for installer from launch.py instead of requirements_versions.txt? webui parses that file on each startup. anyhow. hard-coding installer in launch.py is typically reserved for core repos. not saying it cannot be done, but it should be a last resort.

Will change.

also, why note "compatible with xformers"? note something that is not working as expected, not general positives.

I think for such general speedups users expect that they won't stack. Swapping from args to settings though, it could make more sense.

and while on that topic, is it compatible with sdp? it is picking up on usage quickly since its much simpler to use as its built-in into torch 2.0.

I think @dbolya mentioned it being compatible in the paper.

is it compatible with sdp?

Yeah, just tested by upgrading to torch 2.0 for the webui and using the option --opt-sdp-attention. Works just as well as with xformers.
Should be compatible with anything that just speeds up the forward functions of attention / cross attn / the mlp. ToMe only edits the block itself.

Suggestion: add options to Settings tab so that user can tune the ratio on the fly

Ok i had to disable it, for now it craps on controlnet , you cant generate it , errors out and some other extensions, i suspect when its gonna be merged, lotsa people will complain

I think the best way to implement it would be to add one of those boxes under the generation parameters, like the controlnet plugin.

I don't know this for certain, but I'm going to go forward right now with the assumption that AUTO would not want the main enable/disable toggle in the txt2img/img2img panes themselves so I will put it in the settings section.

Ok i had to disable it, for now it craps on controlnet , you cant generate it , errors out and some other extensions, i suspect when its gonna be merged, lotsa people will complain

I knew about that issue when making the PR. A lot of users still don't use controlnet. So, I think just adding a note about it being incompatible next to the toggle will be good enough for now.

I think maybe we should mark this PR as Draft until we get more opinions on the various points above.

@asagi4 Should be working now. (you'll have to swap to the dev branch to test it though)

I installed the branch and am testing it on my M1 8 GB RAM MacBook. Made a few simple images, but don't see much difference in performance. Running ./webui.sh --token-merging, and have these settings:

I turned on most checkboxes without knowing what they're doing just to see if I can see a difference. How do I know tomesd is applied and that it works?

I turned on most checkboxes without knowing what they're doing just to see if I can see a difference. How do I know tomesd is applied and that it works?

You should see increased performance, but particularly with larger high-res pass scale. There should also be info added to the image indicating it was used as well.

You can turn off "apply only to hr pass" and that will get you more performance by applying to the first pass also. However, the speed-up would be negligible.

Just wondering, now that auto merged dev into master, how I would be able to use this PR on master instead of dev?

@papuSpartan Thanks!

@Panchvzluck Looking at https://github.com/papuSpartan/stable-diffusion-webui/tree/tomesd, it seems the branch is being updated with updates from upstream, last time yesterday.

My generic task of making a 512x768 image and upscaling it to 2x goes from 18.5 seconds to 14-15.

I played with settings and it seems only the main slider really has effect on performance - Merge attention should be checked and two other merges don't really help. I couldn't figure out the benefit of Maximum down sampling and strides.

All those settings can be useful for exploring the method, and they would be appropriate for an extension, but for the main repo I would like to have a simple configuration.

There can be a single slider for token merging from 0 to 0.9, set to 0 by default. And another slider for hires fix, from 0 to 0.9, also at 0 by default. 0 disables token merging. No comandline settings.

From the commits it looks like you reworked it the way I asked. If you don't plan to add more and don't have any complaints about reduction in number of options do write back and I'll merge it into dev.

@AUTOMATIC1111 Go ahead and merge this much into dev.

There were 3 commits made at dbolya/tomesd after the 0.1.2 release, to improve support for mps and "fix rand generator". Should a request be made for a new tomesd release at this time, to support this merge?

@jrittvo The version of the package is specified loosely as tomesd>=0.1.2 in the requirements. That means if a new release is published, it fits the version range and gets installed. So as far as I understand this, if @dbolya cuts a new version, that's definitely nice, but not doing so or doing so doesn't block anything here in the automatic webui.

@jrittvo The version of the package is specified loosely as tomesd>=0.1.2 in the requirements. That means if a new release is published, it fits the version range and gets installed. So as far as I understand this, if @dbolya cuts a new version, that's definitely nice, but not doing so or doing so doesn't block anything here in the automatic webui.

Auto already changed it to an explicit == 0.1.2. It will need to be changed to 0.1.3 as @jrittvo suggested because of some issues MPS users were having.

Hello! To use token merging, do I have to install any package or extension? The reason I ask is, that I've tested the token merge ratio slider (quicksetting) and it did not effect the outcome whatever I chose there.

Hello! To use token merging, do I have to install any package or extension? The reason I ask is, that I've tested the token merge ratio slider (quicksetting) and it did not effect the outcome whatever I chose there.

The settings are within the settings of the newest update itself. Check under Optimizations. Only thing i myself can't find is the one that you can see the images being generated when you're doing multiple ones :/