I have set these options:

So when I scroll to 50% of the site height, it switches from light to dark mode.

When I enable only dark mode, it still switches from light to dark on scroll:

I have to go back and enable "Light/Dark" and in there, uncheck "Enable Color Transform":

Now I can set the color mode to Light or Dark and it sticks, how it is intended.

So what happens is that an option that isn't even visible is still applied. That is quite confusing for the user. :)