Got a little concerned with the failure for test_ImageView, but looking at the CI history, I can see its failed before.

I have tried to restore the behavior to pre-#793, that is to say, levels-only is converted to lut-only. i.e. rescaleData() is traded away for applyLookupTable()

VideoSpeedTest.py --size=3072x3072
both levels and lut bypassed (fastest path)
VideoSpeedTest.py --size=3072x3072 --levels=0,255
both levels and lut bypassed (equivalent to not specifying levels)

all the following cases bypass levels and applies lut only
VideoSpeedTest.py --size=3072x3072 --levels=50,200
VideoSpeedTest.py --size=3072x3072 --lut
VideoSpeedTest.py --size=3072x3072 --levels=50,200 --lut
VideoSpeedTest.py --dtype=uint16 --size=3072x3072
VideoSpeedTest.py --dtype=uint16 --size=3072x3072 --levels=0,65535
VideoSpeedTest.py --dtype=uint16 --size=3072x3072 --levels=100,10000
VideoSpeedTest.py --dtype=uint16 --size=3072x3072 --lut
VideoSpeedTest.py --dtype=uint16 --size=3072x3072 --levels=100,10000 --lut

This would also allow us to drop the fits_int32 specialization in #1648.

Timings on a Linux machine with numpy 1.20.2
Results suggest that this PR commit 2 converting rescale-only to lut-only is a pessimization.
So if we are not seeking to restore behavior to pre-#793, then this PR commit 1 gives the better outcome.

python examples/VideoSpeedTest.py --size=3072x3072 --levels=50,200

master
> Entering functions.makeARGB
  check inputs: 0.0100 ms
  apply levels: 20.3686 ms
  apply lut: 0.0098 ms
  allocate: 0.0012 ms
  reorder channels: 5.0273 ms
  alpha channel: 0.0057 ms
< Exiting functions.makeARGB, total time: 25.4271 ms

this PR commit 1
> Entering functions.makeARGB
  check inputs: 0.0134 ms
  apply levels: 19.1901 ms
  apply lut: 0.0083 ms
  allocate: 0.0014 ms
  reorder channels: 5.2166 ms
  alpha channel: 0.0050 ms
< Exiting functions.makeARGB, total time: 24.4391 ms

this PR commit 2
> Entering functions.makeARGB
  check inputs: 0.0119 ms
  apply levels: 0.0083 ms
  apply lut: 39.0100 ms
  allocate: 0.0067 ms
  reorder channels: 6.9034 ms
  alpha channel: 0.0062 ms
< Exiting functions.makeARGB, total time: 45.9511 ms

python examples/VideoSpeedTest.py --size=3072x3072 --levels=50,200 --lut

master
> Entering functions.makeARGB
  check inputs: 0.0129 ms
  apply levels: 19.0156 ms
  apply lut: 38.1424 ms
  allocate: 0.0057 ms
  reorder channels: 5.4858 ms
  alpha channel: 0.0060 ms
< Exiting functions.makeARGB, total time: 62.6724 ms

this PR commit 1
> Entering functions.makeARGB
  check inputs: 0.0145 ms
  apply levels: 0.0086 ms
  apply lut: 40.4389 ms
  allocate: 0.0062 ms
  reorder channels: 6.0117 ms
  alpha channel: 0.0062 ms
< Exiting functions.makeARGB, total time: 46.4907 ms

this PR commit 2
> Entering functions.makeARGB
  check inputs: 0.0114 ms
  apply levels: 0.0081 ms
  apply lut: 38.5458 ms
  allocate: 0.0062 ms
  reorder channels: 5.5625 ms
  alpha channel: 0.0062 ms
< Exiting functions.makeARGB, total time: 44.1446 ms

So if we are not seeking to restore behavior to pre-#793, then this PR commit 1 gives the better outcome.

I'm going to have to defer to @outofculture and @campagnola for input here. Right now we have our hands full with writing a submission for a conference; so it might be a few days before you get helpful follow up.

My stance regarding #793 behavior is that ideally the issue presented in #792 (where scale=3 ideally remains resolved)

I'm going to have to defer to @outofculture and @campagnola for input here. Right now we have our hands full with writing a submission for a conference; so it might be a few days before you get helpful follow up.

No worries.

My stance regarding #793 behavior is that ideally the issue presented in #792 (where scale=3 ideally remains resolved)

I made the first 2 commits erroneously thinking that issue #792 didn't apply to integer data.
I think I have fixed that in the last commit. During levels + lut combination, the scale value used needs to include the fix done in #793.

It looks a bit kludgy, the part that calls makeARGB() with a specially chosen scale value in order to cause apply levels to be bypassed.

development continued in #1693