This is a quick reference article where I test the Inno-maker IMX462 sensor on a Raspberry Pi 3. The scene is mostly dark, imagine a room with closed door and all windows covered up. The RPI3 is accompanied with 3 IR LEDs just to have at least some light once we start experimenting.

Requirements:

• Raspberry Pi 3
• 3 x IR LED
• Inno-maker IMX462

$ uname -a
Linux pycam3 6.6.31+rpt-rpi-v7 #1 SMP Raspbian 1:6.6.31-1+rpt1 (2024-05-29) armv7l GNU/Linux
$ libcamera-still --version
rpicam-apps build: 49344f2a8d18 17-06-2024 (12:19:10)
libcamera build: v0.3.2+27-7330f29b

It’s important that we disable Automatic Exposure/Gain Control (AEC/AGC) and Auto White Balance (AWB) algorithms. We can do that with libcamera by using the exposure time (--shutter), the gain (--gain) and the white balance gains (--awbgains) settings. We need this for reproducability, but also for speed as some of these algorithm requires taking extra shots. Typically our command look as following:

$ libcamera-still -o "/home/pi/image.jpg" --shutter 600000 --gain 1 --awbgains 1,1 --immediate --raw -n

Shutter

With the shutter speed setting we control how long the image sensor gets to collect light. It’s often referenced as the exposure time. The longer the shutter speed, the more light is going to fall into the sensor, the more details we will get in our dark scene. Libcamera sets the shutter time in microseconds.

shutter 10ms:

shutter 100ms:

shutter 500ms:

shutter 1s:

shutter 3s:

shutter 5s:

shutter 10s:

shutter 20s:

shutter 1min:

In dark conditions, a 1s shutter reveals some initial details. However, it is still too little to recognize anything. At 3s shutterspeed more details become visible and we can finally recognize objects. Bumping the shutter even higher means bringing even more details into the picture. Additionally we don’t notice a lot of noise in the picture. The only thing we do notice is that the picture becomes a bit white /pale.

Gain

The gain settings controls the combined analog and digital gain. But what is the difference the two? The analog gain comes into play inside the image sensor, where light is converted into an electrical signal (voltage), and then further on using an Analog-to-Digital Converter (ADC) into digital 1’s and 0’s. The analog gain amplifies the voltage signal before it goes into the ADC. In the resulting picture the amplification (referred to as ‘gain’) makes low light scenes appear brighter than without the extra gain.

There is however also a downside to this gain. The photo-detector is sensitive to dark noise, however from perspective of the amplifier this noise is indistinguishable from normal light that was collected in the photo-detector. Therefore the amplifier will also amplify the noise, and as such reduce the dynamic range. Normally the noise of the ADC will dominate over the noise introduced by the gain amplifier. However, as the gain is increased it will take the overhand at some point.

Digital gain is applied after the ADC stage, when the final image has been composed. The multiplication is performed on the digital values and as a result reduces the resolution. This process can be performed by some extra part in the image sensor, or an ISP, but it can also be achieved by post processing. Therefore it’s better not to apply any digital gain in your capturing pipeline as it actually discards some of the information that was captured in the analog stage. Without the digital gain you’re left with the option to apply the multiplication during your post processing stage.

The choice of analog vs digital gain is however not entirely ours to make. Using libcamera the --gain setting controls both. It’s up to the driver to actually decide what gain it will apply. But given the downside of using digital gain it will always prefer using analog gain over digital gain. Looking further in detail we actually see that image sensors have those analog and digital gain amplifiers embedded in hardware. They’re bound to a minimum and maximum value of amplification, which can than be controlled via the CCI (I2C) bus.

When we read the datasheet of the IMX462 we find that gain can be controlled within following rates:

• 0 dB to 29.4 dB: Analog Gain 29.4 dB (step pitch 0.3 dB)
• 29.7 dB to 71.4 dB: Analog Gain 29.4 dB + Digital Gain 0.3 to 42 dB (step pitch 0.3 dB)

In our tests we will avoid using digital gain. Lucky for use the linux driver for the IMX462 already ensures to have only control over the range of analog gain. Looking at the driver we notice that the range goes from 0 to 100, which maps to the ~30dB max and 0.3dB steps (30db/0.3dB = 100).

For our gain tests we fix the exposure time to 100ms.

gain 1:

gain 20:

gain 40:

gain 60:

gain 80:

gain 100: (may be resticted to 98 for IMX462 in the future whenever this gets merged into the kernel)

It takes us up to a gain of 20 before we see any objects appearing in the background. And as we bump up the gain, more and more details will become visible. In some extend it’s similar to what we saw happening when we experimented with the exposure time. We could say that under the same conditions, using a 5s shutter with gain 1, roughly results in the same picture as when we use a 100ms shutter with gain 70.

The mayor difference though is that bumping up the gain also introduces a lot of noise in our pictures. At those higher gain values we can easily spot many horizontal bands and the picture quality is a lot worse than using the longer exposure shots. So if the shutter speed is allowed to go high than it will result in better picture quality in conditions where not a lot of light is available. In case you can’t allow the shutter to go high there is still the option to increase the gain but know that you will have to deliver in on image quality as noise gets amplified to. But in the end gain is also a way of bringing low light signals (like faint stars) into the picture. Keep in mind that from the results we’re mostly talking about the RAW data quality. No de-noising algorithms have been performed, though it could (and would) help to compensate some of the image quality loss of using the higher gain.

LCG vs HCG

The exposure and gain settings are 2 very common settings that you can find in most camera software, including libcamera, and as you can see it gives us quite accurate control over the camera sensor. There is however more to discover. The IMX462 has an extra trick up its sleeve: dual conversion gain. The IMX462 can choice between 2 conversion modes: Low Conversion Gain (LCG) and High Conversion Gain (HCG).

Do not confuse HCG/LCG with the normal gain setting that we saw previously. Those are 2 different things! The gain setting is about amplification, HCG/LCG is about photodiode to voltage conversion. So let’s say in LCG mode a bung of electrons convert to 0.01V, the same amount of electrons may convert in HCG mode to 0.05V. So with the same mount of light, a higher voltage is generated, hence why it’s called “high conversion gain”. In the end it will help in low light conditions.

• Low conversion gain (LCG)
  • the normal mode
  • white is at 90% of pixel saturation
  • good for bright parts in the image
• High conversion gain (HCG)
  • increases sensitivity and reduces readout noise level
  • has advantage in signal-to-noise (SNR) at low illuminance levels
  • good for dark parts in the image

So each gain mode has its own advantages, and they can even be combined by an ISP to achieve a higher dynamic range. There is very interesting topic at cloudynights about HCG. In the consumer market the IMX462 is used for example in the ZWO ASI462 camera. The reason I mention this is that they also advertise the HCG mode. In astro-photography this can play an important role. While HCG is implemented in the IMX462 in a different register than the normal gain setting, ZWO controls it automatically for you once the normal gain is increased to level 80. ZWO has their own gain levels compared to those of libcamera, so here 80 * 0.1dB = 8dB, where for libcamera 8dB gives a gain level of about 8dB * 0.3dB = 0.24dB. Always look at dB when comparing across vendors. Looking back at our previous gain experiments it would mean that if we also implemented auto LCG/HCG switching at the same levels, the switchover to HCG would already happen before noise is becoming dominant. It would also mean that on that moment we would see a big bump in brightness.

For the raspberry pi and libcamera things are currently a bit more complicated. As of November 2024 there is no out-of-the-box support for toggling HCG mode in video4linux, nor in libcamera. However, that doesn’t mean it’s impossible. HCG has already been discussed in a few topics on the raspberry pi forums and meanwhile a pull request (PR) has been opened for quite a bit of time that should allow control of HCG via a kernel module parameter. It means it doesn’t involve video4linux nor libcamera at all, but still if you’d ever need it you can enable it via the sysfs entries for the kernel module. A side effect of having the github PR is that the build server creates a build artifact that can directly be installed on your system. The PR is targetting linux 6.6 which is also the kernel that I’m currently on, so everything should go fairly straightforward. Note: you may not be able to install the build artifact by the time you read this article as the build server only retains the artifacts for few weeks/months.

Before you proceed in patching your kernel there is still one thing we need to take care off: patching libcamera itself. As you may have noticed from the kernel patches is that IMX462, due to small differences with IMX290, is from now on a individual camera device in the linux kernel. You can target the IMX462 specifically in your device tree while in the past you had to set it up as a IMX290/IMX327. So for the best user experience we should make sure to have the device tree overlay for IMX462 activated in the config.txt:

# IMX462
dtoverlay=imx462,clock-frequency=74250000

Now, about the licamera patches themselves I also need to shed some lights on what has been done. The patches are mandatory to make libcamera work with the “new” IMX462 camera driver. Libcamera wasn’t yet aware of this camera device since it never existed in earlier kernels. Libcamera would therefore exit with and error when you tried to take a snapshot. So I patched libcamera to support the new IMX462 cam and I created this PR on raspberry pi fork of libcamera so that the support will make it to the next Raspbian OS release. However it was concluded that the patches should better be upstreamed to the origin libcamera, and so that’s what I did. You can find them here:

• https://git.libcamera.org/libcamera/libcamera.git/commit/?id=8655ba6ffb5cc5d28d9d6cc293bf23812166a7ef
• https://git.libcamera.org/libcamera/libcamera.git/commit/?id=a49052d976921574c159380b690e91f2b163d852
• https://git.libcamera.org/libcamera/libcamera.git/commit/?id=99a871439ba7243bfd46b78b9ca0042ce8e37349
• https://git.libcamera.org/libcamera/libcamera.git/commit/?id=b45ff9ff3c00dd626db5d6e0fc06bcc9191e2854
• https://git.libcamera.org/libcamera/libcamera.git/commit/?id=357dbc9f0e6d0e26f27553c2955cf580bc030394

The patches are merged upstream as we speak, so Raspbian will get the support for IMX462 out of the box anywhere soon, but due to merging strategies and the kernel dependency it’s rather hard to tell when exactly that will happen. Long story short, unless your OS already has the HCG kernel mode parameter in the sysfs (check if you have /sys/module/imx290/parameters/hcg_mode file) you’re on your own for patching your kernel and libcamera software.

If the rpi build artifacts are still available, at least you can already use the kernel as is. To install the patched kernel:

$ sudo rpi-update pulls/5859

This will take a few minutes to install. In my case the PR artifacts slightly upgrades to linux 6.6.57. If needed you can always switch back to a normal RPI kernel by updating to the latest version:

$ sudo rpi-update

Afterwards reboot the machine.

$ uname -a
Linux pycam3 6.6.57-v7+ #1 SMP Sat Oct 19 12:29:20 UTC 2024 armv7l GNU/Linux

The new kernel module entry can be found in the sysfs:

$ cat /sys/module/imx290/parameters/hcg_mode 
N

By default it’s off, but you can enable/disable it by writing 0 or 1 to this file:

$ echo 1 | sudo tee /sys/module/imx290/parameters/hcg_mode

Verify:

$ cat /sys/module/imx290/parameters/hcg_mode 
Y

HCG off:

HCG on:

And here is another one with HCG on, but analog gain reduced to 20:

NOTE: the pics taken for the HCG experiments are performed with a slightly modified camera board. Do not directly compare them to those I took earlier. More details about the mods are upcoming, but essentially what I did is improving the quality of the power supply to the camera which on its turn reduces removes the horizontal banding that we can clearly see at high gain levels.

OK, now about the HCG mode, it’s pretty much clear that it makes the camera again more sensitive to light. At looks as if another level of analog gain is added, and actually it is said that HCG mode sort of brings an additional 5.8x gain. It also make noise stand out a bit more, so it’s not just something that magically fixes things for us. But if you look at it from another angle it just one more option in your toolbox as it allows us to see things in the dark as if we were using long exposure times, while actually the exposure time is set to only 100ms. Also compare the picture with HCG=on,gain=20 to the one with HCG=off,gain=50. Both pictures are pretty much the same in brightness, even though the gain levels are considerably different. Let’s zoom in a bit:

I’m not entirely convinced here but there seems to be a very small, subtle difference between both in that the one with HCG seems to be a tiny bit less noisy. Maybe it’s just the overall brightness that is a tiny bit off, or just some variation that we’re seeing. Anyway, I think it certainly deserves further exploring once I get back to trying astrophotography.

Conclusive thoughts

To conclude, we can state the IMX462 can be used in dark scenes. As a photographer, you have a few tools in your belt to get to the best possible result. There is a considerable range of exposure settings. Analog gain is available up to about 30dB. Finally, the High Conversion Gain can be enabled or disabled using the patches described in this article. I hope you found something interesting. At least for me, it was worth diving into this HCG thingy. It was also valuable to get some sort of reference picture quality on which I can compare my camera modifications. Regarding the latter, stay tuned for another article. It will go more into details on what you should do to get rid of the horizontal banding issues with the Inno-maker IMX462. See you soon.