Support decoding ISO HDR (avif/heic/jpeg-xl, etc) #3778

CloudlessMoon · 2024-12-24T11:00:03Z

For iOS 17, HDR for HEIC images is no longer a problem.
For iOS 18, HDR of JPG images is also fine, but the console reports the warn cicp_tagFromColorSpace:54: Failed to get CICP info for space: <CGColorSpace 0x3009d6a60> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB Linearized)

Use iOS 17+ real device debugging, as the simulator does not support HDR, and XCTest does not have a host app, it is not possible to add related tests.

…extImageDecodeToHDR, it will be decoded to HDR only when loading is complete.

… be decoded to HDR when it is not completed

2、add simulator log for HDR

dreampiggy · 2024-12-25T07:41:49Z

Looks OK.

I'll polish some details and push a fork branch, and merge it later this week.

CloudlessMoon · 2024-12-25T10:36:05Z

看起来不错。

我将完善一些细节并推送一个分支分支，并在本周晚些时候合并它。

There are still some encode problems.

For iOS 17, HDR of HEIC images, SDImageHEICCoder.imageUTType returns public.heics, HDR is lost after encode, when it is public.heic there is no problem
For iOS 18, HDR of JPG images, decoded CGImageSourceGetType(source) is actually public.heic, when I use public.jpeg to encode, its HDR is lost, I'm looking at this issue

dreampiggy · 2024-12-26T09:19:38Z

For iOS 17, HDR of HEIC images, SDImageHEICCoder.imageUTType returns public.heics, HDR is lost after encode, when it is public.heic there is no problem

You seems use the HEICS animation image, you need to not use the animation, call SDImageIOCoder.sharedCoder encodedImageWithData:, or SDImageHEICCoder.sharedCoder encodedImageWithData:

dreampiggy · 2024-12-26T09:20:32Z

For iOS 18, HDR of JPG images, decoded CGImageSourceGetType(source) is actually public.heic, when I use public.jpeg to encode, its HDR is lost, I'm looking at this issue

You mean, JPEG-XL. JPEG2000 does not support HDR.
Or you may use HEIF container with JPEG codec, like HEIC means HEIF container with HEVC codec

See: https://jpeg.org/jpegxl/

And, we have custom coder for JPEG-XL: https://github.com/SDWebImage/SDWebImageJPEGXLCoder

…lost

CloudlessMoon · 2024-12-26T11:21:38Z

For iOS 17, HDR of HEIC images, SDImageHEICCoder.imageUTType returns public.heics, HDR is lost after encode, when it is public.heic there is no problem

You seems use the HEICS animation image, you need to not use the animation, call SDImageIOCoder.sharedCoder encodedImageWithData:, or SDImageHEICCoder.sharedCoder encodedImageWithData:

The code I submitted recently has been solved. I used Core Image and referred to the Apple developer documentation, https://developer.apple.com/documentation/uikit/supporting-hdr-images-in-your-app, erasing the CGImge type, it can encoded to JPEG/HEIC, next, I'll check for a memory leak, there should be no problem after that

dreampiggy · 2024-12-27T06:47:32Z

Found some strange code, maybe need your explaination.

Why HDR image should not force decode ? HDR is just 10 bit or 12 bit per-per-component image (unlike normal 8 bit one), it could be re-draw using correct CGBitmapContext, you should modify the SDImageCoderHelper.m's legacy code, not just hide the issue. If you don't know how to write this, I can provide the help.
Why HDR image need to use CoreImage to keep the Gain Map HDR information ? And actually I don't know why we need to support this non-standard Gain Main HDR or not. I think only Apple device contains Gain Map, and it's deprecated. New devices and other Web standard devices like Windows PC/Mac, always use HEIC standard's HDR information

CloudlessMoon · 2024-12-27T13:53:01Z

Found some strange code, maybe need your explaination.

Why HDR image should not force decode ? HDR is just 10 bit or 12 bit per-per-component image (unlike normal 8 bit one), it could be re-draw using correct CGBitmapContext, you should modify the SDImageCoderHelper.m's legacy code, not just hide the issue. If you don't know how to write this, I can provide the help.

Why HDR image need to use CoreImage to keep the Gain Map HDR information ? And actually I don't know why we need to support this non-standard Gain Main HDR or not. I think only Apple device contains Gain Map, and it's deprecated. New devices and other Web standard devices like Windows PC/Mac, always use HEIC standard's HDR information

I also thought so at the beginning and changed SDImageCoderHelper.m. In fact, if CGBitmapContext is used to force decoding of HDR image, CGImageRef is changed to DP, and HDR can be displayed correctly at this time. and then encode to JPEG, CGImageDestinationFinalize throws an exception information relating to the RGB. after many tests, only when CGImageRef to IP, JPEG encode successfully. If the encoding to HEIC, it can DP or IP, so I uniformly remove the force decode, in fact, observing the HDR image can be seen that when the album opens it will not have HDR effect at the moment, it will have a process from dark to light, so I also guess that it is decoding when rendering.
After iOS 18, apple supports JPEG with Gain Map, which is an iSO standard, so far, apple is compatible with other platforms such as Android 15, but there are exceptions, such as vivo, its orange system is its own HDR standard. with iOS 17, apple only supports HDR for HEIC.
As for why Core Image is used, because it has a ready-made API that can convert HDR, my goal is to remove type and be able to convert from HDR for HEIC to HDR for JPEG.
More articles about HDR, can refer to http://ios.zoopda.com/post/94425.

dreampiggy · 2024-12-30T06:36:26Z

so I also guess that it is decoding when rendering.

From your description, it seems Apple do some Hack code, to only apply HDR color when JPEG with GainMap or HEIC with non-GainMap HDR image, was rendering on the screen. Maybe some logic inside CoreAnimation (not just ImageIO/CoreImage).

It's Screen/hardware awareness decode, not a simple normal 10 bit image bitmap stored in the CGImageRef's buffer. This make me worries about the custom decoder (like libheif, libJPEG-XL) how to correctlly handle the HDR image. And this means, you can never get 10 bit bitmap buffer on iPhone 8 like non-HDR display hardware, even you really want to do so.

dreampiggy · 2024-12-30T06:43:05Z

As for why Core Image is used, because it has a ready-made API that can convert HDR, my goal is to remove type and be able to convert from HDR for HEIC to HDR for JPEG.

It's OK to call Core Image's API. I'm just surprised to find that there are no ImageIO's related C API to replace it. Most of time, we do operate on CGImageRef, which is not CIImage object (I know we can convert it, but this make extra code to take abstract layer between non-HDR encoding and HDR encoding)

In WWDC 23, seems they only mention the CoreImage to write 10 bit HDR into JPEG/HEIC (there are 16 bit PNG, but I'm not familiar with) ?

https://developer.apple.com/videos/play/wwdc2023/10181/

CloudlessMoon · 2024-12-30T06:52:52Z

至于为什么使用Core Image，因为它有一个现成的API可以转换HDR，我的目标是去除类型并能够从HDR for HEIC转换为HDR for JPEG。

调用Core Image的API就可以了。我只是惊讶地发现没有ImageIO 相关的C API 可以替代它。大多数时候，我们确实在操作 CGImageRef，这不是 CIImage 目的

在WWDC 23中，似乎他们只提到CoreImage将10位HDR写入JPEG/HEIC（有16位PNG，但我不熟悉）？

https://developer.apple.com/videos/play/wwdc2023/10181/

iOS 18, imageIO has a new API, CGImageCreateWithContentHeadroom, It can create or copy an HDR image. but what is created or copied is a DP CGImageRef, which cannot be encoded into JPEG. CIContent.createCGImage:fromRect:format:colorSpace:deferred:, If deferred is set to false, which cannot be encoded into JPEG

CloudlessMoon · 2024-12-30T07:03:06Z

this is the CGImageRef after HDR decoding

This is the CGImageRef created by CGImageCreateWithContentHeadroom

When I encode to JPEG, throws error:

CGImageDestinationFinalize will call CGContextDrawImage. If it is not a lazy image, will it render HDR in advance? I guess Apple has not made it compatible here.

@dreampiggy

dreampiggy · 2024-12-31T07:08:48Z

I think actually you don't need to considerate all things in single PR.

For example, you can put Support to Encode HDR with Gain-Map into JPEG as a separate issue. Which is much more complicated from your experiment

This PR only focus on HEIC related HDR (HEIF standard)

CloudlessMoon · 2024-12-31T07:29:02Z

我认为实际上你不需要在单个 PR 中考虑所有事情。

例如，您可以将 Support to Encode HDR with Gain-Map into JPEG 作为一个单独的问题。从你的实验来看，哪个要复杂得多

此 PR 仅关注 HEIC 相关 HDR（HEIF 标准）

My idea is that this PR aims to implement HDR, including HEIC and JPEG. If it only implements HEIC, it will not be cross-platform. at present, through some of my tests, it should be fine. I hope it can be merged into the main branch and a version can be released.

…ompatible with SDR displays while preserving HDR

dreampiggy · 2025-01-06T11:02:51Z

Hi @CloudlessMoon

Recently some people who contact me via email and asking about HDR support on SDWebImage and on iOS platform.

Seems you may have more practical experience than me on HDR concept. Can you join the discussion here ? Since they are all Chinese people, we can take in Chinese and post conclusion here in English instead.

你好，最近有其他同学咨询了我关于SDWebImage 在iOS平台支持HDR 的相关问题

HDR领域这部分我认为你应该比我更有实践经验，能否请你参与讨论一下？这部分可以用中文交流，然后结论在这里用英文总结

Join WeChat here:

* 'master' of https://github.com/SDWebImage/SDWebImage: Fix iOS unit test again Update the GitHub action for Demo build Update the test case `test15ThatQueryCacheTypeWork` to ensure the query from disk only semantic Fix the issue that previous optimization for special case (multiple same URL in prefetcher list) breaks the `queryCacheType` option sematic Add autoreleasepool to DisplayLinkCallback to release objects

dreampiggy · 2025-02-17T09:48:19Z

SDWebImage/Core/SDImageCoderHelper.m

+    if (@available(macOS 10.15, iOS 13, watchOS 6.0, *)) {
+        CGColorSpaceRef colorSpace = CGImageGetColorSpace(cgImage);
+        if (colorSpace) {
+            return CGColorSpaceIsHDR(colorSpace);


这里实现不正确，参考文档重新写一个，获取CGColorSpace检查CGColorSpaceUsesITUR_2100TF

然后macOS AppKit那个NSImage.sd_isHighDynamicRange实现，可以先拿到CGImage（放到UIImage+Compat.h/.m那里），然后调用这里的封装CGImageIsHDR，或者也可以copy代码过去，性能可能好一点？

CGColorSpaceIsHDR应该是包含了CGColorSpaceUsesITUR_2100TF、CGColorSpaceIsPQBased、CGColorSpaceIsHLGBased

这部分你确认一下技术资料，我看这个API标记为deprecated，应该是有背景原因的。是不是会错判？还是ISO HDR本身一定是UsesITUR_2100TF+

SDWebImage/Core/UIImage+Metadata.h

…encoding UTI

Need actually check the HDR info of CGImage, or better, we can pre-decode to drop lazy HDR image

…ics renderer This can inherit the possible info like dynamic range

Tested on macOS 14.5 and iOS 18.0 behavior

…as well

* Support HDR * Support HDR * When using SDWebImageProgressiveLoad and setting it to SDWebImageContextImageDecodeToHDR, it will be decoded to HDR only when loading is complete. * When SDImageIOAnimatedCoder turns on incremental, it does not need to be decoded to HDR when it is not completed * Sample image to remove sensitive information * 1、use remote resources for HDR testing 2、add simulator log for HDR * Support HDR encode * Support HDR encode * imageRef release error * HDR Decoded is not required for decoding, otherwise the type will be lost * HDR encode format * hdrImageRef not released correctly * HDR image must be lazy decode * HDR image must be lazy decode * JEPG HDR image must be lazy decode, otherwise it will crash * HDR, encoding properties add kCGImageDestinationEncodeToISOGainmap, compatible with SDR displays while preserving HDR * 支持 decode to HDR * HDR encoding is not currently supported * add UIImage.sd_isHighDynamicRange, use check UIImage is HDR * refactor: Do not hack on HEICS and distinguish static/aniamted image encoding UTI * refactor: Move cross-platform screen info into SDDeviceHelper * change: Do not disable force decode when turn on decodeToHDR Need actually check the HDR info of CGImage, or better, we can pre-decode to drop lazy HDR image * change: use UIImage.imageRendererFormat when force decode using graphics renderer This can inherit the possible info like dynamic range * fix: When decode HDR image to SDR, need specify the decode request Tested on macOS 14.5 and iOS 18.0 behavior * test: Added unit test for HDR decoding * demo: Update the macOS demo to show the HDR image * test: workaround the SDR decode on Simulator environment * change: The `sd_isHighDynamicRange` should check CGImage as fallback as well * test: temp disable a unused test case --------- Co-authored-by: DreamPiggy <lizhuoli1126@126.com>

dreampiggy · 2025-02-20T14:08:00Z

Note: due to the wrong git operation, this PR is merged into 92a7ab9 instead of f4f93c2

CloudlessMoon added 4 commits December 24, 2024 16:43

Support HDR

3183eb6

Support HDR

b29fe53

When using SDWebImageProgressiveLoad and setting it to SDWebImageCont…

dbe352f

…extImageDecodeToHDR, it will be decoded to HDR only when loading is complete.

When SDImageIOAnimatedCoder turns on incremental, it does not need to…

5161f5b

… be decoded to HDR when it is not completed

CloudlessMoon mentioned this pull request Dec 25, 2024

support HDR image #3553

Open

CloudlessMoon added 2 commits December 25, 2024 12:39

Sample image to remove sensitive information

89c4cae

1、use remote resources for HDR testing

b82f03d

2、add simulator log for HDR

Support HDR encode

ad047fd

CloudlessMoon added 4 commits December 26, 2024 17:42

Support HDR encode

95585bc

imageRef release error

2b87528

HDR Decoded is not required for decoding, otherwise the type will be …

48d19e8

…lost

HDR encode format

6a3fdfd

CloudlessMoon added 4 commits December 27, 2024 08:49

hdrImageRef not released correctly

bafeec3

HDR image must be lazy decode

36a7b5a

HDR image must be lazy decode

b3abab7

JEPG HDR image must be lazy decode, otherwise it will crash

afde004

HDR, encoding properties add kCGImageDestinationEncodeToISOGainmap, c…

22ecfc8

…ompatible with SDR displays while preserving HDR

CloudlessMoon added 3 commits February 17, 2025 16:35

支持 decode to HDR

6bf367d

HDR encoding is not currently supported

4b3a0a7

dreampiggy reviewed Feb 17, 2025

View reviewed changes

add UIImage.sd_isHighDynamicRange, use check UIImage is HDR

556192f

dreampiggy reviewed Feb 17, 2025

View reviewed changes

SDWebImage/Core/UIImage+Metadata.h Show resolved Hide resolved

refactor: Do not hack on HEICS and distinguish static/aniamted image …

c3f19e7

…encoding UTI

dreampiggy added ImageIO Anything related to Apple ImageIO codec HDR High Dynamic Range image labels Feb 19, 2025

dreampiggy added 7 commits February 20, 2025 00:00

refactor: Move cross-platform screen info into SDDeviceHelper

5f4c9e6

change: Do not disable force decode when turn on decodeToHDR

feba692

Need actually check the HDR info of CGImage, or better, we can pre-decode to drop lazy HDR image

change: use UIImage.imageRendererFormat when force decode using graph…

233f1ca

…ics renderer This can inherit the possible info like dynamic range

fix: When decode HDR image to SDR, need specify the decode request

feb2c8d

Tested on macOS 14.5 and iOS 18.0 behavior

test: Added unit test for HDR decoding

7249cdc

demo: Update the macOS demo to show the HDR image

5813462

test: workaround the SDR decode on Simulator environment

757f324

dreampiggy changed the title ~~Support HDR~~ Support decoding ISO HDR (avif/heic/jpeg-xl, etc) Feb 20, 2025

dreampiggy added 2 commits February 20, 2025 21:27

change: The sd_isHighDynamicRange should check CGImage as fallback …

9868122

…as well

test: temp disable a unused test case

e77acfb

dreampiggy merged commit f4f93c2 into SDWebImage:master Feb 20, 2025
6 of 7 checks passed

dreampiggy added this to the 5.21.0 milestone Feb 20, 2025

Uh oh!

Support decoding ISO HDR (avif/heic/jpeg-xl, etc) #3778

Support decoding ISO HDR (avif/heic/jpeg-xl, etc) #3778

Uh oh!

Conversation

CloudlessMoon commented Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CloudlessMoon commented Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CloudlessMoon commented Dec 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CloudlessMoon commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CloudlessMoon commented Dec 30, 2024

Uh oh!

CloudlessMoon commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy commented Dec 31, 2024

Uh oh!

CloudlessMoon commented Dec 31, 2024

Uh oh!

dreampiggy commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dreampiggy Feb 17, 2025

Choose a reason for hiding this comment

Uh oh!

dreampiggy Feb 17, 2025

Choose a reason for hiding this comment

Uh oh!

CloudlessMoon Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

dreampiggy Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dreampiggy commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CloudlessMoon commented Dec 24, 2024 •

edited

Loading

dreampiggy commented Dec 25, 2024 •

edited

Loading

CloudlessMoon commented Dec 25, 2024 •

edited

Loading

dreampiggy commented Dec 26, 2024 •

edited

Loading

dreampiggy commented Dec 26, 2024 •

edited

Loading

CloudlessMoon commented Dec 26, 2024 •

edited

Loading

dreampiggy commented Dec 27, 2024 •

edited

Loading

CloudlessMoon commented Dec 27, 2024 •

edited

Loading

dreampiggy commented Dec 30, 2024 •

edited

Loading

dreampiggy commented Dec 30, 2024 •

edited

Loading

CloudlessMoon commented Dec 30, 2024 •

edited

Loading

dreampiggy commented Jan 6, 2025 •

edited

Loading

dreampiggy Feb 18, 2025 •

edited

Loading