Skip to content

Add one extra byte to return value of deflateBound for small lengths.#1236

Merged
Dead2 merged 2 commits intozlib-ng:developfrom
mtl1979:deflatebound-underflow
Apr 22, 2022
Merged

Add one extra byte to return value of deflateBound for small lengths.#1236
Dead2 merged 2 commits intozlib-ng:developfrom
mtl1979:deflatebound-underflow

Conversation

@mtl1979
Copy link
Copy Markdown
Collaborator

@mtl1979 mtl1979 commented Apr 5, 2022

Due to right shift by 3 in DEFLATE_QUICK_OVERHEAD() macro, the return value of deflateBound() is 1 too small for lengths smaller than 9.

See #1235.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 5, 2022 •
edited
Loading

Codecov Report

Merging #1236 (c0eeba5) into develop (41faa08) will decrease coverage by 0.57%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop    #1236      +/-   ##
===========================================
- Coverage    87.95%   87.38%   -0.58%     
===========================================
  Files          114      114              
  Lines        10147    10147              
  Branches      2539     2555      +16     
===========================================
- Hits          8925     8867      -58     
+ Misses         972      957      -15     
- Partials       250      323      +73
Flag Coverage Δ
macos_clang 30.95% <ø> (ø)
macos_gcc 74.99% <25.00%> (+0.02%) ⬆️
ubuntu_clang 87.42% <100.00%> (+0.16%) ⬆️
ubuntu_clang_debug 86.96% <100.00%> (+0.11%) ⬆️
ubuntu_clang_inflate_allow_invalid_dist 87.12% <100.00%> (+0.14%) ⬆️
ubuntu_clang_inflate_strict 87.37% <100.00%> (+0.16%) ⬆️
ubuntu_clang_mmap 87.56% <100.00%> (+0.26%) ⬆️
ubuntu_clang_pigz 40.94% <0.00%> (-0.36%) ⬇️
ubuntu_clang_pigz_no_optim 41.13% <0.00%> (-0.40%) ⬇️
ubuntu_clang_pigz_no_threads 40.52% <0.00%> (-0.35%) ⬇️
ubuntu_clang_reduced_mem 87.82% <100.00%> (+0.19%) ⬆️
ubuntu_gcc 77.07% <25.00%> (+0.07%) ⬆️
ubuntu_gcc_aarch64 77.37% <25.00%> (+0.09%) ⬆️
ubuntu_gcc_aarch64_compat_no_opt 75.47% <ø> (+0.05%) ⬆️
ubuntu_gcc_aarch64_no_acle 76.27% <25.00%> (+0.02%) ⬆️
ubuntu_gcc_aarch64_no_neon 76.17% <25.00%> (+0.02%) ⬆️
ubuntu_gcc_armhf 77.39% <25.00%> (+0.09%) ⬆️
ubuntu_gcc_armhf_compat_no_opt 75.38% <ø> (+0.06%) ⬆️
ubuntu_gcc_armhf_no_acle 77.40% <25.00%> (+0.09%) ⬆️
ubuntu_gcc_armhf_no_neon 77.16% <25.00%> (+0.09%) ⬆️
ubuntu_gcc_armsf 77.25% <25.00%> (+0.09%) ⬆️
ubuntu_gcc_armsf_compat_no_opt 75.05% <ø> (+0.06%) ⬆️
ubuntu_gcc_benchmark 75.00% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_compat_no_opt 76.61% <ø> (+0.13%) ⬆️
ubuntu_gcc_compat_sprefix 75.10% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_m32 74.84% <25.00%> (+0.13%) ⬆️
ubuntu_gcc_mingw_i686 0.00% <0.00%> (ø)
ubuntu_gcc_mingw_x86_64 0.00% <0.00%> (ø)
ubuntu_gcc_no_avx2 76.73% <25.00%> (+0.01%) ⬆️
ubuntu_gcc_no_ctz 74.77% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_no_ctzll 74.80% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_no_pclmulqdq 75.51% <25.00%> (+0.03%) ⬆️
ubuntu_gcc_no_sse2 76.79% <25.00%> (+0.01%) ⬆️
ubuntu_gcc_no_sse4 76.32% <25.00%> (+0.01%) ⬆️
ubuntu_gcc_o1 75.27% <25.00%> (+0.60%) ⬆️
ubuntu_gcc_osb ∅ <ø> (∅)
ubuntu_gcc_pigz 38.94% <0.00%> (-0.45%) ⬇️
ubuntu_gcc_pigz_aarch64 39.40% <0.00%> (-0.43%) ⬇️
ubuntu_gcc_ppc 73.81% <25.00%> (+0.13%) ⬆️
ubuntu_gcc_ppc64 74.56% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_ppc64le 74.54% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_ppc_no_power8 74.74% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_s390x 74.73% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_s390x_dfltcc 72.03% <0.00%> (-0.11%) ⬇️
ubuntu_gcc_s390x_dfltcc_compat 73.64% <0.00%> (-0.24%) ⬇️
ubuntu_gcc_s390x_no_crc32 74.49% <25.00%> (+0.13%) ⬆️
ubuntu_gcc_sparc64 74.62% <25.00%> (+0.12%) ⬆️
ubuntu_gcc_sprefix 74.89% <25.00%> (+0.12%) ⬆️
win64_gcc 73.95% <25.00%> (+0.23%) ⬆️
win64_gcc_compat_no_opt 74.02% <ø> (+0.31%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
compress.c 97.82% <100.00%> (+0.09%) ⬆️
deflate.c 83.62% <100.00%> (+0.04%) ⬆️
test/fuzz/fuzzer_minigzip.c 50.38% <0.00%> (-20.32%) ⬇️
test/fuzz/fuzzer_example_dict.c 70.66% <0.00%> (-19.46%) ⬇️
test/fuzz/fuzzer_example_small.c 76.66% <0.00%> (-17.09%) ⬇️
test/fuzz/fuzzer_example_large.c 75.00% <0.00%> (-15.67%) ⬇️
test/fuzz/fuzzer_example_flush.c 79.36% <0.00%> (-13.06%) ⬇️
test/fuzz/fuzzer_compress.c 97.50% <0.00%> (-0.18%) ⬇️
uncompr.c 91.30% <0.00%> (ø)
crc32_fold.c 100.00% <0.00%> (ø)
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41faa08...c0eeba5. Read the comment docs.

@iii-i iii-i mentioned this pull request Apr 5, 2022
Closed
@mtl1979 mtl1979 force-pushed the deflatebound-underflow branch from 8522b2e to d201fa1 Compare April 5, 2022 22:20
@mtl1979
Add one extra byte to return value of compressBound and deflateBound …
df44db5 
…for small lengths due to shift returning 0.

* Treat 0 byte input as 1 byte input when calculating compressBound and deflateBound
@mtl1979 mtl1979 force-pushed the deflatebound-underflow branch from d201fa1 to df44db5 Compare April 6, 2022 00:19
@mtl1979
Copy link
Copy Markdown
Collaborator Author

mtl1979 commented Apr 6, 2022 

bytes = 0, deflateBound = 23, total_out = 23
bytes = 1, deflateBound = 24, total_out = 24
bytes = 2, deflateBound = 25, total_out = 25
bytes = 3, deflateBound = 26, total_out = 26
bytes = 4, deflateBound = 27, total_out = 27
bytes = 5, deflateBound = 28, total_out = 28
bytes = 6, deflateBound = 29, total_out = 29
bytes = 7, deflateBound = 30, total_out = 30
bytes = 8, deflateBound = 31, total_out = 31
bytes = 9, deflateBound = 32, total_out = 32
bytes = 10, deflateBound = 33, total_out = 33
bytes = 11, deflateBound = 34, total_out = 34
bytes = 12, deflateBound = 35, total_out = 35
bytes = 13, deflateBound = 36, total_out = 36
bytes = 14, deflateBound = 37, total_out = 37
bytes = 15, deflateBound = 38, total_out = 38
bytes = 16, deflateBound = 39, total_out = 39
bytes = 17, deflateBound = 41, total_out = 40
bytes = 18, deflateBound = 42, total_out = 41
bytes = 19, deflateBound = 43, total_out = 42
bytes = 20, deflateBound = 44, total_out = 43
bytes = 21, deflateBound = 45, total_out = 44
bytes = 22, deflateBound = 46, total_out = 45
bytes = 23, deflateBound = 47, total_out = 46
bytes = 24, deflateBound = 48, total_out = 47
bytes = 25, deflateBound = 50, total_out = 48
bytes = 26, deflateBound = 51, total_out = 49
bytes = 27, deflateBound = 52, total_out = 50
bytes = 28, deflateBound = 53, total_out = 51
bytes = 29, deflateBound = 54, total_out = 52
bytes = 30, deflateBound = 55, total_out = 53
bytes = 31, deflateBound = 56, total_out = 54
bytes = 32, deflateBound = 57, total_out = 55

@Dead2 Now it returns one extra byte at lengths 17 and 25... I didn't try longer than 32 bytes input as the pattern is already clear, every 8 bytes it returns one more extra byte.

@nmoinvaz
Copy link
Copy Markdown
Member

nmoinvaz commented Apr 6, 2022

It would be better imo to add this to google test framework.

@mtl1979
Copy link
Copy Markdown
Collaborator Author

mtl1979 commented Apr 6, 2022

@nmoinvaz It's kinda work in progress... I'm not good at writing tests using google test framework...

@mtl1979
Add test for issue zlib-ng#1235.
c0eeba5 
* Test both compressBound() and deflateBound() as those share same code fragment.
@mtl1979 mtl1979 force-pushed the deflatebound-underflow branch from 35a5603 to c0eeba5 Compare April 7, 2022 15:42
@mtl1979
Copy link
Copy Markdown
Collaborator Author

mtl1979 commented Apr 7, 2022

Added test for compressBound() too, it's obvious it also would have failed with input length of 0, as I added 2 extra bytes in that case. For other lengths, there is now enough spare bytes to allow even uncompressible data. Tested with bytes filled with for loop to simulate worst case scenario.

bytes =  0, compressBound =  9, total_out =  8
bytes =  1, compressBound = 11, total_out =  9
bytes =  2, compressBound = 12, total_out = 10
bytes =  3, compressBound = 13, total_out = 11
bytes =  4, compressBound = 14, total_out = 12
bytes =  5, compressBound = 15, total_out = 13
bytes =  6, compressBound = 16, total_out = 14
bytes =  7, compressBound = 17, total_out = 15
bytes =  8, compressBound = 18, total_out = 16
bytes =  9, compressBound = 20, total_out = 17
bytes = 10, compressBound = 21, total_out = 18
bytes = 11, compressBound = 22, total_out = 19
bytes = 12, compressBound = 23, total_out = 20
bytes = 13, compressBound = 24, total_out = 21
bytes = 14, compressBound = 25, total_out = 22
bytes = 15, compressBound = 26, total_out = 23
bytes = 16, compressBound = 27, total_out = 24
bytes = 17, compressBound = 29, total_out = 25
bytes = 18, compressBound = 30, total_out = 26
bytes = 19, compressBound = 31, total_out = 27
bytes = 20, compressBound = 32, total_out = 28
bytes = 21, compressBound = 33, total_out = 29
bytes = 22, compressBound = 34, total_out = 30
bytes = 23, compressBound = 35, total_out = 31
bytes = 24, compressBound = 36, total_out = 32
bytes = 25, compressBound = 38, total_out = 33
bytes = 26, compressBound = 39, total_out = 34
bytes = 27, compressBound = 40, total_out = 35
bytes = 28, compressBound = 41, total_out = 36
bytes = 29, compressBound = 42, total_out = 37
bytes = 30, compressBound = 43, total_out = 38
bytes = 31, compressBound = 44, total_out = 39
bytes = 32, compressBound = 45, total_out = 40

@nmoinvaz nmoinvaz added the bug label Apr 8, 2022
@nmoinvaz nmoinvaz mentioned this pull request Apr 10, 2022
Merged
@Dead2 Dead2 merged commit 7690c73 into zlib-ng:develop Apr 22, 2022
@Dead2 Dead2 mentioned this pull request Dec 27, 2022
Merged
Dead2 added a commit that referenced this pull request Mar 7, 2023
@Dead2
Version 2.0.7
98ea610 
Changes since 2.0.6:
- Fix CVE-2022-37434 #1328
- Fix chunkmemset #1196
- Fix deflateBound too small #1236
- Fix Z_SOLO #1263
- Fix ACLE variant of crc32 #1274
- Fix inflateBack #1311
- Fix deflate_quick windowsize #1431
- Fix DFLTCC bugs related to adler32 #1349 and #1390
- Fix warnings #1194 #1312 #1362
- MacOS build fix #1198
- Add invalid windowBits handling #1293
- Support for Force TZCNT #1186
- Support for aligned_alloc() #1360
- Minideflate improvements #1175 #1238
- Dont use unaligned access for memcpy #1309
- Build system #1209 #1233 #1267 #1273 #1278 #1292 #1316 #1318 #1365
- Test improvements #1208 #1227 #1241 #1353
- Cleanup #1266
- Documentation #1205 #1359
- Misc improvements #1294 #1297 #1306 #1344 #1348
- Backported zlib fixes
- Backported CI workflows from Develop branch
Dead2 added a commit that referenced this pull request Mar 17, 2023
@Dead2
Version 2.0.7
a583e21 
Changes since 2.0.6:
- Fix CVE-2022-37434 #1328
- Fix chunkmemset #1196
- Fix deflateBound too small #1236
- Fix Z_SOLO #1263
- Fix ACLE variant of crc32 #1274
- Fix inflateBack #1311
- Fix deflate_quick windowsize #1431
- Fix DFLTCC bugs related to adler32 #1349 and #1390
- Fix warnings #1194 #1312 #1362
- MacOS build fix #1198
- Add invalid windowBits handling #1293
- Support for Force TZCNT #1186
- Support for aligned_alloc() #1360
- Minideflate improvements #1175 #1238
- Dont use unaligned access for memcpy #1309
- Build system #1209 #1233 #1267 #1273 #1278 #1292 #1316 #1318 #1365
- Test improvements #1208 #1227 #1241 #1353
- Cleanup #1266
- Documentation #1205 #1359
- Misc improvements #1294 #1297 #1306 #1344 #1348
- Backported zlib fixes
- Backported CI workflows from Develop branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dead2 Dead2 Dead2 approved these changes

Assignees

No one assigned

Labels

bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mtl1979 @nmoinvaz @Dead2