-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
stream.hls: refactor segment decryption #4533
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
back-to
merged 1 commit into
streamlink:master
from
bastimeyer:stream/hls/decryption-log-error
May 18, 2022
Merged
stream.hls: refactor segment decryption #4533
back-to
merged 1 commit into
streamlink:master
from
bastimeyer:stream/hls/decryption-log-error
May 18, 2022
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
db62072 to
e3f2376
Compare
Member
I don't think we should include anything that's useless and adds to confusion. If someone else ever maintains the project they'll have to figure out why this was added. Let's drop it. |
e3f2376 to
542526f
Compare
- Catch ValueError when decrypting and unpadding segments, as well as other exceptions that can be raised when downloading the segment, similar to unencrypted segments, and log error messages accordingly. Don't close the stream on any of those errors. - Remove garbage data truncation from encrypted segments: This was (most likely) implemented because of an old issue related to Content-Length HTTP headers not being handled correctly, leading to partial segment downloads. Cutting off data that doesn't fit into the AES block size doesn't make sense, as it implies that the segment data incomplete, and incomplete segment data misses its padding bytes at the end which must always be included. - Update tests
542526f to
5070010
Compare
Member
Author
|
I've updated the PR and changed the following things compared to the initial push:
|
Billy2011
added a commit
to Billy2011/streamlink-27
that referenced
this pull request
May 19, 2022
- Catch ValueError when decrypting and unpadding segments, as well as other exceptions that can be raised when downloading the segment, similar to unencrypted segments, and log error messages accordingly. Don't close the stream on any of those errors. - Remove garbage data truncation from encrypted segments: This was (most likely) implemented because of an old issue related to Content-Length HTTP headers not being handled correctly, leading to partial segment downloads. Cutting off data that doesn't fit into the AES block size doesn't make sense, as it implies that the segment data incomplete, and incomplete segment data misses its padding bytes at the end which must always be included. - Update tests
Billy2011
added a commit
to Billy2011/streamlink-27
that referenced
this pull request
May 21, 2022
- Catch ValueError when decrypting and unpadding segments, as well as other exceptions that can be raised when downloading the segment, similar to unencrypted segments, and log error messages accordingly. Don't close the stream on any of those errors. - Remove garbage data truncation from encrypted segments: This was (most likely) implemented because of an old issue related to Content-Length HTTP headers not being handled correctly, leading to partial segment downloads. Cutting off data that doesn't fit into the AES block size doesn't make sense, as it implies that the segment data incomplete, and incomplete segment data misses its padding bytes at the end which must always be included. - Update tests
Billy2011
added a commit
to Billy2011/streamlink-27
that referenced
this pull request
May 24, 2022
- Catch ValueError when decrypting and unpadding segments, as well as other exceptions that can be raised when downloading the segment, similar to unencrypted segments, and log error messages accordingly. Don't close the stream on any of those errors. - Remove garbage data truncation from encrypted segments: This was (most likely) implemented because of an old issue related to Content-Length HTTP headers not being handled correctly, leading to partial segment downloads. Cutting off data that doesn't fit into the AES block size doesn't make sense, as it implies that the segment data incomplete, and incomplete segment data misses its padding bytes at the end which must always be included. - Update tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
other exceptions that can be raised when downloading the segment,
similar to unencrypted segments, and log error messages accordingly.
Don't close the stream on any of those errors.
This was (most likely) implemented because of an old issue related to
Content-Length HTTP headers not being handled correctly, leading to
partial segment downloads. Cutting off data that doesn't fit into the
AES block size doesn't make sense, as it implies that the segment data
incomplete, and incomplete segment data misses its padding bytes at
the end which must always be included.
updated commit message above, old initial comment below
ValueErrors can get raised by bothunpadanddecryptor.decrypt(). This change therefore also includes the garbage data cut-off when the segment length is not a multiple of the AES block size (before decrypting).We've had a discussion before where we were not sure why the garbage data cut-off was ever implemented. And to be honest, it doesn't make any sense having it, because if the encrypted segment is not a multiple of the AES block size, then it's an invalid segment and cutting off any remaining bytes surely won't help with fully decrypting it. The result will most likely be junk or incomplete data.
I believe that this was added in c38a543 to avoid
ValueErrors being raised when callingdecryptor.decrypt()when the segment length is invalid. This seemed to be a better solution to theContent-Lengthissue which could occur that resulted in partial/incomplete segment downloads (fixed by #3768), and it meant that the stream could continue.Any opinions on whether the garbage truncation should be removed? It doesn't hurt having it, but it's pretty much useless IMO. The
test_hls_encrypted_aes128_incorrect_block_lengthtest was added by me in b218259 just to have the code path covered.As you can see in the diff, it now logs the
Error while decrypting segment {sequence.num}error message instead of showing an exception call stack. This makes it much more clear that it's an issue with the stream, and not Streamlink.However, if an invalid decryption key is used (eg. #4528), the error message is still not 100% clear. Invalid decryption keys will result in nonsense data after decrypting the segment, so the
unpadfunction call will raise aValueErrorin 99.9% of all cases. I don't think it's too much of a problem though because it can't be distinguished from corrupted/broken segment downloads anyway, so adding something like "Invalid key?" isn't always a good fit.