Skip to content

Adjust the logic added by #1246#1269

Merged
seratch merged 1 commit intoslackapi:mainfrom
seratch:socket-mode-adjustment
Sep 29, 2022
Merged

Adjust the logic added by #1246#1269
seratch merged 1 commit intoslackapi:mainfrom
seratch:socket-mode-adjustment

Conversation

@seratch
Copy link
Copy Markdown
Contributor

@seratch seratch commented Sep 29, 2022

Summary

This pull request tweaks an internal code in the built-in Socket Mode client. Since #1246 changes landed, we've received feedback from users that an active connection can be terminated unexpectedly.

Traceback (most recent call last):
 File "/usr/local/lib/python3.9/site-packages/slack_sdk/socket_mode/builtin/connection.py", line 305, in run_until_completion
   received_messages: List[Tuple[Optional[FrameHeader], bytes]] = _receive_messages(
 File "/usr/local/lib/python3.9/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 213, in _receive_messages
   return _fetch_messages(
 File "/usr/local/lib/python3.9/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 236, in _fetch_messages
   remaining_bytes = receive()  # type: ignore
 File "/usr/local/lib/python3.9/site-packages/slack_sdk/socket_mode/builtin/internals.py", line 208, in receive
   raise ConnectionError("Connection is closed")
ConnectionError: Connection is closed

I think that this specific change can be reverted for the sake of even more stable behavior.

Category (place an x in each of the [ ])

  • slack_sdk.web.WebClient (sync/async) (Web API client)
  • slack_sdk.webhook.WebhookClient (sync/async) (Incoming Webhook, response_url sender)
  • slack_sdk.socket_mode (Socket Mode client)
  • slack_sdk.signature (Request Signature Verifier)
  • slack_sdk.oauth (OAuth Flow Utilities)
  • slack_sdk.models (UI component builders)
  • slack_sdk.scim (SCIM API client)
  • slack_sdk.audit_logs (Audit Logs API client)
  • slack_sdk.rtm_v2 (RTM client)
  • /docs-src (Documents, have you run ./scripts/docs.sh?)
  • /docs-src-v2 (Documents, have you run ./scripts/docs-v2.sh?)
  • /tutorial (PythOnBoardingBot tutorial)
  • tests/integration_tests (Automated tests for this library)

Requirements (place an x in each [ ])

  • I've read and understood the Contributing Guidelines and have done my best effort to follow them.
  • I've read and agree to the Code of Conduct.
  • I've run python3 -m venv .venv && source .venv/bin/activate && ./scripts/run_validation.sh after making the changes.

@seratch seratch added bug M-T: A confirmed bug report. Issues are confirmed when the reproduction steps are documented Version: 3x socket-mode labels Sep 29, 2022
@seratch seratch added this to the 3.18.4 milestone Sep 29, 2022
@seratch seratch self-assigned this Sep 29, 2022
@codecov
Copy link
Copy Markdown

codecov bot commented Sep 29, 2022

Codecov Report

Merging #1269 (a0fe15b) into main (0782f56) will decrease coverage by 0.48%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1269      +/-   ##
==========================================
- Coverage   86.64%   86.15%   -0.49%     
==========================================
  Files         111       84      -27     
  Lines       11027     7771    -3256     
==========================================
- Hits         9554     6695    -2859     
+ Misses       1473     1076     -397     
Impacted Files Coverage Δ
slack_sdk/socket_mode/builtin/internals.py 72.44% <ø> (+0.63%) ⬆️
slack_sdk/oauth/__init__.py
slack_sdk/web/client.py
slack_sdk/oauth/installation_store/__init__.py
slack_sdk/__init__.py
slack_sdk/web/async_base_client.py
slack_sdk/models/__init__.py
slack_sdk/webhook/__init__.py
slack_sdk/web/async_slack_response.py
slack_sdk/signature/__init__.py
... and 19 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

size = specific_buffer_size if specific_buffer_size is not None else receive_buffer_size
with sock_receive_lock:
received_bytes = sock.recv(size)
if not received_bytes:
Copy link
Copy Markdown
Contributor

@WilliamBergamin WilliamBergamin Sep 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, based on the docs of this recv the connection should retry before throwing an Error, and the behavior in _fetch_messages should properly handle the situation where the bytes are None or of length 0 🥇

But let me know if I miss understood the intent of the change

Copy link
Copy Markdown
Contributor

@eddyg eddyg Sep 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would disabling suppress_ragged_eofs when setting up the SSLContext help in this situation, allowing an exception to be caught?

I think the "unexpected EOF error" occurs when the underlying socket indicates EOF, but the connection was not shutdown at the SSL protocol level. Along with EOF from the other end of the connection, it can happen due to a proxy, or anything else on the network that can affect the connection.

Note that there may implications to disabling suppress_ragged_eofs, such as needing to catch SSLEOFError in other places...

(I'm not sure what the intention of this check was, so just throwing this out there... if removing the check "works", it's probably easier to just go that route...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the input @eddyg 💯 I was able to find the source code of this behavior it does seem like enabling suppress_ragged_eofs could allow us to sort actual EOF from SSL_ERROR_EOF

Adding some sort of retry logic in _fetch_messages when a SSL_ERROR_EOF is raised could be a good idea, what do you think @seratch?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that suppress_ragged_eofs is enabled by default in Python, and causes an improperly-terminated SSL session to just return "EOF".

By disabling it in the SSLContext, SSLEOFErrors would propagate out to this code. But as I mentioned, there may be consequences in doing so... there's likely a reason suppressing ragged EOFs defaults to True in Python.

It may be interesting to disable it in a test app and see if SSLEOFErrors are seen communicating with the Slack server pipeline (my understanding is that it is actually quite common to terminate a session without properly tearing down the SSL session inside... that's probably why this parameter defaults to True!)

Anyway, this suggestion may be nothing more than a red herring, because as I mentioned, I'm not sure why the two-lines being proposed for removal were added in the first place! (Apologies for adding confusion!)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eddyg @WilliamBergamin Thanks a lot for you both's comments here! I think it's fine to have this revert but we may add/change the internals more in the future for the betterment of EOF and other pattern handling!

@seratch seratch merged commit a6aac5d into slackapi:main Sep 29, 2022
@seratch seratch deleted the socket-mode-adjustment branch September 29, 2022 21:21
seratch added a commit to seratch/python-slack-sdk that referenced this pull request Sep 30, 2022
@seratch seratch mentioned this pull request Sep 30, 2022
16 tasks
seratch added a commit that referenced this pull request Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug M-T: A confirmed bug report. Issues are confirmed when the reproduction steps are documented socket-mode Version: 3x

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants