Skip to content

Pin dnspython pip dependency due to regression in v2.0.0#4997

Merged
arm4b merged 5 commits intomasterfrom
fix/dnspython-regression
Jul 23, 2020
Merged

Pin dnspython pip dependency due to regression in v2.0.0#4997
arm4b merged 5 commits intomasterfrom
fix/dnspython-regression

Conversation

@arm4b
Copy link
Member

@arm4b arm4b commented Jul 20, 2020 •
edited
Loading

Recently both st2-docker https://circleci.com/gh/StackStorm/st2-docker/2199 and stackstorm-ha https://circleci.com/gh/StackStorm/stackstorm-ha/2393 nightly builds started to fail 🔥.

Fixes StackStorm/stackstorm-k8s#143
Fixes StackStorm/st2-docker#194

The issue is that st2 services started to fail if mongodb is a host and not an IP, meaning all remote-host scenarios are broken:

root@86f87ce3c778:/opt/stackstorm# /opt/stackstorm/st2/bin/st2api --config-file=/etc/st2/st2.conf --config-file=/etc/st2/st2.docker.conf --config-file=/etc/st2/st2.docker.conf
2020-07-19 15:47:53,175 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
2020-07-19 15:47:53,176 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.docker.conf
2020-07-19 15:47:53,179 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2020-07-19 15:47:53,261 INFO [-] Connecting to database "st2" @ "mongo:27017" as user "None".
2020-07-19 15:47:56,299 ERROR [-] Failed to connect to database "st2" @ "mongo:27017" as user "None": No servers found yet
2020-07-19 15:47:56,300 ERROR [-] (PID=791) ST2 API quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/cmd/api.py", line 84, in main
    _setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/cmd/api.py", line 58, in _setup
    service_registry=True, capabilities=capabilities)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 160, in setup
    db_setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 56, in db_setup
    connection = db_init.db_setup_with_retry(**db_cfg)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
    return retrying_obj.call(db_func, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/six.py", line 696, in reraise
    raise value
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 169, in db_setup
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 151, in _db_connect
    raise e
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 144, in _db_connect
    connection.admin.command('ismaster')
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/database.py", line 730, in command
    read_preference, session) as (sock_info, slave_ok):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
    server = self._select_server(read_preference, session)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
    server = topology.select_server(server_selector)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 235, in select_server
    address))
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 193, in select_servers
    selector, server_timeout, address)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: No servers found yet

Looking deeper, the only diff in pip dependencies between the working and broken deb packages was dnspython that updated from v1.16.0 to v2.0.0.

This PR pins dnspython to a working non-breaking version before 2.0.0.

Pin dnspython pip dependency due to regression in v2.0.0
19d71d9 
dnspython updated to v2.0.0 results in the following errors:
```
2020-07-19 15:47:56,299 ERROR [-] Failed to connect to database "st2" @ "mongo:27017" as user "None": No servers found yet
```
and
```
pymongo.errors.ServerSelectionTimeoutError: No servers found yet
```
@arm4b arm4b added the bug label Jul 20, 2020
@pull-request-size pull-request-size bot added the size/XS PR that changes 0-9 lines. Quick fix/merge. label Jul 20, 2020
@arm4b arm4b added this to the 3.3.0 milestone Jul 20, 2020
@arm4b arm4b mentioned this pull request Jul 20, 2020
Merged
nmaludy
nmaludy suggested changes Jul 20, 2020
View reviewed changes
Copy link
Member

@nmaludy nmaludy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need changelog i'm think here too

@arm4b
Copy link
Member Author

arm4b commented Jul 20, 2020 •
edited
Loading

@nmaludy Thanks!
I'll update it once this draft will be a complete ready for review PR.

@punkrokk
Copy link
Member

punkrokk commented Jul 21, 2020

Nice find

@arm4b arm4b mentioned this pull request Jul 21, 2020
Closed
@arm4b
Copy link
Member Author

arm4b commented Jul 22, 2020

I'll need to test the produced packages in both Docker/K8s environments next to verify the fix end-to-end.

@rahulshinde26 rahulshinde26 mentioned this pull request Jul 22, 2020
Closed
@arm4b arm4b marked this pull request as ready for review July 22, 2020 20:52
@arm4b arm4b requested a review from nmaludy July 22, 2020 22:59
@arm4b
Copy link
Member Author

arm4b commented Jul 22, 2020

Confirming the fix:

root@233e127d3f30:/opt/stackstorm# /opt/stackstorm/st2/bin/st2api --config-file=/etc/st2/st2.conf --config-file=/etc/st2/st2.docker.conf --config-file=/etc/st2/st2.user.conf2020-07-22 23:29:37,422 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
2020-07-22 23:29:37,440 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-07-22 23:29:37,456 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2020-07-22 23:29:37,908 INFO [-] Connecting to database "st2" @ "mongo:27017" as user "None".
2020-07-22 23:29:38,100 INFO [-] Successfully connected to database "st2" @ "mongo:27017" as user "None".
2020-07-22 23:29:55,692 INFO [-] (PID=214) ST2 API is serving on http://0.0.0.0:9101.
2020-07-22 23:29:55,693 INFO [-] Creating st2api: StackStorm v3.3dev as OpenAPI app.
2020-07-22 23:30:03,103 WARNING [-] Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
2020-07-22 23:30:04,208 INFO [-] Connected to amqp://guest:**@rabbitmq:5672//
2020-07-22 23:30:04,747 INFO [-] Connected to amqp://guest:**@rabbitmq:5672//

@nmaludy this is RFR now

@arm4b arm4b requested a review from a team July 23, 2020 11:36
@arm4b arm4b merged commit baf8fa7 into master Jul 23, 2020
@arm4b arm4b deleted the fix/dnspython-regression branch July 23, 2020 14:15
@manisha-tanwar
Copy link

manisha-tanwar commented Jul 23, 2020

Thanks for the fix @armab , It works!

@manisha-tanwar manisha-tanwar mentioned this pull request Jul 24, 2020
Closed
@cognifloyd cognifloyd mentioned this pull request Oct 19, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@nmaludy nmaludy nmaludy approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug size/XS PR that changes 0-9 lines. Quick fix/merge.

Projects

None yet

Milestone

3.3.0

4 participants

@arm4b @punkrokk @manisha-tanwar @nmaludy