formlogin plugin does not respect headers #2238
Description
Expected behaviour:
Log into a website with the passed url.
Actual behaviour:
A 403 error, related to #2219
Steps to reproduce:
- Step 1: Configure formlogin to log into anidb.net (or any website with similar restrictions)
Config:
task:
form:
url: "https://anidb.net/perl-bin/animedb.pl?show=login"
username: <username>
password: <password>
Log:
(click to expand)
2018-10-21 18:20 DEBUG task test_form executing test_form
2018-10-21 18:20 DEBUG headers test_form headers to add: {'user-agent': 'test-user-agent/0.0'}
2018-10-21 18:20 DEBUG status test_form Adding new task test_form
2018-10-21 18:20 DEBUG utils.requests test_form GETing URL https://anidb.net/perl-bin/animedb.pl?show=login with args () and kwargs {'allow_redirects': True, 'timeout': 30}
2018-10-21 18:20 DEBUG formlogin test_form Exception getting login page.
Traceback (most recent call last):
File "/home/xvicarious/repos/Flexget/flexget/plugins/operate/formlogin.py", line 53, in on_task_start
response = br.open(url)
File "/home/xvicarious/.pyenv/versions/3.6.6/envs/flexget/lib/python3.6/site-packages/mechanicalsoup/stateful_browser.py", line 133, in open
resp = self.get(url, *args, **kwargs)
File "/home/xvicarious/.pyenv/versions/3.6.6/envs/flexget/lib/python3.6/site-packages/mechanicalsoup/browser.py", line 124, in get
response = self.session.get(*args, **kwargs)
File "/home/xvicarious/.pyenv/versions/3.6.6/envs/flexget/lib/python3.6/site-packages/requests/sessions.py", line 536, in get
return self.request('GET', url, **kwargs)
File "/home/xvicarious/repos/Flexget/flexget/utils/requests.py", line 248, in request
result.raise_for_status()
File "/home/xvicarious/.pyenv/versions/3.6.6/envs/flexget/lib/python3.6/site-packages/requests/models.py", line 929, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://anidb.net/perl-bin/animedb.pl?show=login
2018-10-21 18:20 CRITICAL formlogin test_form Unable to get login page
2018-10-21 18:20 WARNING task test_form Aborting task (plugin: form)
2018-10-21 18:20 DEBUG task_queue task test_form aborted: TaskAbort(reason=Unable to get login page, silent=False)
2018-10-21 18:20 DEBUG task_queue task queue shut down
2018-10-21 18:20 INFO ipc.rpyc server has terminated
2018-10-21 18:20 INFO ipc.rpyc listener closed
2018-10-21 18:20 DEBUG util.simple_persistence Flushing simple persistence for task None to db.
2018-10-21 18:20 DEBUG manager Shutting down
2018-10-21 18:20 INFO manager Removed test database
2018-10-21 18:20 DEBUG manager Removed /home/xvicarious/repos/Flexget/.test-config-lock
Additional information:
- FlexGet version: 2.17.0 (also applies to develop branch)
- Python version: 3.6.6 (also in dev environment)
- Installation method: docker (git pull)
- Using daemon (yes/no): yes (no in dev)
- OS and version: Alpine 3.8 (dev Arch Linux 4.18.14-arch1-1-ARCH)
- Details:
FormLogincreates aStatefulBrowserfrom MechanicalSoup, which inheritsBrowserfrom the same packageFormLoginpasses only the requests session, but it also accepts a keyword argument for a user agentBrowsersets its session to the passes session, and sets headers to what was given touser_agent, if it wasNone, it sets its own