Custom headers not appended (403 when querying zooqle) #1876
Description
Trying to fetch RSS feed (personal and OpenSearch's) from Zooqle.com with custom headers.
Actual behaviour:
Fetching feeds from Zoogle returns 403, custom headers don't seem to be appended to the query.
Steps to reproduce:
flexget -L debug execute
Config:
task-b:
headers:
Host: "zoogle.com"
User-Agent: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Accept-Language: "en-US,en;q=0.5"
Accept-Encoding: "gzip, deflate, br"
Cookie: "__cfduid=d01f224e777a2e8e7bbec44205f036cf91497676643; cf_clearance=d3d599bf7b7fb1e8751ec716943b855e4135bb25-1497676663-28800"
Upgrade-Insecure-Requests: "1"
Connection: "keep-alive"
Pragma: "no-cache"
Cache-Control: "no-cache"
rss: https://zooqle.com/search?q=silicon+valley&fmt=rss
all_series: yes
transmission:
host: localhost
port: 9091
path: /var/lib/transmission-daemon/downloads/
addpaused: noLog:
2017-06-17 07:42 DEBUG task task-b executing task-b
2017-06-17 07:42 DEBUG headers task-b headers to add: {u'Accept-Language': u'en-US,en;q=0.5', u'Accept-Encoding': u'gzip, deflate, br', u'Host': u'zoogle.com', u'Accept': u'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', u'User-Agent': u'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0', u'Connection': u'keep-alive', u'Cookie': u'__cfduid=d01f224e777a2e8e7bbec44205f036cf91497676643; cf_clearance=d3d599bf7b7fb1e8751ec716943b855e4135bb25-1497676663-28800', u'Pragma': u'no-cache', u'Cache-Control': u'no-cache', u'Upgrade-Insecure-Requests': u'1'}
2017-06-17 07:42 DEBUG remember_rej task-b Task config has changed since last run, purging remembered entries.
2017-06-17 07:42 DEBUG input_cache task-b cache name: rss_6925a066760ed7562f1971d606b8ec8e (has: )
2017-06-17 07:42 DEBUG rss task-b Requesting task `task-b` url `https://zooqle.com/search?q=silicon+valley&fmt=rss`
2017-06-17 07:42 DEBUG utils.requests task-b Fetching URL https://zooqle.com/search?q=silicon+valley&fmt=rss with args () and kwargs {'headers': {}, 'allow_redirects': True, 'timeout': 60, 'auth': None}
2017-06-17 07:42 CRITICAL rss task-b HTTP error 403 received from https://zooqle.com/search?q=silicon+valley&fmt=rss
2017-06-17 07:42 WARNING task task-b Aborting task (plugin: rss)
2017-06-17 07:42 DEBUG task_queue task task-b aborted: TaskAbort(reason=HTTP error 403 received from https://zooqle.com/search?q=silicon+valley&fmt=rss, silent=False)
Specially:
headers to add: {u'Accept-Language': u'en-US,en;q=0.5', u'Accept-Encoding': u'gzip, deflate, br', u'Host': u'zoogle.com', u'Accept': u'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', u'User-Agent': u'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0', u'Connection': u'keep-alive', u'Cookie': u'__cfduid=d01f224e777a2e8e7bbec44205f036cf91497676643; cf_clearance=d3d599bf7b7fb1e8751ec716943b855e4135bb25-1497676663-28800', u'Pragma': u'no-cache', u'Cache-Control': u'no-cache', u'Upgrade-Insecure-Requests': u'1'}
Fetching URL https://zooqle.com/search?q=silicon+valley&fmt=rss with args () and kwargs {'headers': {}, 'allow_redirects': True, 'timeout': 60, 'auth': None}
`
Additional information:
- FlexGet version:
2.10.61 - Python version:
Python 2.7.12 - Installation method:
pip install flexget - Using daemon (yes/no): no
- OS and version:
Ubuntu 16.04.2 LTS 4.4.0-78-generic