On the old boto library is was simple enough to use the proxy, proxy_port, proxy_user and proxy_pass parameters when you open a connection. However, I could not find any equivalent way of programmatically define the proxy parameters on boto3. :(
4 Answers
As of at least version 1.5.79, botocore accepts a proxies argument in the botocore config.
e.g.
import boto3
from botocore.config import Config
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'}))
boto3 resource https://boto3.readthedocs.io/en/latest/reference/core/session.html#boto3.session.Session.resource
botocore config https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config
3 Comments
albarji
I'm using boto3==1.4.6, botocore==1.6.6, but this does not seem to be working for me. Could you please provide a full example loading a file into a bucket, or something similar?
Brad
Since AWS uses HTTPS for all endpoints, try this configuration:
boto3.resource('s3', config=Config(proxies={'https': 'foo.bar:3128'})) (note that it's https)
PedroMVM
I had the same issue with STS. I've put that config into client and it worked, In case someone else need:
conn = boto3.client('sts', config=Config(proxies={'http': 'myproxy', 'https': 'myproxy'}))If you user proxy server does not have a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://proxy.com:port"
os.environ["HTTPS_PROXY"] = "https://proxy.com:port"
if you user proxy server has a password try the following:
import os
os.environ["HTTP_PROXY"] = "http://user:[email protected]:port"
os.environ["HTTPS_PROXY"] = "https://user:[email protected]:port"
3 Comments
mightymephisto
Although ensure you correctly include quotations around your envirion values. This answer has them missing at the beginning of the proxy
albarji
This works, but changing an environment variable is troublesome. If along your program you need to perform an http request to other server, such request will get routed through the s3 proxy server, which is not what you want. You can kind of solve this by restoring both env variables to their original values after you are finished querying s3, but I wish there was a better solution.
user881703
@albarji, it depends on how you design your application, it is a basic principle to environment values isolate from the main application. Then you may have s3_proxy.config, serverA_proxy.config
Apart from altering the environment variable, I'll present what I found in the code.
Since boto3 uses botocore, I had a look through the source code:
From this link, we end up at:
def _get_proxies(self, url):
# We could also support getting proxies from a config file,
# but for now proxy support is taken from the environment.
return get_environ_proxies(url)
...which is called by proxies = self._get_proxies(final_endpoint_url) in the EndpointCreator class.
Long story short, if you're using python2 it will use the getproxies method from urllib2 and if you're using python3, it will use urllib3.
get_environ_proxies is expecting a dict containing {'http:' 'url'} (and I'm guessing https too).
You could always patch the code, but that is poor practice.
Comments
This is one of the rare occasions when I would recommend monkey-patching, at least until the Boto developers allow connection-specific proxy settings:
import botocore.endpoint
def _get_proxies(self, url):
return {'http': 'http://someproxy:1234/', 'https': 'https://someproxy:1234/'}
botocore.endpoint.EndpointCreator._get_proxies = _get_proxies
import boto3
import os ;os.environ["HTTP_PROXY"] = "http://yourproxy.com:1234"; os.environ["HTTPS_PROXY"] = "https://yourproxy1.com:1234"HTTPS_PROXYwould not work, onlyHTTP_PROXY.