Fixing http adapter to allow HTTPS connections via HTTP by Khaledgarbaya · Pull Request #959 · axios/axios

Khaledgarbaya · 2017-06-15T12:14:04Z

Summary

We are using axios at Contentful for both our JS SDKs contentful.js and contentful-management.js.

Axios was assuming always that the proxy protocol is the same as the URL it is requesting. This a problem when you use a http proxy when requesting data from https endpoint.
see more at #925 and #753

Purpose of the PR

This PR make httpAdapter to allow https connections via http by introducing a new config param to the proxy object called isHttps which defaults to false

TODO

Add Integration test ?
Add documentation

coveralls · 2017-06-15T12:16:12Z

Coverage remained the same at 93.734% when pulling 27375aa on contentful:fix/https-via-http-proxy into 46e275c on mzabriskie:master.

coveralls · 2017-06-15T16:09:46Z

Coverage remained the same at 93.734% when pulling b829df3 on contentful:fix/https-via-http-proxy into 46e275c on mzabriskie:master.

coveralls · 2017-06-15T16:19:39Z

Coverage remained the same at 93.734% when pulling b2c7cb4 on contentful:fix/https-via-http-proxy into 46e275c on mzabriskie:master.

coveralls · 2017-06-16T12:28:12Z

Coverage remained the same at 93.734% when pulling e6c944f on contentful:fix/https-via-http-proxy into 46e275c on mzabriskie:master.

coveralls · 2017-06-16T12:32:28Z

Coverage remained the same at 93.734% when pulling 486c956 on contentful:fix/https-via-http-proxy into 46e275c on mzabriskie:master.

wpegg-dev · 2017-06-28T19:32:29Z

@mzabriskie When is this going to get merged and released? I need it for a project I'm working on and will need to move to something else if it's not going to be soon

Implemented changes from axios#959

vcfvct · 2017-07-05T19:30:45Z

Please merge this, for now, we have to use the contentful 3.8.1 as a work around.

rubennorte · 2017-08-14T11:49:53Z

Please remove all changes relating the package.json and the generated files in dist as that would be part of our release process.

rubennorte · 2017-08-14T11:51:11Z

index.d.ts

 export interface AxiosProxyConfig {
  host: string;
  port: number;
+  isHttps?: boolean;


I'd use a protocol option instead, accepting both http and http: formats so parsed URL can be passed as the proxy option.

rubennorte · 2017-08-14T11:51:57Z

lib/adapters/http.js

    }

    var transport;
+    var useHttps = isHttps && (proxy ? proxy.isHttps : true);


I'd use isHttpsProxy instead as it's clearer.

karuppasamy · 2017-09-07T08:42:47Z

Please merge this, I need this behavior for my project. I switched to axios from request.js recently. I don't wanna do any doubts/ in my decision.

coveralls · 2017-09-07T09:46:13Z

Coverage remained the same at 93.812% when pulling e9a9c62 on contentful:fix/https-via-http-proxy into 07a7b7c on mzabriskie:master.

axe312ger · 2017-09-07T11:36:37Z

<3

ctyu · 2017-09-12T07:06:23Z

Please merge this, i need this for a project that i working on.
Thanks.

tmpace · 2017-09-25T21:16:31Z

Can we get this merged? We're also having issues with this, and are using a fork for now.

colin-riddell · 2017-10-10T10:19:14Z

FWIW: It would be excellent to see this merged in sometime :)

mrchief · 2017-10-12T13:20:19Z

Doesn't fix the real problem for me. Instead, fails with a nicer (and faster) error:

Error: Protocol "http:" not supported. Expected "https:"

zcei · 2017-10-16T20:09:17Z

@mrchief are you trying to connect to an HTTP endpoint through an HTTPS proxy? Do you have a small code snippet that fails?

mrchief · 2017-10-16T20:36:48Z

@zcei I was using Fiddler proxy. I set it up to decrypt HTTPS traffic so that it listens for https requests.

I don't have any small snippet but it should be reproducible fairly easily by:

configure fiddler proxy to decrypt https
pass proxy to axios (127.0.0.1:8888 - fiddler defaults)
run unit tests or make a call to any mock HTTPS API (requestbin, mockbin etc. will work) via axios.

The request should fail with above error.

zcei · 2017-10-18T11:30:16Z

@mrchief Did you pass it like this literal value? '127.0.0.1:8888'? Pretty sure it should be 'https://127.0.0.1:8888' then, to indicate your intention to connect via HTTPS

mrchief · 2017-10-18T13:28:02Z

Tried passing it as https://.... This time its a different error and looks all weird. Here are the axios configs (some parts redacted):

 Message -> getaddrinfo ENOTFOUND https://127.0.0.1 https://127.0.0.1:8888
 ---------------------------------------------------------
 Config -> { adapter: [Function: httpAdapter],
   transformRequest: { '0': [Function: transformRequest] },
   transformResponse: { '0': [Function: transformResponse] },
   timeout: 0,
   xsrfCookieName: 'XSRF-TOKEN',
   xsrfHeaderName: 'X-XSRF-TOKEN',
   maxContentLength: -1,
   validateStatus: [Function: validateStatus],
   headers:
    { Accept: 'application/json, text/plain, */*',
      Authorization: 'Bearer ...',
      'User-Agent': 'axios/0.16.2',
      host: 'api.test.io' },
   method: 'get',
   url: 'https://api.test.io/test',
   params: { ... },
   data: undefined,
   proxy: { host: 'https://127.0.0.1', port: 8888 } }
 ---------------------------------------------------------
 Request -> Writable {
   _writableState:
    WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      bufferedRequest: null,
      lastBufferedRequest: null,
      pendingcb: 0,
      prefinished: false,
Emitted: false,
      bufferedRequestCount: 0,
      corkedRequestsFree:
       { next: null,
         entry: null,
         finish: [Function: bound onCorkedFinish] } },
   writable: true,
   domain: null,
   _events:
    { response: [Function: handleResponse],
: [Funct] },
   _eventsCount: 2,
   _maxListeners: undefined,
   _options:
    { maxRedirects: 21,
      protocol: 'http:',
      hostname: 'https://127.0.0.1',
      port: 8888,
      path: 'https://api.test.io/test?...',
      method: 'get',
      headers:
       { Accept: 'application/json, text/plain, */*',         
         Authorization: 'Bearer ...',
         'User-Agent': 'axios/0.16.2',
         host: 'api.test.io' },
      agent: undefined,
      auth: undefined,
      host: 'https://127.0.0.1',
      pathname: 'https://api.test.io/test',
      search: '?...' },
   _redirectCount: 0,
   _bufferedWrites: [],
   _onNativeResponse: [Function],
   _currentRequest:
    ClientRequest {
      domain: null,
      _events:
       { response: { [Function: bound onceWrapper] listener: [Function] },
         socket: [Function],
         abort: [Function],
         aborted: [Function],
: [Function] },
      _eventsCount: 5,
      _maxListeners: undefined,
      output: [],
      outputEncodings: [],
      outputCallbacks: [],
      outputSize: 0,
      writable: false,
      _last: true,
      upgrading: false,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket:
       Socket {
         connecting: false,
: true,
         _handle: null,
         _parent: null,
         _host: 'https://127.0.0.1',
         _readableState:
          ReadableState {
            objectMode: false,
            highWaterMark: 16384,
            buffer: BufferList { head: null, tail: null, length: 0 },
            length: 0,
            pipes: null,
            pipesCount: 0,
            flowing: true,
            ended: false,
            endEmitted: false,
            reading: true,
            sync: false,
            needReadable: true,
            emittedReadable: false,
            readableListening: false,
            resumeScheduled: false,
            destroyed: true,
            defaultEncoding: 'utf8',
            awaitDrain: 0,
            readingMore: false,
            decoder: null,
            encoding: null },
         readable: false,
         domain: null,
         _events:
          { end: { [Function: bound onceWrapper] listener: [Function: onend] },
            finish: [Function: onSocketFinish],
            _socketEnd: [Function: onSocketEnd],
            connect:
             [ { [Function: bound onceWrapper] listener: [Function: oncreate] },
               { [Function: bound onceWrapper] listener: [Function: connect] },
               { [Function: bound onceWrapper] listener: [Function] } ],
            free: [Function: onFree],
            close: [ [Function: onClose], [Function: socketCloseListener] ],
            agentRemove: [Function: onRemove],
            drain: [Function: ondrain],
:Listener] },
         _eventsCount: 9,
         _maxListeners: undefined,
         _writableState:
          WritableState {
            objectMode: false,
            highWaterMark: 16384,
            finalCalled: false,
            needDrain: false,
            ending: false,
            ended: false,
            finished: false,
            destroyed: true,
            decodeStrings: false,
            defaultEncoding: 'utf8',
            length: 1456,
            writing: true,
            corked: 0,
            sync: false,
            bufferProcessing: false,
            onwrite: [Function: bound onwrite],
            writecb: [Function: bound onFinish],
            writelen: 1456,
            bufferedRequest: null,
            lastBufferedRequest: null,
            pendingcb: 1,
            prefinished: false,
Emitted: true,
            bufferedRequestCount: 0,
            corkedRequestsFree:
             { next: null,
               entry: null,
               finish: [Function: bound onCorkedFinish] } },
         writable: false,
         allowHalfOpen: false,
         _bytesDispatched: 0,
         _sockname: null,
         _pendingData: 'GET https://api.test.io/test?... HTTP/1.1\r\nAccept: application/json,
 text/plain, */*\r\nAuthorization: Bearer ...\r\nUser-Agent: axios/0.16.2\r\nhost: api.test.io\r\nConnectio
n: close\r\n\r\n',
         _pendingEncoding: 'latin1',
         server: null,
         _server: null,
         parser: null,
         _httpMessage: [Circular],
         read: [Function],
         _consuming: true,
         _idleNext: null,
         _idlePrev: null,
         _idleTimeout: -1,
         [Symbol(asyncId)]: 292884,
         [Symbol(bytesRead)]: 0 },
      connection:
       Socket {
         connecting: false,
: true,
         _handle: null,
         _parent: null,
         _host: 'https://127.0.0.1',
         _readableState:
          ReadableState {
            objectMode: false,
            highWaterMark: 16384,
            buffer: BufferList { head: null, tail: null, length: 0 },
            length: 0,
            pipes: null,
            pipesCount: 0,
            flowing: true,
            ended: false,
            endEmitted: false,
            reading: true,
            sync: false,
            needReadable: true,
            emittedReadable: false,
            readableListening: false,
            resumeScheduled: false,
            destroyed: true,
            defaultEncoding: 'utf8',
            awaitDrain: 0,
            readingMore: false,
            decoder: null,
            encoding: null },
         readable: false,
         domain: null,
         _events:
          { end: { [Function: bound onceWrapper] listener: [Function: onend] },
            finish: [Function: onSocketFinish],
            _socketEnd: [Function: onSocketEnd],
            connect:
             [ { [Function: bound onceWrapper] listener: [Function: oncreate] },
               { [Function: bound onceWrapper] listener: [Function: connect] },
               { [Function: bound onceWrapper] listener: [Function] } ],
            free: [Function: onFree],
            close: [ [Function: onClose], [Function: socketCloseListener] ],
            agentRemove: [Function: onRemove],
            drain: [Function: ondrain],
:Listener] },
         _eventsCount: 9,
         _maxListeners: undefined,
         _writableState:
          WritableState {
            objectMode: false,
            highWaterMark: 16384,
            finalCalled: false,
            needDrain: false,
            ending: false,
            ended: false,
            finished: false,
            destroyed: true,
            decodeStrings: false,
            defaultEncoding: 'utf8',
            length: 1456,
            writing: true,
            corked: 0,
            sync: false,
            bufferProcessing: false,
            onwrite: [Function: bound onwrite],
            writecb: [Function: bound onFinish],
            writelen: 1456,
            bufferedRequest: null,
            lastBufferedRequest: null,
            pendingcb: 1,
            prefinished: false,
Emitted: true,
            bufferedRequestCount: 0,
            corkedRequestsFree:
             { next: null,
               entry: null,
               finish: [Function: bound onCorkedFinish] } },
         writable: false,
         allowHalfOpen: false,
         _bytesDispatched: 0,
         _sockname: null,
         _pendingData: 'GET https://api.test.io/test?... HTTP/1.1\r\nAccept: application/json,
 text/plain, */*\r\nAuthorization: Bearer ...\r\nUser-Agent: axios/0.16.2\r\nhost: api.test.io\r\nConnectio
n: close\r\n\r\n',
         _pendingEncoding: 'latin1',
         server: null,
         _server: null,
         parser: null,
         _httpMessage: [Circular],
         read: [Function],
         _consuming: true,
         _idleNext: null,
         _idlePrev: null,
         _idleTimeout: -1,
         [Symbol(asyncId)]: 292884,
         [Symbol(bytesRead)]: 0 },
      _header: 'GET https://api.test.io/test?... HTTP/1.1\r\nAccept: application/json, text/pl
ain, */*\r\nAuthorization: Bearer ...\r\nUser-Agent: axios/0.16.2\r\nhost: api.test.io\r\nConnection: close
\r\n\r\n',
      _onPendingData: [Function: noopPendingOutput],
      agent:
       Agent {
         domain: null,
         _events: { free: [Function] },
         _eventsCount: 1,
         _maxListeners: undefined,
         defaultPort: 80,
         protocol: 'http:',
         options: { path: null },
         requests: {},
         sockets:
          { 'https://127.0.0.1:8888:':
             [ Socket {
                 connecting: false,
  : true,
                 _handle: null,
                 _parent: null,
                 _host: 'https://127.0.0.1',
                 _readableState:
                  ReadableState {
                    objectMode: false,
                    highWaterMark: 16384,
                    buffer: BufferList { head: null, tail: null, length: 0 },
                    length: 0,
                    pipes: null,
                    pipesCount: 0,
                    flowing: true,
                    ended: false,
                    endEmitted: false,
                    reading: true,
                    sync: false,
                    needReadable: true,
                    emittedReadable: false,
                    readableListening: false,
                    resumeScheduled: false,
                    destroyed: true,
                    defaultEncoding: 'utf8',
                    awaitDrain: 0,
                    readingMore: false,
                    decoder: null,
                    encoding: null },
                 readable: false,
                 domain: null,
                 _events:
                  { end: { [Function: bound onceWrapper] listener: [Function: onend] },
                    finish: [Function: onSocketFinish],
                    _socketEnd: [Function: onSocketEnd],
                    connect:
                     [ { [Function: bound onceWrapper] listener: [Function: oncreate] },
                       { [Function: bound onceWrapper] listener: [Function: connect] },
                       { [Function: bound onceWrapper] listener: [Function] } ],
                    free: [Function: onFree],
                    close: [ [Function: onClose], [Function: socketCloseListener] ],
                    agentRemove: [Function: onRemove],
                    drain: [Function: ondrain],
  :Listener] },
                 _eventsCount: 9,
                 _maxListeners: undefined,
                 _writableState:
                  WritableState {
                    objectMode: false,
                    highWaterMark: 16384,
                    finalCalled: false,
                    needDrain: false,
                    ending: false,
                    ended: false,
                    finished: false,
                    destroyed: true,
                    decodeStrings: false,
                    defaultEncoding: 'utf8',
                    length: 1456,
                    writing: true,
                    corked: 0,
                    sync: false,
                    bufferProcessing: false,
                    onwrite: [Function: bound onwrite],
                    writecb: [Function: bound onFinish],
                    writelen: 1456,
                    bufferedRequest: null,
                    lastBufferedRequest: null,
                    pendingcb: 1,
                    prefinished: false,
  Emitted: true,
                    bufferedRequestCount: 0,
                    corkedRequestsFree:
                     { next: null,
                       entry: null,
                       finish: [Function: bound onCorkedFinish] } },
                 writable: false,
                 allowHalfOpen: false,
                 _bytesDispatched: 0,
                 _sockname: null,
                 _pendingData: 'GET https://api.test.io/test?... HTTP/1.1\r\nAccept: applicati
on/json, text/plain, */*\r\nAuthorization: Bearer ...\r\nUser-Agent: axios/0.16.2\r\nhost: api.test.io\r\nC
onnection: close\r\n\r\n',
                 _pendingEncoding: 'latin1',
                 server: null,
                 _server: null,
                 parser: null,
                 _httpMessage: [Circular],
                 read: [Function],
                 _consuming: true,
                 _idleNext: null,
                 _idlePrev: null,
                 _idleTimeout: -1,
                 [Symbol(asyncId)]: 292884,
                 [Symbol(bytesRead)]: 0 } ] },
         freeSockets: {},
         keepAliveMsecs: 1000,
         keepAlive: false,
         maxSockets: Infinity,
         maxFreeSockets: 256 },
      socketPath: undefined,
      timeout: undefined,
      method: 'GET',
      path: 'https://api.test.io/test?...',
      _ended: false,
      res: null,
      aborted: undefined,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      _redirectable: [Circular],
      [Symbol(outHeadersKey)]:
       { accept: [ 'Accept', 'application/json, text/plain, */*' ],
         authorization:
          [ 'Authorization',
            'Bearer ...' ],
         'user-agent': [ 'User-Agent', 'axios/0.16.2' ],
         host: [ 'host', 'api.test.io' ] } },
   _currentUrl: 'http://https://127.0.0.1/https://api.test.io/test?...' }
 ---------------------------------------------------------
 Response -> {}
 ---------------------------------------------------------

mrchief · 2017-10-18T13:46:05Z

It seems passing it on the request object works:

  const request = {
    headers
    , url: `${baseUrl}/${path}`
    , params
    , data
    , method
    , proxy: {
      host: '127.0.0.1'
      , port: 8888
    }
  }

axios.request(request) // works and the call shows up on fiddler

I was trying to set it on the instance before:

// doesn't work

instance = axios.create({
  proxy: {
      host: '127.0.0.1'
      , port: 8888
    }
})

The readme indicates that the same config can be used (for create and request) or maybe its just an unfortunate allusion due to usage of same word (config).

It might help to separate them if that's the case, i.e create config is not the same as request config.

The same thing without your patch gives me this error: write EPROTO 101057795:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:openssl\ssl\s23_clnt.c:797:

So I guess your patch does work! :)

zcei · 2017-10-23T09:27:25Z

@mrchief thanks for the additional info! 🙏

The create vs request thing caused a lot of confusion/issues beforehand, as it is (or at least has been, when I checked last) some kind of weird global shared state, so when you do a request, you're actually using a shared default instance. Maybe there's something wrong with applying this configuration to new instances, but I guess that's out of scope for this PR.

Glad you faced it, though, so other people can refer to your findings in case they face it as well.

This was referenced Jun 15, 2017

Request to HTTPS with HTTP proxy fails #925

Closed

Socket hang up when request behind $https_proxy #753

Closed

Khaledgarbaya mentioned this pull request Jun 15, 2017

fix(axios): fix proxy contentful/contentful-management.js#112

Merged

zcei force-pushed the fix/https-via-http-proxy branch from e6c944f to 486c956 Compare June 16, 2017 12:29

ssddanbrown pushed a commit to CleverTouch/axios that referenced this pull request Jul 3, 2017

Patched with cross-protocol proxy requests

fcafef1

Implemented changes from axios#959

rubennorte reviewed Aug 14, 2017

View reviewed changes

zcei force-pushed the fix/https-via-http-proxy branch from 486c956 to e9a9c62 Compare September 7, 2017 09:43

Khaledgarbaya mentioned this pull request Oct 4, 2017

Missing dependency for Axios 0.16.3 contentful/contentful-management.js#141

Closed

This was referenced Apr 30, 2020

[Snyk] Upgrade axios from 0.18.0 to 0.19.2 leonyork/cognito-oauth2-demo#3

Open

[Snyk] Upgrade axios from 0.18.1 to 0.19.2 absalomedia/paperless-desktop#53

Open

axios locked and limited conversation to collaborators May 3, 2020

Uh oh!

Conversation

Khaledgarbaya commented Jun 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Purpose of the PR

TODO

Uh oh!

coveralls commented Jun 15, 2017

Uh oh!

coveralls commented Jun 15, 2017

Uh oh!

coveralls commented Jun 15, 2017

Uh oh!

coveralls commented Jun 16, 2017

Uh oh!

coveralls commented Jun 16, 2017

Uh oh!

wpegg-dev commented Jun 28, 2017

Uh oh!

vcfvct commented Jul 5, 2017

Uh oh!

rubennorte commented Aug 14, 2017

Uh oh!

rubennorte Aug 14, 2017

Choose a reason for hiding this comment

Uh oh!

rubennorte Aug 14, 2017

Choose a reason for hiding this comment

Uh oh!

karuppasamy commented Sep 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 7, 2017

Uh oh!

axe312ger commented Sep 7, 2017

Uh oh!

ctyu commented Sep 12, 2017

Uh oh!

tmpace commented Sep 25, 2017

Uh oh!

colin-riddell commented Oct 10, 2017

Uh oh!

mrchief commented Oct 12, 2017

Uh oh!

zcei commented Oct 16, 2017

Uh oh!

mrchief commented Oct 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zcei commented Oct 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrchief commented Oct 18, 2017

Uh oh!

mrchief commented Oct 18, 2017

Uh oh!

zcei commented Oct 23, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Khaledgarbaya commented Jun 15, 2017 •

edited

Loading

karuppasamy commented Sep 7, 2017 •

edited

Loading

mrchief commented Oct 16, 2017 •

edited

Loading

zcei commented Oct 18, 2017 •

edited

Loading