net.i2p.util

## Class EepGet

- java.lang.Object
- - net.i2p.util.EepGet

- Direct Known Subclasses:
    EepHead, EepPost, I2PSocketEepGet, PartialEepGet, SSLEepGet

      public class EepGet
      extends Object

  EepGet [-p 127.0.0.1:4444] [-n #retries] [-o outputFile] [-m markSize lineLen] url

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `protected static class` | `EepGet.AUTH_MODE` Authentication mode enumeration.
    `protected class` | `EepGet.AuthState` Manage the authentication parameters Ref: RFC 2617 Supports both Basic and Digest, however i2ptunnel HTTP proxy has migrated all previous Basic support to Digest.
    `protected class` | `EepGet.CLIStatusListener`
    `protected class` | `EepGet.Gunzipper` Decompressor thread.
    `static interface` | `EepGet.StatusListener` Callback interface

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected boolean` | `_aborted`
    `protected String` | `_acceptRanges`
    `protected String` | `_actualURL` the URL we actually fetch from (may differ from the _url in case of redirect)
    `protected long` | `_alreadyTransferred`
    `protected EepGet.AuthState` | `_authState`
    `protected long` | `_bytesRemaining`
    `protected long` | `_bytesTransferred`
    `protected String` | `_cacheControl`
    `protected String` | `_contentEncoding`
    `protected String` | `_contentLanguage`
    `protected String` | `_contentType`
    `protected I2PAppContext` | `_context`
    `protected String` | `_cookie`
    `protected String` | `_csp`
    `protected int` | `_currentAttempt`
    `protected IOException` | `_decompressException`
    `protected boolean` | `_encodingChunked`
    `protected String` | `_etag`
    `protected String` | `_etagOrig`
    `protected String` | `_expiryDate`
    `protected List<String>` | `_extraHeaders`
    `protected int` | `_fetchHeaderTimeout`
    `protected int` | `_fetchInactivityTimeout`
    `protected int` | `_fetchTotalTimeout`
    `protected boolean` | `_isGzippedResponse`
    `protected boolean` | `_keepFetching`
    `protected String` | `_lastModified`
    `protected String` | `_lastModifiedOrig`
    `protected List<EepGet.StatusListener>` | `_listeners`
    `protected Log` | `_log`
    `protected boolean` | `_notModified`
    `protected int` | `_numRetries`
    `protected OutputStream` | `_out`
    `protected String` | `_outputFile`
    `protected OutputStream` | `_outputStream`
    `protected Socket` | `_proxy`
    `protected String` | `_proxyHost`
    `protected InputStream` | `_proxyIn`
    `protected OutputStream` | `_proxyOut`
    `protected int` | `_proxyPort`
    `protected String` | `_redirectLocation`
    `protected int` | `_redirects`
    `protected String` | `_referrerPolicy`
    `protected int` | `_responseCode`
    `protected String` | `_responseText`
    `protected String` | `_server`
    `protected boolean` | `_shouldProxy`
    `protected boolean` | `_shouldWriteErrorToOutput`
    `protected String` | `_status`
    `protected String` | `_transferEncoding`
    `protected boolean` | `_transferFailed`
    `protected String` | `_url` url we were asked to fetch
    `protected String` | `_vary`
    `protected String` | `_xContentTypeOptions`
    `protected String` | `_xframeOptions`
    `protected String` | `_xPoweredBy`
    `protected String` | `_xssProtection`
    `protected static int` | `CONNECT_TIMEOUT`
    `protected static int` | `INACTIVITY_TIMEOUT`
    `protected static int` | `MAX_COMPLETE_FAILS` maximum times to try without getting any data at all, even if numRetries is higher @since 0.7.14
    `protected static String` | `USER_AGENT` this will be replaced by the HTTP Proxy if we are using it

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String postData)`
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, long minSize, long maxSize, String outputFile, OutputStream outputStream, String url, boolean allowCaching, String etag, String lastModified, String postData)`
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url)`
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching, String etag)`
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching, String etag, String lastModified)`
    `EepGet(I2PAppContext ctx, boolean shouldProxy, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, String postData)`
    `EepGet(I2PAppContext ctx, int numRetries, String outputFile, String url)`
    `EepGet(I2PAppContext ctx, int numRetries, String outputFile, String url, boolean allowCaching)`
    `EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url)`
    `EepGet(I2PAppContext ctx, String proxyHost, int proxyPort, int numRetries, String outputFile, String url, boolean allowCaching)`

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `void` | `addAuthorization(String userName, String password)` Add basic authorization header for the proxy.
    `void` | `addHeader(String name, String value)` Add an extra header to the request.
    `void` | `addStatusListener(EepGet.StatusListener lsnr)`
    `protected void` | `doFetch(SocketTimeout timeout)` This reads the response to a single fetch.
    `boolean` | `fetch()` Blocking fetch, returning true if the URL was retrieved, false if all retries failed.
    `boolean` | `fetch(long fetchHeaderTimeout)` Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given.
    `boolean` | `fetch(long fetchHeaderTimeout, long totalTimeout, long inactivityTimeout)` Blocking fetch.
    `String` | `getAcceptRanges()` Show the Accept-Ranges field
    `String` | `getCacheControl()` Show the Cache-Control field
    `String` | `getContentEncoding()` Show the Content-Encoding field
    `String` | `getContentLanguage()` Show the Content-Language field
    `String` | `getContentType()` After fetch, the received value from the server, or null if none.
    `String` | `getCookie()` Show the Set-Cookie field
    `String` | `getCSP()` Show the Content-Security-Policy field
    `String` | `getEtag()` After fetch, the received value from the server, or null if none.
    `String` | `getExpiryDate()` Show the Expires field
    `String` | `getLastModified()` After fetch, the received value from the server, or null if none.
    `boolean` | `getNotModified()`
    `String` | `getReferrerPolicy()` Show the Referrer-Policy field
    `protected String` | `getRequest()`
    `String` | `getServer()` Show the Server field
    `String` | `getStatus()` Show a combined server status message e.g.
    `int` | `getStatusCode()` The server response (200, etc).
    `String` | `getStatusText()` The server text ("OK", "Not Found", etc).
    `String` | `getTransferEncoding()` Show the Transfer-Encoding field
    `String` | `getVary()` Show the Vary field
    `String` | `getXContentTypeOptions()` Show the X-Content-Type-Options field
    `String` | `getXframeOptions()` Show the X-frame-Options field
    `String` | `getXPoweredBy()` Show the X-Powered-By field
    `String` | `getXSSProtection()` Show the X-XSS-Protection field
    `static void` | `main(String[] args)` EepGet [-p 127.0.0.1:4444] [-n #retries] [-e etag] [-o outputFile] [-m markSize lineLen] url As of 0.9.45, supports https and redirect to https
    `static Map<String,String>` | `parseAuthArgs(String args)` Parse the args in an authentication header.
    `protected long` | `readChunkLength()` TODO this does not skip over chunk extensions (RFC 2616 sec.
    `protected void` | `readHeaders()`
    `protected void` | `sendRequest(SocketTimeout timeout)`
    `protected void` | `setPostData(String contentType, byte[] data)` Set post data.
    `protected void` | `setPostData(String contentType, File data)` Set post data.
    `protected void` | `setPostData(String contentType, String data)` Set post data.
    `void` | `setWriteErrorToOutput()` If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc).
    `protected boolean` | `shouldReadBody()` Should we read the body of the response?
    `void` | `stopFetching()`
    `static String` | `suggestName(String url)` Parse URL for a viable filename.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### _aborted

          protected volatile boolean _aborted

    - #### _acceptRanges

          protected String _acceptRanges

    - #### _actualURL

          protected String _actualURL

      the URL we actually fetch from (may differ from the _url in case of redirect)

    - #### _alreadyTransferred

          protected long _alreadyTransferred

    - #### _authState

          protected EepGet.AuthState _authState

    - #### _bytesRemaining

          protected long _bytesRemaining

    - #### _bytesTransferred

          protected long _bytesTransferred

    - #### _cacheControl

          protected String _cacheControl

    - #### _contentEncoding

          protected String _contentEncoding

    - #### _contentLanguage

          protected String _contentLanguage

    - #### _contentType

          protected String _contentType

    - #### _context

          protected final I2PAppContext _context

    - #### _cookie

          protected String _cookie

    - #### _csp

          protected String _csp

    - #### _currentAttempt

          protected int _currentAttempt

    - #### _decompressException

          protected IOException _decompressException

    - #### _encodingChunked

          protected volatile boolean _encodingChunked

    - #### _etag

          protected String _etag

    - #### _etagOrig

          protected final String _etagOrig

    - #### _expiryDate

          protected String _expiryDate

    - #### _extraHeaders

          protected List<String> _extraHeaders

    - #### _fetchHeaderTimeout

          protected volatile int _fetchHeaderTimeout

    - #### _fetchInactivityTimeout

          protected volatile int _fetchInactivityTimeout

    - #### _fetchTotalTimeout

          protected volatile int _fetchTotalTimeout

    - #### _isGzippedResponse

          protected boolean _isGzippedResponse

    - #### _keepFetching

          protected volatile boolean _keepFetching

    - #### _lastModified

          protected String _lastModified

    - #### _lastModifiedOrig

          protected final String _lastModifiedOrig

    - #### _listeners

          protected final List<EepGet.StatusListener> _listeners

    - #### _log

          protected final Log _log

    - #### _notModified

          protected volatile boolean _notModified

    - #### _numRetries

          protected final int _numRetries

    - #### _out

          protected OutputStream _out

    - #### _outputFile

          protected final String _outputFile

    - #### _outputStream

          protected final OutputStream _outputStream

    - #### _proxy

          protected Socket _proxy

    - #### _proxyHost

          protected final String _proxyHost

    - #### _proxyIn

          protected InputStream _proxyIn

    - #### _proxyOut

          protected OutputStream _proxyOut

    - #### _proxyPort

          protected final int _proxyPort

    - #### _redirectLocation

          protected String _redirectLocation

    - #### _redirects

          protected volatile int _redirects

    - #### _referrerPolicy

          protected String _referrerPolicy

    - #### _responseCode

          protected volatile int _responseCode

    - #### _responseText

          protected String _responseText

    - #### _server

          protected String _server

    - #### _shouldProxy

          protected final boolean _shouldProxy

    - #### _shouldWriteErrorToOutput

          protected volatile boolean _shouldWriteErrorToOutput

    - #### _status

          protected String _status

    - #### _transferEncoding

          protected String _transferEncoding

    - #### _transferFailed

          protected volatile boolean _transferFailed

    - #### _url

          protected final String _url

      url we were asked to fetch

    - #### _vary

          protected String _vary

    - #### _xContentTypeOptions

          protected String _xContentTypeOptions

    - #### _xframeOptions

          protected String _xframeOptions

    - #### _xPoweredBy

          protected String _xPoweredBy

    - #### _xssProtection

          protected String _xssProtection

    - #### CONNECT_TIMEOUT

          protected static final int CONNECT_TIMEOUT

      See Also:
        Constant Field Values

    - #### INACTIVITY_TIMEOUT

          protected static final int INACTIVITY_TIMEOUT

      See Also:
        Constant Field Values

    - #### MAX_COMPLETE_FAILS

          protected static final int MAX_COMPLETE_FAILS

      maximum times to try without getting any data at all, even if numRetries is higher @since 0.7.14
      See Also:
        Constant Field Values

    - #### USER_AGENT

          protected static final String USER_AGENT

      this will be replaced by the HTTP Proxy if we are using it
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        long minSize,
                        long maxSize,
                        String outputFile,
                        OutputStream outputStream,
                        String url,
                        boolean allowCaching,
                        String etag,
                        String postData)

      Parameters:
        `outputFile` - ignored if outputStream is non-null
        `outputStream` - takes precedence over outputFile

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        long minSize,
                        long maxSize,
                        String outputFile,
                        OutputStream outputStream,
                        String url,
                        boolean allowCaching,
                        String etag,
                        String lastModified,
                        String postData)

      Parameters:
        `outputFile` - ignored if outputStream is non-null
        `outputStream` - takes precedence over outputFile

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url,
                        boolean allowCaching,
                        String etag)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url,
                        boolean allowCaching,
                        String etag,
                        String lastModified)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        boolean shouldProxy,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url,
                        String postData)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        int numRetries,
                        String outputFile,
                        String url)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        int numRetries,
                        String outputFile,
                        String url,
                        boolean allowCaching)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url)

    - #### EepGet

          public EepGet(I2PAppContext ctx,
                        String proxyHost,
                        int proxyPort,
                        int numRetries,
                        String outputFile,
                        String url,
                        boolean allowCaching)

  - ### Method Detail

    - #### addAuthorization

          public void addAuthorization(String userName,
                                       String password)

      Add basic authorization header for the proxy. Only added if the request is going through a proxy. Must be called before fetch().

    - #### addHeader

          public void addHeader(String name,
                                String value)

      Add an extra header to the request. Must be called before fetch(). Not supported by EepHead. As of 0.9.10, If name is User-Agent, this will replace the default User-Agent header. As of 0.9.14, If name is If-None-Match or If-Modified-Since, this will replace the etag or last-modified value given in the constructor. Note that headers may be subsequently modified or removed in the I2PTunnel HTTP Client proxy. In proxied SSLEepGet, these headers are sent to the remote server, NOT the proxy.

    - #### addStatusListener

          public void addStatusListener(EepGet.StatusListener lsnr)

    - #### doFetch

          protected void doFetch(SocketTimeout timeout)
                          throws IOException

      This reads the response to a single fetch. Call after sendRequest()
      Parameters:
        `timeout` - may be null
      Throws:
        `IOException`

    - #### fetch

          public boolean fetch()

      Blocking fetch, returning true if the URL was retrieved, false if all retries failed. Header timeout default 45 sec, total timeout default none, inactivity timeout default 60 sec.
      Returns:
        success

    - #### fetch

          public boolean fetch(long fetchHeaderTimeout)

      Blocking fetch, timing out individual attempts if the HTTP response headers don't come back in the time given. If the timeout is zero or less, this will wait indefinitely. Total timeout default none, inactivity timeout default 60 sec.
      Returns:
        success

    - #### fetch

          public boolean fetch(long fetchHeaderTimeout,
                               long totalTimeout,
                               long inactivityTimeout)

      Blocking fetch.
      Parameters:
        `fetchHeaderTimeout` - <= 0 for none (proxy will timeout if none, none isn't recommended if no proxy)
        `totalTimeout` - <= 0 for default none
        `inactivityTimeout` - <= 0 for default 60 sec
      Returns:
        success

    - #### getAcceptRanges

          public String getAcceptRanges()

      Show the Accept-Ranges field

    - #### getCacheControl

          public String getCacheControl()

      Show the Cache-Control field

    - #### getContentEncoding

          public String getContentEncoding()

      Show the Content-Encoding field

    - #### getContentLanguage

          public String getContentLanguage()

      Show the Content-Language field

    - #### getContentType

          public String getContentType()

      After fetch, the received value from the server, or null if none.

    - #### getCookie

          public String getCookie()

      Show the Set-Cookie field

    - #### getCSP

          public String getCSP()

      Show the Content-Security-Policy field

    - #### getEtag

          public String getEtag()

      After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor.

    - #### getExpiryDate

          public String getExpiryDate()

      Show the Expires field

    - #### getLastModified

          public String getLastModified()

      After fetch, the received value from the server, or null if none. Before fetch, and after some errors, may be the value passed in the constructor.

    - #### getNotModified

          public boolean getNotModified()

      Returns:
        true if the server returned 304

    - #### getReferrerPolicy

          public String getReferrerPolicy()

      Show the Referrer-Policy field

    - #### getRequest

          protected String getRequest()
                               throws IOException

      Throws:
        `IOException`

    - #### getServer

          public String getServer()

      Show the Server field

    - #### getStatus

          public String getStatus()

      Show a combined server status message e.g. "200 OK"

    - #### getStatusCode

          public int getStatusCode()

      The server response (200, etc).
      Returns:
        -1 if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns 200.

    - #### getStatusText

          public String getStatusText()

      The server text ("OK", "Not Found", etc). Note that the text may contain % encoding.
      Returns:
        null if invalid, or if the proxy never responded, or if no proxy was used and the server never responded. If a non-proxied request partially succeeded (for example a redirect followed by a fail, or a partial fetch followed by a fail), this will be the last status code received. Note that fetch() may return false even if this returns "OK".

    - #### getTransferEncoding

          public String getTransferEncoding()

      Show the Transfer-Encoding field

    - #### getVary

          public String getVary()

      Show the Vary field

    - #### getXContentTypeOptions

          public String getXContentTypeOptions()

      Show the X-Content-Type-Options field

    - #### getXframeOptions

          public String getXframeOptions()

      Show the X-frame-Options field

    - #### getXPoweredBy

          public String getXPoweredBy()

      Show the X-Powered-By field

    - #### getXSSProtection

          public String getXSSProtection()

      Show the X-XSS-Protection field

    - #### main

          public static void main(String[] args)

      EepGet [-p 127.0.0.1:4444] [-n #retries] [-e etag] [-o outputFile] [-m markSize lineLen] url As of 0.9.45, supports https and redirect to https

    - #### parseAuthArgs

          public static Map<String,String> parseAuthArgs(String args)

      Parse the args in an authentication header. Modified from LoadClientAppsJob. All keys are mapped to lower case. Double quotes around values are stripped. Ref: RFC 2617 Public for I2PTunnelHTTPClientBase; use outside of tree at own risk, subject to change or removal
      Parameters:
        `args` - non-null, starting after "Digest " or "Basic "

    - #### readChunkLength

          protected long readChunkLength()
                                  throws IOException

      TODO this does not skip over chunk extensions (RFC 2616 sec. 3.6.1)
      Throws:
        `IOException`

    - #### readHeaders

          protected void readHeaders()
                              throws IOException

      Throws:
        `IOException`

    - #### sendRequest

          protected void sendRequest(SocketTimeout timeout)
                              throws IOException

      Parameters:
        `timeout` - may be null
      Throws:
        `IOException`

    - #### setPostData

          protected void setPostData(String contentType,
                                     byte[] data)

      Set post data. Must be called before fetch().
      Throws:
        `IllegalStateException` - if already set

    - #### setPostData

          protected void setPostData(String contentType,
                                     File data)

      Set post data. Must be called before fetch().
      Throws:
        `IllegalStateException` - if already set

    - #### setPostData

          protected void setPostData(String contentType,
                                     String data)

      Set post data. Must be called before fetch().
      Throws:
        `IllegalStateException` - if already set

    - #### setWriteErrorToOutput

          public void setWriteErrorToOutput()

      If called (before calling fetch()), data from the server or proxy will be written to the output file or stream even on an error response code (4xx, 5xx, etc). The error data will only be written if no previous data was written on an earlier try. Caller must of course check getStatusCode() or the fetch() return value.

    - #### shouldReadBody

          protected boolean shouldReadBody()

      Should we read the body of the response?
      Returns:
        true always, overridden in EepHead

    - #### stopFetching

          public void stopFetching()

    - #### suggestName

          public static String suggestName(String url)

      Parse URL for a viable filename.
      Parameters:
        `url` - a URL giving the location of an online resource
      Returns:
        a filename to save the resource as on local filesystem