Skip to content

PySAM does not handle 404 responses properly when fetching the weather files #94

Closed
Closed
PySAM does not handle 404 responses properly when fetching the weather files#94
@ghazi-git

Description

@ghazi-git
ghazi-git
opened on Jul 27, 2021

This issue happened while using pysam to get the weather file for these coordinates (lng=-76.2002669, lat=43.0402873). The problem is that the current implementation expects the response to be always json when the response.status >= 400. However, a 404 response returns some html instead.

It should be noted that I was not able to replicate the error again (running the same code with the above coordinates returns the weather file). Some possible explanations:

  • I checked the API calls that happen when getting the weather file. A successful response from https://gds-hsds-nsrdb-files.nrelcloud.org usually has the header x-amz-expiration: expiry-date="Thu, 29 Jul 2021 00:00:00 GMT", rule-id="24 hour retention" on it. The error I'm reporting happened at Jul 17, 2021 12:33:53 AM UTC which seems to be somewhat close to when those files are usually set to expire.
  • some temporary issue happened at the time of the request that made the file unavailable.

Even though the issue can't be replicated, I think it's worth it to fix it.

Additional Info

The API call in question that returned a 404 instead of the expected csv file 
GET https://gds-hsds-nsrdb-files.nrelcloud.org/data/solar/1171700_43.05_-76.22_tmy-2020_86eaf692167c07fd77a315d87871e274.csv
PySAM expected the response to be json while the response was actually some html 
<html>
<head>
  <title>404 Not Found</title>
</head>
<body>
  <h1>404 Not Found</h1>
  <ul>
    <li>Code: NoSuchKey</li>
    <li>Message: The specified key does not exist.</li>
    <li>Key: data/solar/1171700_43.05_-76.22_tmy-2020_86eaf692167c07fd77a315d87871e274.csv</li>
    <li>RequestId: ...</li>
    <li>HostId: ...</li>
  </ul>
  <hr/>
</body>
</html>
The relevant part of the traceback 
  ...
  File "path/to/helpers.py", line 51, in get_weather_file_from_api
    fetcher = wtkfetcher.fetch(lon_lats)
  File "PySAM/ResourceTools.py", line 577, in fetch
    results.append(self.data_function(job))
  File "PySAM/ResourceTools.py", line 503, in _NSRDB_worker
    data_response_json = data_response.json()
  File "requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
The code used to get the weather file 
import tempfile

def get_weather_file_from_api(lng, lat) -> str:
    weather_files_tmp_dir = os.path.join(tempfile.gettempdir(), "pysam")
    wtkfetcher = tools.FetchResourceFiles(
        tech="solar",
        nrel_api_key="API_KEY",
        nrel_api_email="EMAIL",
        resource_dir=weather_files_tmp_dir,
    )
    lon_lats = [(lng, lat)]
    fetcher = wtkfetcher.fetch(lon_lats)
    return fetcher.resource_file_paths[0]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions