Skip to content

feat: Onboard NOAA dataset#353

Merged
adlersantos merged 4 commits into
GoogleCloudPlatform:mainfrom
nlarge-google:noaa
Jun 10, 2022
Merged

feat: Onboard NOAA dataset#353
adlersantos merged 4 commits into
GoogleCloudPlatform:mainfrom
nlarge-google:noaa

Conversation

@nlarge-google

@nlarge-google nlarge-google commented May 5, 2022

Copy link
Copy Markdown
Contributor

Description

ghcnd_by_year
ghcnd_countries
ghcdn_inventory
ghcnd_states
ghcnd_stations
gsod_stations
ghcnd_hurricanes
lightning_strikes_by_year

Checklist

  • (Required) This pull request is appropriately labeled
  • Please merge this pull request after it's approved
  • I'm adding or editing a dataset
    • The Google Cloud Datasets team is aware of the proposed dataset
    • I put all my code inside datasets/noaa and nothing outside of that directory
  • I'm refactoring or cleaning up some code

Comment thread datasets/noaa/pipelines/_images/ghcnd_states_schema.json Outdated
Comment thread datasets/noaa/pipelines/_images/ghcnd_stations_schema.json Outdated
int_date_list: typing.List[str],
gen_location_list: dict,
) -> None:
if pipeline_name == "GHCND by year":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps "GHCND by year" can be defined as a constant at the top of this file (or better yet, read from a yaml file). Same goes for other similar cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What added value would that bring to the code? What would the variable be called?

Comment thread datasets/noaa/pipelines/_images/run_csv_transform_kub/csv_transform.py Outdated
Comment thread datasets/noaa/pipelines/_images/run_csv_transform_kub/csv_transform.py Outdated

@adlersantos adlersantos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just fix the indentation on the JSON files and we're good to go.

Comment thread datasets/noaa/pipelines/_images/ghcnd_inventory_schema.json Outdated
@nlarge-google

Copy link
Copy Markdown
Contributor Author

@adlersantos Fixes are complete as per your request. Testing showed one other issue which is now resolved. Please Review. Thanks!

@adlersantos adlersantos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

depends_on_past: False
start_date: '2021-03-01'
max_active_runs: 1
schedule_interval: "0 1 0 0 6"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment for clarity.

Suggested change
schedule_interval: "0 1 0 0 6"
schedule_interval: "0 1 0 0 6" # Every Saturday at 1am server time

@adlersantos adlersantos merged commit 0f1c696 into GoogleCloudPlatform:main Jun 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants