feat: add scaffold script for directory + dataset.yaml setup#412

Merged

adlersantos merged 7 commits into

GoogleCloudPlatform:mainfrom

alick-at-google:generate-pipeline-files-script

Jul 29, 2022

alick-at-google commented Jul 12, 2022 •

edited by adlersantos

Loading

Contributor

Description

Note: If you are adding or editing a dataset, please specify the dataset folder involved, e.g. datasets/google_trends

Checklist

(Required) This pull request is appropriately labeled
Please merge this pull request after it's approved

Use the sections below based on what's applicable to your PR and delete the rest:

Feature

I'm adding or editing a feature
I have added/revised tests for the feature

alick-at-google added 2 commits

July 11, 2022 21:39


          feat: add generate pipeline directory + dataset.yaml script

156d3d6


          feat: add in pipeline yaml function v1

49f3e84

adlersantos reviewed

View reviewed changes

scripts/generate_pipeline_files.py Outdated

adlersantos reviewed

View reviewed changes

scripts/generate_pipeline_files.py Outdated

adlersantos reviewed

View reviewed changes

scripts/generate_pipeline_files.py Outdated

adlersantos commented Jul 14, 2022

Member

When generating the resulting files, please add the license header:

public-datasets-pipelines/scripts/generate_dag.py

Lines 191 to 194 in 492b973

    
           def write_to_file(contents: str, filepath: pathlib.Path): 
        
               license_header = pathlib.Path(TEMPLATE_PATHS["license"]).read_text() + "\n" 
        
               with open(filepath, "w") as file_: 
        
                   file_.write(license_header + contents.replace(license_header, ""))

adlersantos reviewed

View reviewed changes

scripts/generate_pipeline_files.py Outdated

adlersantos reviewed

View reviewed changes

scripts/generate_pipeline_files.py Outdated

adlersantos commented Jul 14, 2022

Member

This is looking good for a 1st version! 👍🏼 Time to add tests.

alick-at-google added 2 commits

July 22, 2022 11:38


          fix: code review changes + add test

9080e14


          fix: updates from working sesion 7/22

310e696

adlersantos requested changes

View reviewed changes

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

scripts/generate_pipeline_files.py Outdated

tests/scripts/test_generate_pipeline.py Outdated

tests/scripts/test_generate_pipeline.py Outdated

scripts/generate_pipeline_files.py Outdated

adlersantos added the feature request label

alick-at-google added 3 commits

July 29, 2022 11:01


          fix: code review updates

817913c


          fix: update script name + reformat code

a34e8a7


          update tests + remove unused package

b56a0cf

adlersantos changed the title ~~feat: add generate pipeline directory + dataset.yaml script~~ feat: add scaffold script for directory + dataset.yaml setup

adlersantos approved these changes

View reviewed changes

adlersantos left a comment

Member

Great work!

adlersantos merged commit 5bf354b into GoogleCloudPlatform:main

release-please Bot mentioned this pull request

chore(main): release 5.1.0 #410

Merged

vijay-google pushed a commit to vijay-google/public-datasets-pipelines that referenced this pull request


          feat: add scaffold script for directory + dataset.yaml setup (GoogleC…

3ce4ea6

…loudPlatform#412)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature request