Skip to content

Validate that directory does not contain -#572

Merged
ruflin merged 2 commits intoelastic:masterfrom
ruflin:validate-directory-names
Jul 3, 2020
Merged

Validate that directory does not contain -#572
ruflin merged 2 commits intoelastic:masterfrom
ruflin:validate-directory-names

Conversation

@ruflin
Copy link
Copy Markdown
Collaborator

@ruflin ruflin commented Jun 30, 2020 •
edited
Loading

As we try to ensure no directories contain -, we can also check for this. This adds a check.

The additional NewPackage method with assets was removed and merged into NewPackage for now. This means NewPackage is a bit more heavy but we have all the validation in place. It also means, the memory consumption to keep all packages is a bit higher. For the number of packages we have today, this should be ok.

Instead I plan a future refactoring that serves the /search endpoint only from the info of the BasePackage. So not all package information with the assets has to be kept in memory.

This removes all the - assets in the packages.

@ruflin ruflin self-assigned this Jun 30, 2020
@ruflin ruflin mentioned this pull request Jun 30, 2020
Closed
29 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jun 30, 2020 •
edited
Loading

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-07-03T08:01:18.150+0000

  • Duration: 8 min 14 sec

Test stats 🧪

Test Results
Failed 18
Passed 145
Skipped 0
Total 163

Test errors

Expand to view the tests failures

  • Name: Test / TestPackageIndex – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestPackageIndex//package/example/1.0.0/ – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex – package-registry

    • Age: 12
    • Duration: 0.06
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/base/0.2.0 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/default_pipeline/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/example/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/example/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/multiple_false/0.0.1 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/yamlpipeline/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.02
    • Error Details: Failed

  • Name: TestIntegration / TestPackageIndex – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestPackageIndex//package/example/1.0.0/ – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex – package-registry

    • Age: 12
    • Duration: 0.04
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/base/0.2.0 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/default_pipeline/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/example/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/example/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/multiple_false/0.0.1 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/yamlpipeline/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

Log output

Expand to view the last 100 lines of log output 

[2020-07-03T08:09:15.206Z] Step 2/19 : FROM golang:${GO_VERSION} AS builder
[2020-07-03T08:09:15.206Z]  ---> 2421885b04da
[2020-07-03T08:09:15.206Z] Step 3/19 : ENV GO111MODULE=on
[2020-07-03T08:09:15.206Z]  ---> Using cache
[2020-07-03T08:09:15.206Z]  ---> 1f0cf3d53d83
[2020-07-03T08:09:15.206Z] Step 4/19 : COPY ./ /package-registry
[2020-07-03T08:09:15.806Z]  ---> 45d30ed9d1ac
[2020-07-03T08:09:15.806Z] Step 5/19 : WORKDIR /package-registry
[2020-07-03T08:09:16.085Z]  ---> Running in fa528ce13911
[2020-07-03T08:09:16.086Z] Removing intermediate container fa528ce13911
[2020-07-03T08:09:16.086Z]  ---> 596865516e36
[2020-07-03T08:09:16.086Z] Step 6/19 : RUN go build .
[2020-07-03T08:09:16.351Z]  ---> Running in 2a58a63df626
[2020-07-03T08:09:16.925Z] go: downloading gopkg.in/yaml.v2 v2.2.8
[2020-07-03T08:09:16.925Z] go: downloading github.com/Masterminds/semver/v3 v3.1.0
[2020-07-03T08:09:16.925Z] go: downloading github.com/gorilla/mux v1.7.4
[2020-07-03T08:09:16.925Z] go: downloading github.com/radovskyb/watcher v1.0.7
[2020-07-03T08:09:16.925Z] go: downloading github.com/aymerick/raymond v2.0.2+incompatible
[2020-07-03T08:09:16.925Z] go: downloading github.com/elastic/go-ucfg v0.8.4-0.20200415140258-1232bd4774a6
[2020-07-03T08:09:16.925Z] go: downloading github.com/pkg/errors v0.9.1
[2020-07-03T08:09:16.926Z] go: downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
[2020-07-03T08:09:21.197Z] Removing intermediate container 2a58a63df626
[2020-07-03T08:09:21.197Z]  ---> e8470e3dbfd1
[2020-07-03T08:09:21.197Z] Step 7/19 : FROM centos:7
[2020-07-03T08:09:21.197Z]  ---> b5b4d78bc90c
[2020-07-03T08:09:21.197Z] Step 8/19 : RUN yum install -y zip rsync && yum clean all
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> e42f56f81e7c
[2020-07-03T08:09:21.197Z] Step 9/19 : COPY --from=builder /package-registry/package-registry /package-registry/package-registry
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> cf7d0d9af5db
[2020-07-03T08:09:21.197Z] Step 10/19 : WORKDIR /package-registry
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> 3d988467c62e
[2020-07-03T08:09:21.197Z] Step 11/19 : COPY config.docker.yml /package-registry/config.yml
[2020-07-03T08:09:21.197Z]  ---> 19229b23bad7
[2020-07-03T08:09:21.197Z] Step 12/19 : EXPOSE 8080
[2020-07-03T08:09:21.197Z]  ---> Running in c1eabea83fcd
[2020-07-03T08:09:21.197Z] Removing intermediate container c1eabea83fcd
[2020-07-03T08:09:21.197Z]  ---> 5cf17b1c6f41
[2020-07-03T08:09:21.197Z] Step 13/19 : ENTRYPOINT ["./package-registry"]
[2020-07-03T08:09:21.197Z]  ---> Running in 265940b90cd2
[2020-07-03T08:09:21.463Z] Removing intermediate container 265940b90cd2
[2020-07-03T08:09:21.463Z]  ---> 472b08bbde1c
[2020-07-03T08:09:21.463Z] Step 14/19 : CMD ["--address=0.0.0.0:8080"]
[2020-07-03T08:09:21.463Z]  ---> Running in 46f3de4b3f73
[2020-07-03T08:09:21.725Z] Removing intermediate container 46f3de4b3f73
[2020-07-03T08:09:21.726Z]  ---> 3cc9a2520781
[2020-07-03T08:09:21.726Z] Step 15/19 : HEALTHCHECK --interval=1s --retries=30 CMD curl --silent --fail localhost:8080/health || exit 1
[2020-07-03T08:09:21.726Z]  ---> Running in a0cbebc5fca6
[2020-07-03T08:09:21.992Z] Removing intermediate container a0cbebc5fca6
[2020-07-03T08:09:21.992Z]  ---> b45c60f67acf
[2020-07-03T08:09:21.992Z] Step 16/19 : LABEL BRANCH_NAME=PR-572
[2020-07-03T08:09:21.992Z]  ---> Running in a22f3195757a
[2020-07-03T08:09:22.253Z] Removing intermediate container a22f3195757a
[2020-07-03T08:09:22.253Z]  ---> afd1f84ae529
[2020-07-03T08:09:22.253Z] Step 17/19 : LABEL GIT_SHA=5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:22.253Z]  ---> Running in 518254c41cfa
[2020-07-03T08:09:22.253Z] Removing intermediate container 518254c41cfa
[2020-07-03T08:09:22.253Z]  ---> 51b94e8c0b1f
[2020-07-03T08:09:22.253Z] Step 18/19 : LABEL GO_VERSION=1.13.0
[2020-07-03T08:09:22.518Z]  ---> Running in d44cca9a7e3a
[2020-07-03T08:09:22.518Z] Removing intermediate container d44cca9a7e3a
[2020-07-03T08:09:22.518Z]  ---> 69db0dbb3437
[2020-07-03T08:09:22.518Z] Step 19/19 : LABEL TIMESTAMP=2020-07-03_08:09
[2020-07-03T08:09:22.518Z]  ---> Running in 1b43da3fa6fd
[2020-07-03T08:09:22.791Z] Removing intermediate container 1b43da3fa6fd
[2020-07-03T08:09:22.791Z]  ---> 8abd0621b1f3
[2020-07-03T08:09:22.791Z] Successfully built 8abd0621b1f3
[2020-07-03T08:09:22.791Z] Successfully tagged docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:23.158Z] + docker push docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:23.158Z] The push refers to repository [docker.elastic.co/observability-ci/package-registry]
[2020-07-03T08:09:23.158Z] 538443ead664: Preparing
[2020-07-03T08:09:23.158Z] 5389b0f5ba5e: Preparing
[2020-07-03T08:09:23.158Z] 62df25568c70: Preparing
[2020-07-03T08:09:23.158Z] edf3aa290fb3: Preparing
[2020-07-03T08:09:23.424Z] edf3aa290fb3: Layer already exists
[2020-07-03T08:09:25.371Z] 538443ead664: Pushed
[2020-07-03T08:09:26.347Z] 5389b0f5ba5e: Pushed
[2020-07-03T08:09:26.935Z] 62df25568c70: Pushed
[2020-07-03T08:09:29.499Z] 5bf6104468ff4f08439655ba20eea1c1cc2f923c: digest: sha256:55adfe3508be8ee3c233a567d851c378fc200b2dc268498e7ec61b713e714c7e size: 1158
[2020-07-03T08:09:29.821Z] + docker tag docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c docker.elastic.co/observability-ci/package-registry:PR-572
[2020-07-03T08:09:30.137Z] + docker push docker.elastic.co/observability-ci/package-registry:PR-572
[2020-07-03T08:09:30.137Z] The push refers to repository [docker.elastic.co/observability-ci/package-registry]
[2020-07-03T08:09:30.137Z] 538443ead664: Preparing
[2020-07-03T08:09:30.137Z] 5389b0f5ba5e: Preparing
[2020-07-03T08:09:30.137Z] 62df25568c70: Preparing
[2020-07-03T08:09:30.137Z] edf3aa290fb3: Preparing
[2020-07-03T08:09:30.398Z] edf3aa290fb3: Layer already exists
[2020-07-03T08:09:30.399Z] 5389b0f5ba5e: Layer already exists
[2020-07-03T08:09:30.399Z] 538443ead664: Layer already exists
[2020-07-03T08:09:30.399Z] 62df25568c70: Layer already exists
[2020-07-03T08:09:30.974Z] PR-572: digest: sha256:55adfe3508be8ee3c233a567d851c378fc200b2dc268498e7ec61b713e714c7e size: 1158
[2020-07-03T08:09:31.467Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_package-registry_PR-572
[2020-07-03T08:09:31.504Z] [INFO] getVaultSecret: Getting secrets
[2020-07-03T08:09:31.606Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-03T08:09:32.241Z] + chmod 755 generate-build-data.sh
[2020-07-03T08:09:32.241Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12 UNSTABLE 493825
[2020-07-03T08:09:32.241Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12/steps/?limit=10000 -o steps-info.json
[2020-07-03T08:09:32.492Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12/tests/?status=FAILED -o tests-errors.json

@ruflin
Validate that directory do not contain -
0d856cd 
As we try to ensure no directories contain -, we can also check for this. This adds a check.

The additional NewPackage method with assets was removed and merged into NewPackage for now. This means NewPackage is a bit more heavy but we have all the validation in place. It also means, the memory consumption to keep all packages is a bit higher. For the number of packages we have today, this should be ok.

Instead I plan a future refactoring that serves the /search endpoint only from the info of the BasePackage. So not all package information with the assets has to be kept in memory.

This removes all the - assets in the packages.
@ruflin ruflin force-pushed the validate-directory-names branch from 5bf6104 to 0d856cd Compare July 3, 2020 08:26
@ruflin ruflin marked this pull request as ready for review July 3, 2020 08:27
@ruflin ruflin requested a review from mtojek July 3, 2020 08:27
mtojek
mtojek approved these changes Jul 3, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: does not contain (commit message)

@ruflin ruflin merged commit a4caba7 into elastic:master Jul 3, 2020
@ruflin ruflin changed the title Validate that directory do not contain - Validate that directory does not contain - Jul 3, 2020
@ruflin
Copy link
Copy Markdown
Collaborator Author

ruflin commented Jul 3, 2020

Fixed commit message and merged ;-)

@ruflin ruflin deleted the validate-directory-names branch July 3, 2020 08:52
This was referenced Jul 14, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@mtojek mtojek mtojek approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@ruflin ruflin

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ruflin @elasticmachine @mtojek