Skip to content

Properly compare test files between jobs#17

Merged
jbhoosreddy merged 3 commits intobabel:masterfrom
jbhoosreddy:detect-circleci-master
Nov 9, 2019
Merged

Properly compare test files between jobs#17
jbhoosreddy merged 3 commits intobabel:masterfrom
jbhoosreddy:detect-circleci-master

Conversation

@jbhoosreddy
Copy link
Copy Markdown
Collaborator

@jbhoosreddy jbhoosreddy commented Nov 7, 2019

Summary of changes

  • With the help of the previous PR Prevent comparision on CircleCI master #16, I've been able to compare a master branch and my PR branch artifacts (by downloading them locally) and figuring out what are the remaining issues.
    • As you can see, it is correctly erroring on test failures my PR is causing.
    • I'm guessing the success values are coming from the fact that my babel PR is a month old and something else might have been fixed along the way. Now, I'll update my main PR and see if these errors remain.

But this is great progress.

➜ CIRCLE_BRANCH=PR/1234 node lib/compare-results master.tap test262.tap | $(npm bin)/tap-merge
TAP version 13
#  CIRCLECI = undefined
#  CIRCLE_BRANCH = PR/1234
#  is CircleCI master branch job: undefined
not ok 1 test/language/expressions/arrow-function/prototype-rules.js default # (expected success, got runtime error)
  ---
  name: Test262Error
  message: 'Expected SameValue(«true», «false») to be true'
  stack: |
    Test262Error: Expected SameValue(«true», «false») to be true
    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 2 test/language/expressions/arrow-function/prototype-rules.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: 'Expected SameValue(«true», «false») to be true'
  stack: |
    Test262Error: Expected SameValue(«true», «false») to be true
    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 3 test/language/expressions/arrow-function/throw-new.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: Expected a TypeError to be thrown but no exception was thrown at all
  stack: >
    Test262Error: Expected a TypeError to be thrown but no exception was thrown at
    all

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
not ok 4 test/language/expressions/arrow-function/throw-new.js default # (expected success, got runtime error)
  ---
  name: Test262Error
  message: Expected a TypeError to be thrown but no exception was thrown at all
  stack: >
    Test262Error: Expected a TypeError to be thrown but no exception was thrown at
    all

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
ok 5 test/language/literals/regexp/u-invalid-optional-lookbehind.js default # (parser error)

ok 6 test/language/literals/regexp/u-invalid-optional-lookbehind.js strict mode # (parser error)

ok 7 test/language/literals/regexp/u-invalid-optional-negative-lookbehind.js default # (parser error)

ok 8 test/language/literals/regexp/u-invalid-optional-negative-lookbehind.js strict mode # (parser error)

ok 9 test/language/literals/regexp/u-invalid-range-lookbehind.js default # (parser error)

ok 10 test/language/literals/regexp/u-invalid-range-lookbehind.js strict mode # (parser error)

ok 11 test/language/literals/regexp/u-invalid-range-negative-lookbehind.js default # (parser error)

ok 12 test/language/literals/regexp/u-invalid-range-negative-lookbehind.js strict mode # (parser error)

not ok 13 test/language/computed-property-names/class/method/symbol.js strict mode # (expected success, got runtime error)
  ---
  name: Test262Error
  message: >-
    `compareArray(Object.getOwnPropertyNames(C.prototype), ['constructor', 'a',
    'c'])` returns `true`
  stack: >
    Test262Error: `compareArray(Object.getOwnPropertyNames(C.prototype),
    ['constructor', 'a', 'c'])` returns `true`

    /home/circleci/babel/babel-test262-runner/lib/run-tests/index.js:62:11

  ...
ok 14 test/language/literals/regexp/named-groups/invalid-identity-escape-in-capture-u.js default # (parser error)

ok 15 test/language/literals/regexp/named-groups/invalid-identity-escape-in-capture-u.js strict mode # (parser error)

ok 16 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-2-u.js default # (parser error)

ok 17 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-2-u.js strict mode # (parser error)

ok 18 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-u.js default # (parser error)

ok 19 test/language/literals/regexp/named-groups/invalid-incomplete-groupname-u.js strict mode # (parser error)


0..0

@jbhoosreddy
Copy link
Copy Markdown
Collaborator Author

jbhoosreddy commented Nov 8, 2019

Working in babel/babel#10579

jbhoosreddy
jbhoosreddy commented Nov 8, 2019
View reviewed changes
lib/compare-results/index.js Outdated
`);
}

function getfileNameFromTitle(title) {
Copy link
Copy Markdown
Collaborator Author

@jbhoosreddy jbhoosreddy Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to get the consistent file name from this title
not ok 1 test/language/expressions/arrow-function/prototype-rules.js default # (expected success, got runtime error) ---

nicolo-ribaudo
nicolo-ribaudo reviewed Nov 8, 2019
View reviewed changes
@jbhoosreddy
Copy link
Copy Markdown
Collaborator Author

jbhoosreddy commented Nov 9, 2019

My last commit was driven by some weirdness I was noticing in test results:

"unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---*/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m"
 stack: "Consider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nSyntaxError: unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nagents.withAgent (/Users/jbhoosreddy/Projects/babel-test262-runner/lib/run-tests/index.js:62:11)\nSyntaxError: unknown: 'import' and 'export' may appear only with 'sourceType: \"module\"' (208:0)\nConsider renaming the file to '.mjs', or setting sourceType:module or sourceType:unambiguous in your Babel config for this file.\n\n\e[0m \e[90m 206 | \e[39m\e[32m---*/\e[39m\e[0m\n\e[0m \e[90m 207 | \e[39m\e[0m\n\e[0m\e[31m\e[1m>\e[22m\e[39m\e[90m 208 | \e[39m\e[32mimport './instn-uniq-env-rec-other_FIXTURE.js'\e[39m\e[0m\n\e[0m \e[90m     | \e[39m\e[31m\e[1m^\e[22m\e[39m\e[0m\n\e[0m \e[90m 209 | \e[39m\e[32mvar first = 1;\e[39m\e[0m\n\e[0m \e[90m 210 | \e[39m\e[32mlet second = 2;\e[39m\e[0m\n\e[0m \e[90m 211 | \e[39m\e[32mconst third = 3;\e[39m\e[0m\nParser.raise (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:6387:17)\nParser.assertModuleNodeAllowed (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10040:12)\nParser.parseStatementContent (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10011:16)\nParser.parseStatement (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:9900:17)\nParser.parseBlockOrModuleBlockBody (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10476:25)\nParser.parseBlockBody (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:10463:10)\nParser.parseTopLevel (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:9829:10)\nParser.parse (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:11341:17)\nparse (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/parser/lib/index.js:11377:38)\nparser (/Users/jbhoosreddy/Projects/babel-test262-runner/node_modules/@babel/core/lib/transformation/normalize-file.js:166:34)"

nicolo-ribaudo
nicolo-ribaudo approved these changes Nov 9, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@nicolo-ribaudo nicolo-ribaudo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, feel free to merge!

@jbhoosreddy jbhoosreddy merged commit b5eaa57 into babel:master Nov 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicolo-ribaudo nicolo-ribaudo nicolo-ribaudo approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jbhoosreddy @nicolo-ribaudo