Skip to content

🏗✅ Add a presubmit check to help migrate top-level describe tests to describes.{realWin|integration}#24090

Merged
rsimha merged 1 commit intoampproject:masterfrom
rsimha:2019-08-16-TopLevelDescribe
Aug 22, 2019
Merged

🏗✅ Add a presubmit check to help migrate top-level describe tests to describes.{realWin|integration}#24090
rsimha merged 1 commit intoampproject:masterfrom
rsimha:2019-08-16-TopLevelDescribe

Conversation

@rsimha
Copy link
Copy Markdown
Contributor

@rsimha rsimha commented Aug 20, 2019

This is part of a larger effort to isolate AMP's tests and prevent global state from leaking across tests.

PR Highlights:

  • Adds a presubmit check to prevent future tests from being written with a top-level describe
  • Adds all existing use cases to a whitelist

Addresses #23839 (comment)
Partial fix for #23839
Partial fix for #23837
Partial fix for #15658
Partial fix for #14360

@lannka
Copy link
Copy Markdown
Contributor

lannka commented Aug 21, 2019

Could you explain your overall plan for the refactoring? realWin wraps them in friendly iframe, which means those tests can still access global document. So this single PR does not change that behavior. Also, some tests might choose describes.fakeWin or describes.sandbox depends on their own need.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Aug 21, 2019

Could you explain your overall plan for the refactoring? realWin wraps them in friendly iframe, which means those tests can still access global document. So this single PR does not change that behavior.

You're correct. The next step is to add a mechanism to describes to detect when a test touches global state, and either prevent it or clean it up.

Also, some tests might choose describes.fakeWin or describes.sandbox depends on their own need.

I'm going to leave that decision to test authors. Right now, all we're doing is converting legacy describe tests to use describes.realWin (for unit tests) or describes.integration (for integration tests). I've added a presubmit check to this PR so that future tests written with legacy describe will be flagged as an error, with a suggestion to use one of the describes types.

@rsimha rsimha changed the title ✅♻️ Wrap all top-level describe blocks in describes.realWin ✅♻️ Wrap all top-level describe blocks in describes.{realWin|integration} Aug 21, 2019
@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Aug 21, 2019

@lannka @jridgewell @choumx This PR is now ready for a full review.

@lannka
Copy link
Copy Markdown
Contributor

lannka commented Aug 21, 2019

The next step is to add a mechanism to describes to detect when a test touches global state, and either prevent it or clean it up.

Any idea how would you do that?

I've added a presubmit check to this PR so that future tests written with legacy describe will be flagged as an error, with a suggestion to use one of the describes types.

This sounds a right thing to do.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Aug 21, 2019

The next step is to add a mechanism to describes to detect when a test touches global state, and either prevent it or clean it up.

Any idea how would you do that?

I believe this can be done in setup() and teardown() in RealWinFixture and IntegrationFixture in testing/describes.js. I'll have to do some experimentation to see what state is currently being changed by tests.

@lannka
Copy link
Copy Markdown
Contributor

lannka commented Aug 21, 2019

I believe this can be done in setup() and teardown() in RealWinFixture and IntegrationFixture in testing/describes.js. I'll have to do some experimentation to see what state is currently being changed by tests.

It might be possible to detect those calls, but I don't see a straightforward way to automatically fix all those global access. That said, we will likely end up fixing tests one by one. If that's the case, I don't see too much value of wrapping those tests in realWin, which in fact makes the tests code a bit more unreadable.

To force all new tests to use describes, I guess you can create a whitelist of those legacy tests, and have the team to fix them one by one?

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Aug 22, 2019

I don't see too much value of wrapping those tests in realWin, which in fact makes the tests code a bit more unreadable.

Fair enough. I'll revert the test changes.

To force all new tests to use describes, I guess you can create a whitelist of those legacy tests, and have the team to fix them one by one?

Can do. I'll rework this PR to be a presubmit check instead, with a whitelist containing all the files that currently use describe.

It's theoretically possible to force anyone who touches a test file to also change describe to describes.*. But this will become annoying during large refactors. (We did this in the past for JSDoc errors, and it didn't go down well.)

@rsimha rsimha changed the title ✅♻️ Wrap all top-level describe blocks in describes.{realWin|integration} 🏗✅ Add a presubmit check to help migrate top-level describe tests to describes.{realWin|integration} Aug 22, 2019
@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Aug 22, 2019

@lannka Changed this to a presubmit check and whitelisted all existing uses. PTAL.

Copy link
Copy Markdown
Contributor

@lannka lannka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change!

@rsimha rsimha merged commit e5e055b into ampproject:master Aug 22, 2019
@rsimha rsimha deleted the 2019-08-16-TopLevelDescribe branch August 22, 2019 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants