Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

chore: Add tests documenting error invariants#62992

Merged
varungandhi-src merged 8 commits into
mainfrom
vg/err-invariants
Jun 3, 2024
Merged

chore: Add tests documenting error invariants#62992
varungandhi-src merged 8 commits into
mainfrom
vg/err-invariants

Conversation

@varungandhi-src

@varungandhi-src varungandhi-src commented May 31, 2024

Copy link
Copy Markdown
Contributor

Based on the discussion in Slack, I thought it would be useful to solidify the invariants that hold (and provide counter-examples for things which you think ought to hold but don't actually hold) in code itself, instead of just via docs. So this PR adds property-based tests for quickly generating different kinds of error shapes, and check which invariants hold and which ones don't.

TODO:

  • Add tests for the various types of errors
  • Update docs for the various functions Is, As and HasType.
  • Add more helpful descriptive text for tests

Test plan

Adds new tests

@cla-bot cla-bot Bot added the cla-signed label May 31, 2024
@github-actions github-actions Bot added team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) team/product-platform labels May 31, 2024

@bobheadxi bobheadxi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL about rapid 👁️

I think this is a nice-to-have demonstration, but personally I feel like reading this would not strictly help me understand the error behaviours better. Could we include some more verbose docstrings that:

  1. Explain what invariants_test.go is for in the first place
  2. Explain each of the helpers in detail
  3. Explain the behaviours being tested better ("payloadLessStruct" doesn't seem very explanatory to me)

It might be nice to incorporate coverage of our existing error constructs (like our custom multi-error) as well

@varungandhi-src

Copy link
Copy Markdown
Contributor Author

Could we include some more verbose docstrings that:

I think some of this will make more sense once I add tests with the other types. But yes, I'll add more descriptions.

It might be nice to incorporate coverage of our existing error constructs (like our custom multi-error) as well

multiError is already incorporated into the tree structure (see errorTree and the counter-examples)

Comment thread lib/errors/invariants_test.go Outdated
Comment on lines 49 to 220

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Does this behavior actually makes sense? Based on my understanding of the expected behavior of HasType this seems wrong.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't want to ignore the comment so just chiming in to say I am unsure 😅 IMO I don't really understand the use case for HasType; if there isn't one, I'm inclined to suggest we do not export it from lib/errors and discourage its use, to keep things simple

@varungandhi-src varungandhi-src May 31, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, it seems like the implementation of HasType is wrong. It ought to instead be something along the lines of:

  typ := reflect.TypeOf(t)
  zero := reflect.Zero(typ)
  return errors.As(err, &zero)

Or perhaps, the bug is with our multiError type, not sure.

@varungandhi-src varungandhi-src May 31, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I tracked down the cause of this.

For a multi-error, we mark the last element as the Cause or result of Unwrap. code

func (e *multiError) Cause() error  { return e.errs[len(e.errs)-1] }
func (e *multiError) Unwrap() error { return e.errs[len(e.errs)-1] }

HasType calls If which calls UnwrapOnce inside its predicate function while doing a depth-first traversal, which has this:

func UnwrapOnce(err error) (cause error) {
	switch e := err.(type) {
	case interface{ Cause() error }:
		return e.Cause()
	case interface{ Unwrap() error }:
		return e.Unwrap()
	}
	return nil
}

Notice this comment:

// UnwrapOnce treats multi-errors (those implementing the
// `Unwrap() []error` interface as leaf-nodes since they cannot
// reasonably be iterated through to a single cause

This means that HasType will only compare against the last element in a multiError. 🤦🏽

In contrast, Is handles multi-errors explicitly, and As also handles multi-errors explicitly.

@varungandhi-src varungandhi-src May 31, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a slightly different API, here's an alternate implementation of HasType that doesn't suffer from the same problem. It seems like we should be able to tweak this using some comby-fu.

func HasType[T error](err error) bool {
	var zero T
	return As(err, &zero)
}

@varungandhi-src varungandhi-src May 31, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO I don't really understand the use case for HasType

The use case, as I understand it, is "is this error type present in the tree" without caring about the actual value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Branch fixing the implementation of HasType. https://github.com/sourcegraph/sourcegraph/tree/vg/replace-HasType

Will create a PR with that after merging this.

Comment thread lib/errors/.gitignore Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ignored because you want to be selective toward what gets committed here, amongst what it finds, or is that because we'll always intend this folder content to be ignored?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a more detailed comment here, does that answer your question?

# Generated by rapid after finding counter-examples.
#
# When re-running tests locally during development,
# this enables faster failure finding.
# https://github.com/flyingmutant/rapid/commit/4db076b4aa4d41036416da11abd5bbfc228a324e
#
# During development, if a check fails, it makes sense
# to disable or refine the check, and add specific,
# minimal tests using the counter-examples found by rapid.

Comment thread lib/errors/BUILD.bazel Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/sourcegraph/sourcegraph/pull/62992/files#r1622408899

If this is always going to be filled with examples generated locally, then it'll always end up empty in CI.

Additionally, if the folder doesn't exist, the glob eval will break in CI. Best to add a .gitkeep in there if you truly want to keep it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, not sure why this got added, I have no memory of modifying any BUILD files myself. I'll remove this and see if something breaks. AIUI, the only purpose is to record failures, so next time it can try them quickly.

flyingmutant/rapid@4db076b

However, in CI, the test won't be re-run, so it doesn't make sense to save the file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@varungandhi-src varungandhi-src requested a review from jhchabran May 31, 2024 17:40
@varungandhi-src

Copy link
Copy Markdown
Contributor Author

Going to merge this for now; can address other comments post-merge.

@varungandhi-src varungandhi-src merged commit 2c3e813 into main Jun 3, 2024
@varungandhi-src varungandhi-src deleted the vg/err-invariants branch June 3, 2024 11:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla-signed team/graph Graph Team (previously Code Intel/Language Tools/Language Platform) team/product-platform

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants