Addresses issues with race conditions encountered when observing invalid ownerReferences

Fixes #65200

• address review comments
• add comments to each commit on the scenario being addresses
• add unit tests for GC controller and graph builder fixes
• ensure the coverage of the following testcases are represented:

• child in namespace A with owner reference to namespaced type in namespace B

  • should be deleted immediately
  • event should be logged in namespace A with involvedObject of bad-child indicating the error
  • test both orders:
    • real parent added to graph, then bad child added to graph
    • bad child added to graph, virtual parent added to graph, then real parent added to graph

• child that is cluster-scoped with owner reference to namespaced type in namespace B

  • should not be deleted
  • event should be logged in namespace kube-system with involvedObject of bad-child indicating the error
  • test both orders:
    • real parent added to graph, then bad child added to graph
    • bad child added to graph, virtual parent added to graph, then real parent added to graph

• child pointing at non-preferred still-served apiVersion of parent object (e.g. rbac/v1beta1)

  • should not be deleted prematurely
  • should not repeatedly poll attemptToDelete while waiting
  • should be deleted when the actual parent is deleted

• child pointing at no-longer-served apiVersion of still-existing parent object (e.g. extensions/v1beta1 deployment)

  • should not be deleted (this is indistinguishable from referencing an unknown kind/version)
  • should not repeatedly poll attemptToDelete once real parent is observed
  • TODO: follow up: should be fixed up by a controller adjusting known ownerReference API versions

• child pointing at no-longer-served apiVersion of no-longer-existing parent object (e.g. extensions/v1beta1 deployment)

  • should not be deleted (this is indistinguishable from referencing an unknown kind/version)
  • should repeatedly poll attemptToDelete
  • should not block deletion of legitimate children of missing deployment
  • TODO: follow up: should be fixed up by a controller adjusting known ownerReference API versions and once fixed up, should be deleted since parent does not exist

• child pointing at incorrect apiVersion/kind of still-existing parent object (e.g. core/v1 Secret with uid=123, where an apps/v1 Deployment with uid=123 exists)

  • should be deleted immediately
  • should not trigger deletion of legitimate children of parent

Follow-ups:

• add a controller/mechanism to migrate apiVersions of ownerReferences for known kubernetes types that have graduated or moved API groups (this is a pre-existing problem) - tracked in Add migration of ownerReferences that refer to deprecated/no-longer-served Kubernetes API versions #96650
• before 1.20 release - write a tool that scans items in the cluster and identifies invalid ownerReferences that would be impacted by this change, update the release note to describe running that tool on a 1.19 cluster to determine if there is existing bad data - https://github.com/kubernetes-sigs/kubectl-check-ownerreferences

/kind bug
/cc @jpbetz @deads2k
/sig api-machinery

Resolves non-deterministic behavior of the garbage collection controller when ownerReferences with incorrect data are encountered. Events with a reason of `OwnerRefInvalidNamespace` are recorded when namespace mismatches between child and owner objects are detected. The [kubectl-check-ownerreferences](https://github.com/kubernetes-sigs/kubectl-check-ownerreferences) tool can be run prior to upgrading to locate existing objects with invalid ownerReferences.
* A namespaced object with an ownerReference referencing a uid of a namespaced kind which does not exist in the same namespace is now consistently treated as though that owner does not exist, and the child object is deleted.
* A cluster-scoped object with an ownerReference referencing a uid of a namespaced kind is now consistently treated as though that owner is not resolvable, and the child object is ignored by the garbage collector.