What type of PR is this?

In order to support different scheduler as backends we modify Grove and import scheduler backend interface

What this PR does / why we need it:

In the current PodGang component's sync flow we do the following:

• Get the list of PodGangs that are expected to be created for the PCS.
• Check which ones are pending to be created. For each pending PodGang we do the following:
  • Check if all pods have been created for the PodGang.
  • Check if all pods have required PodGang label which adds back reference to the PodGang.
• If all the above checks are satisfied then it will go ahead and create the PodGang resource.
  So you can see PodGang will be created after pods. However, there is a problem with upcoming Workload API support and kube-scheduler backend.

We don't want break current PodGang working flow. We import this scheduler backend framework to leave the Workload management work to scheduler backend in Grove. For other scheduler, scheduler backend in Grove may manage different CR based on PodGang(Just like KAI, it will create PodGroups. In the future, we will move this management from KAI scheduler to Grove scheduler backend).

To create a Workload object, you will need to create PodGang resource. The PodGang resource cannot be created before the Pods have been created and have a back reference to the PodGang. The issue is that only after the Workload object is created will the kube-scheduler choose to run scoring/filtering plugins to reserve node capacity to schedule this workload PodGroups. The Pods need to have a reference to the Workload object in their spec.

So to accommodate Workload API the flow needs to be changed as below in the PodGang component:

• Create PodGang with PodGroups(having empty PodReferences as none will exist at this point) and Initialized condition set to False.
• Creation of PodGang will trigger the creation of the Workload object in the schedulerbackend reconciler which will use the kube scheduler backend.

  This is out of scope of this PR and should be included in the next PR which specifically handles the Workload APi and kube-scheduler.

• Once all Pod references are updated then set it to true
• Pods should not lift their scheduling gate till PodGang has Initialized condition set to True. - done in the PCLQ reconciler.

Which issue(s) this PR fixes:

Fixes #275
Fixes #445

Special notes for your reviewer:

Does this PR introduce a API change?

Yes. We will introduce a new API SchedulerBackend

type SchedulerBackend interface {
	// Name is a unique name of the scheduler backend.
	Name() string

	// Init provides a hook to initialize/setup one-time scheduler resources,
	// called at the startup of grove operator.
	Init() error

	// SyncPodGang synchronizes (creates/updates) scheduler specific resources for a PodGang
	// reacting to a creation or update of a PodGang resource.
	SyncPodGang(ctx context.Context, podGang *groveschedulerv1alpha1.PodGang) error

	// OnPodGangDelete cleans up scheduler specific resources for the given PodGang.
	OnPodGangDelete(ctx context.Context, podGang *groveschedulerv1alpha1.PodGang) error

	// PreparePod adds scheduler backend specific configuration to the given Pod object
	// prior to its creation. This includes setting schedulerName, scheduling gates,
	// annotations, etc.
	PreparePod(pod *corev1.Pod)
}

Additional documentation e.g., enhancement proposals, usage docs, etc.: