…index (#55530)

Related PRs on `main`: 
- [55180](https://github.com/sourcegraph/sourcegraph/pull/55180):
Implements new feature
- [55528](https://github.com/sourcegraph/sourcegraph/pull/55528):
Updates migration metadata to use leaves that are present on 5.1 instead
of using a leaves only available on main.

Steps for this PR: 

1. cherry-picked
[55180](https://github.com/sourcegraph/sourcegraph/pull/55180) commit
2. updated `metadata.yaml` with 5.1's resulting value from running `sg
migration leaves`
3. ran `sg generate` to confirm no other changes required

Description:

When a text input is submitted for generating embeddings the response
may be null. If we attempt retries and still cannot generate embeddings
for this input text then we return an error which calls for failing the
entire embed repo job.

[Slack
thread](https://sourcegraph.slack.com/archives/C053L1AQ0BC/p1688676751106069)

https://github.com/sourcegraph/sourcegraph/issues/55469

This PR introduces a configuration ExcludeChunkOnError. When set to true
an embed repo job will proceed with the rest of the embed repo job when
these generate embeddings errors occur. However, the file that generated
the input text which received an error is excluded from the index as to
avoid partially indexing the file.

I'll add more details on the first iteration of this solution and the
trade offs in a separate comment.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

new tests for embed.go and embedding clients

…anch (#55528)

Update exclude chunks migration metadata to align with 5.1 release
branch. This is to support backporting this
[PR](https://github.com/sourcegraph/sourcegraph/pull/55180) to 5.1.

Parent `1687792857` exists for both 5.1 and main. Alignment refers to
following the migration steps
[here](https://handbook.sourcegraph.com/departments/engineering/dev/tools/backport/#prs-with-migration-changes).

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

n/a

…anch (#55528)

Update exclude chunks migration metadata to align with 5.1 release
branch. This is to support backporting this
[PR](https://github.com/sourcegraph/sourcegraph/pull/55180) to 5.1.

Parent `1687792857` exists for both 5.1 and main. Alignment refers to
following the migration steps
[here](https://handbook.sourcegraph.com/departments/engineering/dev/tools/backport/#prs-with-migration-changes).

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

n/a