[Serverless] Add retries to get default Fleet Server URL by mrodm · Pull Request #2901 · elastic/elastic-package

mrodm · 2025-09-12T08:03:18Z

While creating Elastic stacks using the serverless provider, sometimes the elastic-package stack up command fails to find the Fleet Server URL. This can be observed specially in the CI daily jobs. For instance:

https://buildkite.com/elastic/elastic-package-test-serverless/builds/546#01992cd8-eb64-4fca-8aa8-c841c1f0e045

Considering that the Kibana Client already performs some retries in case of failures, this PR adds retries to the query that gets the default Fleet Server.

Example of this new wait process (buildkite link):

2025/09/12 08:58:17 DEBUG project <project> initialized
2025/09/12 08:58:18 DEBUG Fleet Server URL not found yet, retrying...
2025/09/12 08:58:20 DEBUG Fleet Server found
Elasticsearch host: ...

mrodm · 2025-09-12T08:03:42Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T08:06:38Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/557

mrodm · 2025-09-12T08:24:28Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T08:27:13Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/558

mrodm · 2025-09-12T08:52:03Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T08:54:43Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/559

mrodm · 2025-09-12T09:24:23Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T09:27:45Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/560

mrodm · 2025-09-12T10:10:33Z

test serverless

mrodm · 2025-09-12T10:13:05Z

internal/stack/serverless.go

+	fleetServerURL := ""
+	wait.UntilTrue(ctx, func(ctx context.Context) (bool, error) {
+		fleetServerURL, err = project.DefaultFleetServerURL(ctx, sp.kibanaClient)
+		if errors.Is(err, kibana.ErrFleetServerNotFound) {
+			logger.Debug("Fleet Server URL not found yet, retrying...")
+			return false, nil
+		}
+		if err != nil {
+			return false, err
+		}
+		logger.Debug("Fleet Server found")
+		return true, nil
+	}, sp.retriesDefaultFleetServerPeriod, sp.retriesDefaultFleetServerTimeout)
+	if fleetServerURL == "" {
 		return Config{}, fmt.Errorf("failed to get fleet URL: %w", err)
 	}


This kibana request could be a valid response , but it could not contain any default Fleet Server.
This is why I've added this loop to query a few times until getting the value if possible.

Should DefaultFleetServerURL return kibana.ErrFleetServerNotFound in this case?

DefaultFleetServerURL (from the Kibana Client code) already returns that error in that scenario:

elastic-package/internal/kibana/fleet.go

Line 71 in 3cbb744

return "", ErrFleetServerNotFound

I use that error in the retry logic to keep iterating until a fleet server URL is retrieved or until the timeout is reached

https://github.com/elastic/elastic-package/pull/2901/files#diff-b4efb1ac558d15d3077525cb9be59dce12ee723a4749253e9ca7cc860419f1f8R116

EDIT: kibana.ErrFleetServerNotFound is part of the error (wrapped) in project.DefaultFleetServerURL:

elastic-package/internal/serverless/project.go

Line 127 in 2868c22

return "", fmt.Errorf("failed to query fleet server hosts: %w", err)

elastic-vault-github-plugin-prod · 2025-09-12T10:15:09Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/561

jsoriano · 2025-09-12T15:47:07Z

internal/stack/serverless.go

+		logger.Debug("Fleet Server found")
+		return true, nil
+	}, sp.retriesDefaultFleetServerPeriod, sp.retriesDefaultFleetServerTimeout)
+	if fleetServerURL == "" {
 		return Config{}, fmt.Errorf("failed to get fleet URL: %w", err)
 	}


Should we include this check inside the retry logic? Or if an empty url is returned this is never solved?

Suggested change

logger.Debug("Fleet Server found")

return true, nil

}, sp.retriesDefaultFleetServerPeriod, sp.retriesDefaultFleetServerTimeout)

if fleetServerURL == "" {

return Config{}, fmt.Errorf("failed to get fleet URL: %w", err)

}

if fleetServerURL == "" {

return false, fmt.Errorf("failed to get fleet URL: %w", err)

}

logger.Debug("Fleet Server found")

return true, nil

}, sp.retriesDefaultFleetServerPeriod, sp.retriesDefaultFleetServerTimeout)

IIUC if it is returned an error by introducing the check inside the retry logic, this retry loop will stop without finding the actual fleet server:

elastic-package/internal/wait/wait.go

Line 23 in 3cbb744

return false, err

Here, in this retry loop it is likely that a valid response does not contain any fleet url, so we need to try in that scenario.

jsoriano · 2025-09-12T15:47:58Z

internal/stack/serverless.go

+	fleetServerURL := ""
+	wait.UntilTrue(ctx, func(ctx context.Context) (bool, error) {
+		fleetServerURL, err = project.DefaultFleetServerURL(ctx, sp.kibanaClient)
+		if errors.Is(err, kibana.ErrFleetServerNotFound) {
+			logger.Debug("Fleet Server URL not found yet, retrying...")
+			return false, nil
+		}
+		if err != nil {
+			return false, err
+		}
+		logger.Debug("Fleet Server found")
+		return true, nil
+	}, sp.retriesDefaultFleetServerPeriod, sp.retriesDefaultFleetServerTimeout)
+	if fleetServerURL == "" {
 		return Config{}, fmt.Errorf("failed to get fleet URL: %w", err)
 	}


Should DefaultFleetServerURL return kibana.ErrFleetServerNotFound in this case?

jsoriano · 2025-09-12T16:00:15Z

internal/stack/serverless.go

-	config.Parameters[ParamServerlessFleetURL], err = project.DefaultFleetServerURL(ctx, sp.kibanaClient)
-	if err != nil {
+	fleetServerURL := ""
+	wait.UntilTrue(ctx, func(ctx context.Context) (bool, error) {


We should check the error returned by wait.UntilTrue.

Right, I missed that check.
I'll add it.

Added in 91accb9

mrodm · 2025-09-12T16:12:07Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T16:15:07Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/562

mrodm · 2025-09-12T16:37:32Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-12T16:41:13Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/563

elasticmachine · 2025-09-12T16:55:16Z

💚 Build Succeeded

Buildkite Build
Commit: abaf7a9

History

💔 Build #6104 failed 91accb9
💚 Build #6102 succeeded 831ceca

cc @mrodm

mrodm · 2025-09-15T07:55:52Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-15T07:58:57Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/567

elastic-vault-github-plugin-prod · 2025-09-15T08:22:14Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/568

elastic-vault-github-plugin-prod · 2025-09-15T08:52:13Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/569

elastic-vault-github-plugin-prod · 2025-09-15T09:21:39Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/570

mrodm · 2025-09-15T09:44:21Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-15T09:48:20Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/573

mrodm · 2025-09-15T10:42:53Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-15T10:45:47Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/574

mrodm · 2025-09-15T11:10:51Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-15T11:15:32Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/575

mrodm · 2025-09-15T12:46:05Z

test serverless

elastic-vault-github-plugin-prod · 2025-09-15T12:49:00Z

Triggered serverless pipeline: https://buildkite.com/elastic/elastic-package-test-serverless/builds/576

Add retries to get default Fleet Server URL

0f8e157

mrodm self-assigned this Sep 12, 2025

mrodm changed the title ~~Add retries to get default Fleet Server URL~~ [Serverless]Add retries to get default Fleet Server URL Sep 12, 2025

mrodm changed the title ~~[Serverless]Add retries to get default Fleet Server URL~~ [Serverless] Add retries to get default Fleet Server URL Sep 12, 2025

mrodm mentioned this pull request Sep 12, 2025

Creating Serverless projects fail to find Fleet Server URL #2902

Closed

Add log messages

1d31b64

Add constants

831ceca

mrodm force-pushed the http_retry_serverless branch from 3ba63fc to 831ceca Compare September 12, 2025 09:24

mrodm commented Sep 12, 2025

View reviewed changes

mrodm marked this pull request as ready for review September 12, 2025 10:34

mrodm requested a review from a team September 12, 2025 10:34

jsoriano reviewed Sep 12, 2025

View reviewed changes

Check values returned by wait.UntilTrue

91accb9

Fix linting issue

abaf7a9

Remove unnecessary variable

83f3aa4

mrodm requested a review from jsoriano September 15, 2025 12:54

jsoriano approved these changes Sep 15, 2025

View reviewed changes

mrodm merged commit 4912d2f into elastic:main Sep 15, 2025
3 checks passed

mrodm mentioned this pull request Sep 15, 2025

[CI] Add retry to test serverless pipeline #2894

Closed

Conversation

mrodm commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

mrodm commented Sep 12, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 12, 2025

Uh oh!

elasticmachine commented Sep 12, 2025

💚 Build Succeeded

History

Uh oh!

mrodm commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

mrodm commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

mrodm commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

mrodm commented Sep 15, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 15, 2025

Uh oh!

mrodm commented Sep 15, 2025

Uh oh!

mrodm commented Sep 12, 2025 •

edited

Loading

mrodm Sep 12, 2025 •

edited

Loading