Skip to content

chore(): add telemetry event for auto mode routing fallback#3780

Merged
TylerLeonhardt merged 10 commits intomicrosoft:mainfrom
Sid200026:Sid200026/auto-router-telemetry-fallback
Feb 19, 2026
Merged

chore(): add telemetry event for auto mode routing fallback#3780
TylerLeonhardt merged 10 commits intomicrosoft:mainfrom
Sid200026:Sid200026/auto-router-telemetry-fallback

Conversation

@Sid200026
Copy link
Member

Added telemetry when the auto mode router errors out and fallback model is used.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds telemetry tracking when the auto mode router fails to return a valid model and the system falls back to using the reserved model. This helps monitor router reliability and understand when fallback behavior occurs.

Changes:

  • Added a new telemetry event automode.routerDecisionFallback with GDPR annotations to track router failures
  • The event captures available models, preferred models, and the chosen fallback model when routing fails
Comments suppressed due to low confidence (1)

src/platform/endpoint/node/automodeService.ts:273

  • The control flow logic has a critical bug. The code after the telemetry block (lines 262-272) will only execute when routing fails (inside the if (!selectedModel) block). However, when routing succeeds, selectedModel will be set and the function will skip all the remaining endpoint creation logic, causing it to reach line 273 without returning a value. This will result in a compilation error since the function is declared to return Promise<IChatEndpoint> but doesn't return anything in the success path. The code after line 261 (including the _applyVisionFallback call and endpoint creation) should be moved outside of the if (!selectedModel) block so it executes in both the fallback path and the success path.
			if (selectedModel && shouldRoute) {
				// If routing was attempted but failed, emit event that we are falling back to the reserved model
				{
					/* __GDPR__
						"automode.routerDecisionFallback" : {
							"owner": "tyleonha",
							"comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.",
							"availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" },
							"preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" }
							"chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" },
						}
					*/
					this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", {
						"availableModels": availableModels.join(','),
						"preferredModels": preferredModels.join(','),
						"chosenModel": selectedModel.model,
					})
				}
			}
			selectedModel = this._applyVisionFallback(chatRequest, selectedModel, reserveToken.available_models, knownEndpoints);

			// If the session token changed, invalidate all cached endpoints so they get recreated with the new token
			const existingEndpoints = (entry && entry.lastSessionToken === reserveToken.session_token) ? entry.endpoints : [];
			let autoEndpoint = existingEndpoints.find(e => e.model === selectedModel.model);
			if (!autoEndpoint) {
				autoEndpoint = this._instantiationService.createInstance(AutoChatEndpoint, selectedModel, reserveToken.session_token, reserveToken.discounted_costs?.[selectedModel.model] || 0, this._calculateDiscountRange(reserveToken.discounted_costs));
				existingEndpoints.push(autoEndpoint);
			}
			this._autoModelCache.set(conversationId, { endpoints: existingEndpoints, tokenBank: reserveTokenBank, lastSessionToken: reserveToken.session_token, lastRoutedPrompt: prompt });
			return autoEndpoint;
		}

Comment on lines +243 to +260
if (selectedModel && shouldRoute) {
// If routing was attempted but failed, emit event that we are falling back to the reserved model
{
/* __GDPR__
"automode.routerDecisionFallback" : {
"owner": "tyleonha",
"comment": "Reports a fallback event when the router fails to return a valid model and we have to fall back to the reserved model.",
"availableModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of available models for this request" },
"preferredModels": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "Comma-separated list of preferred models for this request, ordered by preference with the reserved model first" }
"chosenModel": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "comment": "The model selected by the router" },
}
*/
this._telemetryService.sendMSFTTelemetryEvent("automode.routerDecisionFallback", {
"availableModels": availableModels.join(','),
"preferredModels": preferredModels.join(','),
"chosenModel": selectedModel.model,
})
}
Copy link

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new telemetry for router fallback lacks test coverage. Consider adding a test case in automodeService.spec.ts that verifies the telemetry event is emitted when the router call fails (e.g., throws an error or returns an invalid model) and the system falls back to the reserved model. This would ensure the fallback telemetry is properly captured and that the code path works correctly.

Copilot uses AI. Check for mistakes.
Sid200026 and others added 7 commits February 17, 2026 02:32
@TylerLeonhardt
Copy link
Member

@Sid200026 aren't you more interested in error telemetry in this case?

@vs-code-engineering vs-code-engineering bot added this to the February 2026 milestone Feb 19, 2026
@TylerLeonhardt TylerLeonhardt added this pull request to the merge queue Feb 19, 2026
Merged via the queue into microsoft:main with commit 8f0a331 Feb 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants