Skip to content

Live API v2#9215

Merged
dlarocque merged 5 commits intodl/livefrom
dl/live-v2
Aug 25, 2025
Merged

Live API v2#9215
dlarocque merged 5 commits intodl/livefrom
dl/live-v2

Conversation

@dlarocque
Copy link
Contributor

@dlarocque dlarocque commented Aug 19, 2025 •
edited
Loading

Adds startAudioConversation, a new helper function for creating real-time voice conversations with the model.

The existing LiveSession API is low-level and requires a lot of boilerplate to manage microphone audio and playback. This helper handles all that complexity, making it much easier for developers to add voice features to their apps.

  • Handles microphone permissions and captures audio.
  • Processes raw audio into the format required by the server.
  • Plays back the model's audio response smoothly, without gaps or clicks.
  • Supports user interruptions (talking over the model).
  • Integrates function calling via a simple handler.
  • Includes unit tests with mocks for the Web Audio API.

Sample

const liveSession = await model.connect();
let conversationController;

// This function must be called from within a click handler.
async function startConversation() {
  try {
    conversationController = await startAudioConversation(liveSession);
  } catch (e) {
    // Handle AI-specific errors
    if (e instanceof AIError) {
      console.error("AI Error:", e.message);
    }
    // Handle microphone permission and hardware errors
    else if (e instanceof DOMException) {
      console.error("Microphone Error:", e.message);
    }
    // Handle other unexpected errors
    else {
      console.error("An unexpected error occurred:", e);
    }
  }
}

// Later, to stop the conversation:
// if (conversationController) {
//   await conversationController.stop();
// }

@changeset-bot
Copy link

changeset-bot bot commented Aug 19, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: cb546f5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@dlarocque dlarocque changed the base branch from main to dl/live August 19, 2025 19:19
@google-oss-bot
Copy link
Collaborator

google-oss-bot commented Aug 20, 2025 •
edited
Loading

Size Report 1

Affected Products

  • @firebase/ai

    TypeBase (3bb3f36)Merge (21a9c42)Diff
    browser47.2 kB54.5 kB+7.25 kB (+15.3%)
    main50.6 kB58.0 kB+7.35 kB (+14.5%)
    module47.2 kB54.5 kB+7.25 kB (+15.3%)

  • firebase

    TypeBase (3bb3f36)Merge (21a9c42)Diff
    firebase-ai.js37.3 kB43.6 kB+6.33 kB (+17.0%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/11lcLqVPHI.html

@google-oss-bot
Copy link
Collaborator

google-oss-bot commented Aug 20, 2025 •
edited
Loading

Size Analysis Report 1

Affected Products

  • @firebase/ai

    • LiveGenerativeModel

      Size

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      size8.39 kB8.41 kB+26 B (+0.3%)
      size-with-ext-deps26.0 kB26.0 kB+26 B (+0.1%)

    • LiveSession

      Size

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      size4.92 kB4.95 kB+26 B (+0.5%)
      size-with-ext-deps22.5 kB22.5 kB+26 B (+0.1%)

    • ResponseModality

      Size

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      size2.47 kB2.49 kB+14 B (+0.6%)
      size-with-ext-deps20.0 kB20.0 kB+14 B (+0.1%)

    • getLiveGenerativeModel

      Size

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      size10.9 kB10.9 kB+26 B (+0.2%)
      size-with-ext-deps28.5 kB28.5 kB+26 B (+0.1%)

    • startAudioConversation

      Size

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      size?8.52 kB? (?)
      size-with-ext-deps?26.3 kB? (?)

      Dependency

      TypeBase (3bb3f36)Merge (21a9c42)Diff
      functions?

      decodeInstanceIdentifier
registerAI
startAudioConversation

      		?
      classes?

      AIError
AIService
AudioConversationRunner
Backend
GoogleAIBackend
VertexAIBackend

      		?
      variables?

      12 dependencies

      AIErrorCode
AI_TYPE
AUDIO_PROCESSOR_NAME
BackendType
DEFAULT_LOCATION
SERVER_INPUT_SAMPLE_RATE
SERVER_OUTPUT_SAMPLE_RATE
Task
audioProcessorWorkletString
logger
name
version

      		?
      enums??

      External Dependency

      ModuleBase (3bb3f36)Merge (21a9c42)Diff
      @firebase/app?

      _registerComponent
registerVersion

      		?
      @firebase/component?

      Component

      		?
      @firebase/logger?

      Logger

      		?
      @firebase/util?

      Deferred
FirebaseError

      		?

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/XPvItYvgtq.html

@dlarocque dlarocque marked this pull request as ready for review August 21, 2025 19:15
@dlarocque dlarocque requested review from a team as code owners August 21, 2025 19:15
hsubox76
hsubox76 reviewed Aug 21, 2025
View reviewed changes
hsubox76
hsubox76 reviewed Aug 21, 2025
View reviewed changes
@dlarocque dlarocque merged commit 923c6bb into dl/live Aug 25, 2025
41 of 45 checks passed
@dlarocque dlarocque deleted the dl/live-v2 branch August 25, 2025 17:20
@dlarocque dlarocque mentioned this pull request Aug 25, 2025
Merged
@firebase firebase locked and limited conversation to collaborators Sep 24, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@hsubox76 hsubox76 hsubox76 approved these changes

@rachelsaunders rachelsaunders rachelsaunders approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dlarocque @google-oss-bot @hsubox76 @rachelsaunders