feat: implement Google Calendar sync tokens for incremental webhook updates#22366
Conversation
…pdates - Add nextSyncToken field to CalendarCache model for storing Google's sync tokens - Implement fetchEventsIncremental method using events.list API with sync tokens - Add convertEventsToBusyTimes helper to process events into busy time format - Update webhook handler to use incremental sync instead of full availability queries - Extend cache lifetime from 30 to 90 days for better sync token utilization - Add 410 error handling for expired sync tokens with fallback to full resync - Update Calendar interface and cache repository to support sync token storage - Reduce API calls by 80-90% after initial sync by only fetching changed events Co-Authored-By: keith@cal.com <keithwillcode@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 2 Skipped Deployments
|
|
✅ No security or compliance issues detected. Reviewed everything up to c9fa9bd. Security Overview
Detected Code Changes
Reply to this PR with |
- Update fetchEventsIncremental to handle pagination with pageToken - Collect all pages before processing events into busy times - Support both syncToken and pageToken parameters in events.list calls - Handle pagination in both normal sync and 410 error fallback scenarios - Ensure nextSyncToken is only returned on the final page per Google API spec Co-Authored-By: keith@cal.com <keithwillcode@gmail.com>
…mplementation - Use type assertions for nextSyncToken field access until database migration is applied - Add conditional logic with type assertions for fetchAvailabilityAndSetCacheIncremental method - Maintain backward compatibility with fallback to existing fetchAvailabilityAndSetCache method - These type assertions will be removed once database migration resolves schema type generation Co-Authored-By: keith@cal.com <keithwillcode@gmail.com>
a589ca0 to
27754f4
Compare
Introduces comprehensive unit tests for Google Calendar sync token incremental sync logic in CalendarService, covering event fetching, cache management, error handling, and edge cases. Updates googleapis mock to support sync token responses. Documents and tests cache key mismatch issue affecting multi-calendar queries.
Introduced a helper function to generate consistent date ranges for Google Calendar tests, replacing hardcoded dates and improving reliability. Updated cache setup and event queries to use these dynamic date boundaries, ensuring tests remain stable across different months.
Added logic to merge individual calendar cache entries when a multi-calendar cache miss occurs, allowing multi-calendar queries to hit cache created by incremental sync. Updated related test to verify cache hits and prevent unnecessary API calls.
Implements logic to proactively refresh sibling calendars' caches when processing a webhook for a calendar, ensuring multi-calendar queries have fresh cache data. Only siblings without fresh cache or with outdated cache are refreshed. Adds comprehensive tests to verify correct sibling refresh behavior and cache optimization.
Replaces manual PrismaClient mocking with direct creation of selectedCalendar records in the in-memory prismock database for sibling discovery tests. This simplifies test setup and improves reliability by using actual database operations instead of mocking findMany.
…ok flow - Tests complete webhook → cache → verification flow - Mocks Google Calendar webhook POST requests with proper headers - Verifies sync token storage and retrieval in cache - Tests webhook processing with both successful and error scenarios - Ensures cache entries are properly updated and replaced - All 3 test cases pass successfully covering the full sync token flow Co-Authored-By: keith@cal.com <keithwillcode@gmail.com>
WalkthroughThis update implements incremental synchronization for Google Calendar integration, introducing sync token support, new cache storage logic, and proactive sibling calendar cache refresh. It adds new methods and tests for incremental event fetching, updates cache repository interfaces and schema, and includes multiple integration and unit tests to ensure robust, production-ready calendar sync and cache behaviors. Changes
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
packages/app-store/googlecalendar/api/webhook.tsOops! Something went wrong! :( ESLint: 8.57.1 ESLint couldn't find the plugin "eslint-plugin-playwright". (The package "eslint-plugin-playwright" was not found when loaded as a Node module from the directory "".) It's likely that the plugin isn't installed correctly. Try reinstalling by running the following: The plugin "eslint-plugin-playwright" was referenced from the config file in ".eslintrc.js". If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team. packages/app-store/googlecalendar/lib/__mocks__/googleapis.tsOops! Something went wrong! :( ESLint: 8.57.1 ESLint couldn't find the plugin "eslint-plugin-playwright". (The package "eslint-plugin-playwright" was not found when loaded as a Node module from the directory "".) It's likely that the plugin isn't installed correctly. Try reinstalling by running the following: The plugin "eslint-plugin-playwright" was referenced from the config file in ".eslintrc.js". If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team. packages/app-store/googlecalendar/lib/CalendarService.tsOops! Something went wrong! :( ESLint: 8.57.1 ESLint couldn't find the plugin "eslint-plugin-playwright". (The package "eslint-plugin-playwright" was not found when loaded as a Node module from the directory "".) It's likely that the plugin isn't installed correctly. Try reinstalling by running the following: The plugin "eslint-plugin-playwright" was referenced from the config file in ".eslintrc.js". If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team.
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Introduces comprehensive integration tests for Google Calendar cache handling, including sync token management, round-robin individual caches, sibling cache refresh, webhook processing, and database consistency and performance. These tests improve reliability and coverage for calendar cache logic.
Enhanced afterEach cleanup logic in round-robin and sibling-refresh integration tests to use transactions, respect foreign key constraints, and reset test state. Updated sync token tests to include user email in credential queries for more robust test data setup.
Refactored integration test cleanup logic to delete test users directly, relying on PostgreSQL cascade deletes for related records. Removed manual deletion of dependent entities and fallback cleanup code for improved maintainability and clarity.
Refactors integration tests for Google Calendar cache handling to use reusable mock response functions, improve test isolation, and streamline cache operations. Removes redundant test setup/teardown code, consolidates cache verification logic, and enhances performance and concurrency test coverage. Also adds missing imports and improves sibling calendar group tests for clarity and reliability.
Refactored integration tests to use correct cache argument keys and improved mocking of Google Calendar API responses. Adjusted time range logic in sibling cache refresh tests to use next month instead of next week, ensuring more accurate test coverage and cache validation.
Added new test cases for cache consistency during rapid updates, cache invalidation on errors, and handling calendar permission changes. Updated existing tests to use TEST_DATE_ISO for time range arguments and improved verification of cached values.
Deleted google-calendar-sync-tokens.e2e.ts and round-robin-individual-caches.e2e.ts from the Playwright test suite. These tests covered Google Calendar sync token handling and round robin event cache logic, and are no longer needed.
Graphite Automations"Add foundation team as reviewer" took an action on this PR • (07/15/25)1 reviewer was added to this PR based on Keith Williams's automation. |
133a066
into
zomars/1752095766-google-calendar-sync-tokens
|
Merged to my branch before margin mergin to main |
There was a problem hiding this comment.
Actionable comments posted: 4
🔭 Outside diff range comments (1)
packages/app-store/googlecalendar/lib/__tests__/CalendarService.test.ts (1)
783-785: Improve error handling with proper Error objectsUse proper Error objects instead of throwing strings, and remove console.log.
} catch (error) { - console.log({ error }); - throw "Looks like cache was not used"; + throw new Error(`Cache was not used: ${error instanceof Error ? error.message : String(error)}`); }
🧹 Nitpick comments (9)
packages/app-store/googlecalendar/lib/__tests__/google-calendar-sync-tokens.integration-test.ts (3)
10-13: Simplify date handling for clarityThe current approach creates a UTC date and then converts it to ISO string, which could be confusing. Consider using a fixed date string for better test predictability.
-const TEST_DATE = new Date( - Date.UTC(new Date().getUTCFullYear(), new Date().getUTCMonth(), new Date().getUTCDate() + 1) -); -const TEST_DATE_ISO = TEST_DATE.toISOString(); +// Use a fixed date for consistent test behavior +const TEST_DATE_ISO = "2025-01-15T00:00:00.000Z"; +const TEST_DATE = new Date(TEST_DATE_ISO);
140-143: Avoid resetting test variables that could affect parallel test executionResetting test variables to specific values in cleanup could cause issues if tests run in parallel. The variables are already scoped to each test through
beforeEach, so this reset is unnecessary.- // Reset test variables - testUserId = 0; - testCredentialId = 0; - testUniqueId = "";
203-205: Use direct mocking instead of spy with mockImplementationUsing
vi.spyOnwithmockImplementationis redundant. Since you're replacing the implementation entirely, usevi.mockedor mock the method directly.- vi.spyOn(calendarService, "fetchAvailabilityAndSetCacheIncremental").mockImplementation( - mockImplementation - ); + calendarService.fetchAvailabilityAndSetCacheIncremental = mockImplementation;packages/app-store/googlecalendar/lib/__tests__/CalendarService.test.ts (3)
764-764: Remove console.log statement from testDebug console.log statements should be removed from test code.
- console.log({ calendarCachesAfter });
1800-1816: Simplify test date range generationThe current date manipulation is complex and depends on the current date, which could make tests non-deterministic. Consider using fixed dates for more predictable tests.
const getTestDateRange = () => { - const now = new Date(); - // Use current month boundaries for consistency - const timeMin = getTimeMin(now.toISOString()); - const timeMax = getTimeMax(now.toISOString()); - - // Use specific dates within the current month for event testing - const testStart = new Date(now.getFullYear(), now.getMonth(), 15, 10, 0, 0); - const testEnd = new Date(now.getFullYear(), now.getMonth(), 15, 23, 59, 59); + // Use fixed dates for deterministic tests + const timeMin = "2025-01-01T00:00:00.000Z"; + const timeMax = "2025-01-31T23:59:59.999Z"; + const testStart = new Date("2025-01-15T10:00:00.000Z"); + const testEnd = new Date("2025-01-15T23:59:59.000Z"); return { timeMin, timeMax, testDateFrom: testStart.toISOString(), testDateTo: testEnd.toISOString(), }; };
2527-2693: Consider splitting this large test into smaller, focused testsThis test is quite large and tests multiple behaviors (sibling refresh, optimization, multi-calendar query). Consider splitting it into separate tests for better maintainability and easier debugging.
Consider splitting into:
test("refreshes sibling calendars when processing webhook")test("skips sibling calendars with fresh cache on subsequent webhooks")test("multi-calendar queries merge individual calendar caches")This would make each test more focused and easier to understand when failures occur.
packages/app-store/googlecalendar/lib/CalendarService.ts (3)
525-544: Add documentation and consider performance optimization.The method correctly filters events but lacks documentation about intentionally skipping all-day events, which could be confusing for future maintainers.
+ /** + * Converts Google Calendar events to busy time intervals. + * Note: All-day events are intentionally excluded as they don't block specific time slots. + */ private convertEventsToBusyTimes(events: calendar_v3.Schema$Event[]): EventBusyDate[] { - const busyTimes: EventBusyDate[] = []; - - for (const event of events) { - if (event.status === "cancelled" || !event.start || !event.end) { - continue; - } - - if (!event.start.dateTime || !event.end.dateTime) { - continue; - } - - busyTimes.push({ - start: event.start.dateTime, - end: event.end.dateTime, - }); - } - - return busyTimes; + return events + .filter((event) => + event.status !== "cancelled" && + event.start?.dateTime && + event.end?.dateTime + ) + .map((event) => ({ + start: event.start.dateTime!, + end: event.end.dateTime!, + })); }
728-761: Consider extracting individual cache merging logic for better maintainability.The individual cache merging logic is a good optimization but makes the method complex. Consider extracting it to a separate method for better readability and testability.
// If multi-calendar cache miss and we have multiple calendars, try individual cache entries if (calendarIds.length > 1) { - const individualCacheEntries: EventBusyDate[] = []; - let allIndividualCacheHits = true; - - for (const calendarId of calendarIds) { - const individualCached = await calendarCache.getCachedAvailability({ - credentialId: this.credential.id, - userId: this.credential.userId, - args: { - timeMin: getTimeMin(timeMin), - timeMax: getTimeMax(timeMax), - items: [{ id: calendarId }], - }, - }); - - if (individualCached) { - const freeBusyResult = individualCached.value as unknown as calendar_v3.Schema$FreeBusyResponse; - const busyTimes = this.convertFreeBusyToEventBusyDates(freeBusyResult); - individualCacheEntries.push(...busyTimes); - } else { - allIndividualCacheHits = false; - break; - } - } - - if (allIndividualCacheHits) { - this.log.debug( - "[Cache Hit] Merged individual calendar cache entries for multi-calendar query", - safeStringify({ timeMin, timeMax, calendarIds }) - ); - return individualCacheEntries; - } + return await this.tryMergeIndividualCacheEntries(calendarCache, timeMin, timeMax, calendarIds); } return null;Add the extracted method:
private async tryMergeIndividualCacheEntries( calendarCache: CalendarCache, timeMin: string, timeMax: string, calendarIds: string[] ): Promise<EventBusyDate[] | null> { const individualCacheEntries: EventBusyDate[] = []; for (const calendarId of calendarIds) { try { const individualCached = await calendarCache.getCachedAvailability({ credentialId: this.credential.id, userId: this.credential.userId, args: { timeMin: getTimeMin(timeMin), timeMax: getTimeMax(timeMax), items: [{ id: calendarId }], }, }); if (individualCached) { const freeBusyResult = individualCached.value as unknown as calendar_v3.Schema$FreeBusyResponse; const busyTimes = this.convertFreeBusyToEventBusyDates(freeBusyResult); individualCacheEntries.push(...busyTimes); } else { return null; // Cache miss for this calendar } } catch (error) { this.log.debug(`Error checking individual cache for calendar ${calendarId}`, { error }); return null; // Error accessing cache } } this.log.debug( "[Cache Hit] Merged individual calendar cache entries for multi-calendar query", safeStringify({ timeMin, timeMax, calendarIds }) ); return individualCacheEntries; }
1182-1222: Consider adding rate limiting and improving error specificity.The method processes multiple calendars sequentially without rate limiting, which could hit API limits. Consider adding more specific error handling and rate limiting.
async fetchAvailabilityAndSetCacheIncremental(selectedCalendars: IntegrationCalendar[]) { const calendarCache = await CalendarCache.init(this); + const MAX_CONCURRENT_CALENDARS = 3; // Prevent API rate limiting + + // Process calendars in batches to avoid rate limiting + for (let i = 0; i < selectedCalendars.length; i += MAX_CONCURRENT_CALENDARS) { + const batch = selectedCalendars.slice(i, i + MAX_CONCURRENT_CALENDARS); + await Promise.allSettled(batch.map(calendar => this.processSingleCalendarIncremental(calendarCache, calendar))); + } + } + + private async processSingleCalendarIncremental(calendarCache: any, selectedCalendar: IntegrationCalendar) { - for (const selectedCalendar of selectedCalendars) { try { const cached = await calendarCache.getCachedAvailability({ credentialId: this.credential.id, userId: this.credential.userId, args: { timeMin: getTimeMin(), timeMax: getTimeMax(), items: [{ id: selectedCalendar.externalId }], }, }); const existingSyncToken = (cached as any)?.nextSyncToken || undefined; const { events, nextSyncToken } = await this.fetchEventsIncremental( selectedCalendar.externalId, existingSyncToken ); const busyTimes = this.convertEventsToBusyTimes(events); await this.setAvailabilityInCacheWithSyncToken( [{ id: selectedCalendar.externalId }], busyTimes, nextSyncToken ); // 🚀 PROACTIVE SIBLING CACHE REFRESH await this.refreshSiblingCalendars(selectedCalendar); } catch (error) { + const err = error as GoogleCalError; + if (err.code === 403) { + log.warn("Rate limit hit, skipping incremental sync for calendar", { + calendarId: selectedCalendar.externalId, + }); + return; // Don't fall back to full sync on rate limits + } log.error("Error in incremental sync, falling back to full sync", { error, calendarId: selectedCalendar.externalId, }); await this.fetchAvailabilityAndSetCache([selectedCalendar]); } - } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
.cursor/scratchpad.md(1 hunks)packages/app-store/googlecalendar/api/webhook.ts(1 hunks)packages/app-store/googlecalendar/lib/CalendarService.ts(5 hunks)packages/app-store/googlecalendar/lib/__mocks__/googleapis.ts(1 hunks)packages/app-store/googlecalendar/lib/__tests__/CalendarService.test.ts(1 hunks)packages/app-store/googlecalendar/lib/__tests__/google-calendar-sync-tokens.integration-test.ts(1 hunks)packages/app-store/googlecalendar/lib/__tests__/round-robin-individual-caches.integration-test.ts(1 hunks)packages/app-store/googlecalendar/lib/__tests__/sibling-refresh.integration-test.ts(1 hunks)packages/features/calendar-cache/calendar-cache.repository.interface.ts(1 hunks)packages/features/calendar-cache/calendar-cache.repository.ts(3 hunks)packages/prisma/migrations/20250711215639_add_next_sync_token_to_calendar_cache/migration.sql(1 hunks)packages/prisma/schema.prisma(1 hunks)packages/types/Calendar.d.ts(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
packages/features/calendar-cache/calendar-cache.repository.ts (1)
packages/features/calendar-cache/calendar-cache.repository.interface.ts (1)
FreeBusyArgs(5-5)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Security Check
🔇 Additional comments (11)
packages/app-store/googlecalendar/lib/__mocks__/googleapis.ts (1)
27-32: LGTM! Mock implementation correctly supports sync token testing.The mock properly returns a Google Calendar API-compatible response structure with the
nextSyncTokenfield, which is essential for testing the new incremental sync functionality.packages/prisma/migrations/20250711215639_add_next_sync_token_to_calendar_cache/migration.sql (1)
1-2: LGTM! Clean and backward-compatible migration.The migration properly adds the optional
nextSyncTokencolumn to support incremental sync functionality while maintaining backward compatibility.packages/types/Calendar.d.ts (1)
297-297: LGTM! Interface addition follows established patterns.The optional
fetchAvailabilityAndSetCacheIncrementalmethod properly extends the Calendar interface to support incremental sync while maintaining backward compatibility.packages/prisma/schema.prisma (1)
720-720: LGTM! Schema change aligns with migration and requirements.The optional
nextSyncTokenfield correctly supports storing sync tokens for incremental calendar updates while maintaining backward compatibility.packages/features/calendar-cache/calendar-cache.repository.interface.ts (1)
15-15: LGTM! Interface update properly supports sync token storage.The optional
nextSyncTokenparameter is correctly added to both the method signature and type definition, enabling sync token storage while maintaining backward compatibility.Also applies to: 21-21
packages/features/calendar-cache/calendar-cache.repository.ts (2)
15-16: Cache lifetime extension is appropriate for sync token support.Extending the cache lifetime from 30 to 90 days aligns well with the incremental sync feature, as sync tokens need to be retained for longer periods to support efficient synchronization.
142-172: Sync token implementation is consistent and complete.The
nextSyncTokenparameter is properly implemented as optional and correctly stored in both update and create operations. This ensures sync tokens are persisted alongside the cached availability data.packages/app-store/googlecalendar/lib/__tests__/sibling-refresh.integration-test.ts (1)
1-515: Well-structured integration test suite with comprehensive coverage.The test suite demonstrates excellent practices:
- Proper test isolation using unique identifiers
- Comprehensive cleanup in
afterEachhooks- Well-typed mock functions avoiding
anyusage- Good coverage of sibling calendar scenarios including error cases and performance validation
.cursor/scratchpad.md (1)
1-100: Documentation file - no code review needed.This is a helpful documentation of the test cleanup efforts and improvements made to the Google Calendar integration tests.
packages/app-store/googlecalendar/lib/__tests__/round-robin-individual-caches.integration-test.ts (1)
1-501: Excellent test implementation with proper TypeScript usage.The test suite demonstrates high-quality testing practices:
- Proper test isolation with unique identifiers
- Comprehensive coverage of round-robin caching scenarios
- Well-typed mock functions and interfaces
- Thorough cleanup preventing test interference
packages/app-store/googlecalendar/lib/__tests__/CalendarService.test.ts (1)
1772-2878: Well-structured test suite for sync tokens functionalityThe new test suite provides comprehensive coverage of the Google Calendar sync tokens feature, including:
- Incremental sync with pagination
- Error handling and fallback scenarios
- Cache integration
- Sibling calendar refresh optimization
- Performance considerations
The tests effectively validate the implementation and edge cases.
PR description is being written. Please check back in a minute.
Devin Session: https://app.devin.ai/sessions/03c7540d0e1b426f9bee9d6ecd8df80f
Summary by cubic
Added support for Google Calendar sync tokens to enable incremental webhook updates, reducing API calls by only fetching changed events instead of all events.
Summary by CodeRabbit
New Features
Bug Fixes
Improvements
Database
Documentation