Mobile Telemetry Processor

We're still figuring out the spec for the mobile telemetry processor. The telemetry buffer is already an approved concept, but not the rest of the processor.

Telemetry Buffer

For the common specification for the telemetry buffer, refer to the Telemetry Processor page. This page describes the mobile-specific implementation of the telemetry buffer. The most important difference is that the mobile telemetry buffer is designed to minimize data loss when sudden process terminations occur, such as crashes or watchdog terminations.

The common requirements specify data forwarding scenarios, which mobile SDKs MUST implement. Additionally, mobile SDKs MUST adhere to this extra data forwarding scenario:

• When the application moves to the background, the TelemetryBuffer SHOULD forward all data in memory to the transport and stop the timer. The transport SHOULD keep its existing behavior, which usually stores the data to disk as an envelope. It is not required to call transport flush.

Each SDK environment is unique. Therefore, SDKs have three options to choose from to minimize data loss. As their number increases, the options get more complex. The first option is the simplest, and the last option is the most complicated. SDKs SHOULD implement the least complex option that is suitable for their environment.

1. Flush All Data

When the SDK detects a sudden process termination, it MUST put all remaining items in the telemetry buffer into one envelope and flush it. If your SDK has an offline cache, it MAY flush the envelope to disk and skip sending it to Sentry, if it ensures to send the envelope the next time the SDK starts. The telemetry buffer MUST keep its existing logic described in the Telemetry Processor Specification page.

Suppose your SDK can't reliably detect sudden process terminations, or it can't reliably flush envelopes to Sentry or disk when a sudden process termination happens. In that case, it SHOULD implement the FileStream Cache or the DoubleRotatingBuffer. It's acceptable to start with this option as a best effort interim solution before adding one of the more complex options.

2. FileStream Cache

SDKs for which blocking the main thread is a nogo, such as Android and Apple, SDKs MUST NOT implement this option. They SHOULD implement the DoubleRotatingBuffer.

With this option, the telemetry buffer stores the data on the calling thread directly to disk. The SDK SHOULD store the telemetry buffer files in a folder that is a sibling of the envelopes or replay folder, named telemetry-buffer. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. In the telemetry-buffer folder, the SDK MUST store two types of cache files:

• cache - The file the processor is actively writing to
• flushing - The file being converted to an envelope and sent to Sentry

When the timeout expires or the cache file hits the size limit, the telemetry buffer renames the cache file to flushing, creates a new cache file for incoming data, converts the data in the flushing file to an envelope, sends it to Sentry, and then deletes the flushing file. When the SDK starts again, it MUST check if there are any cache files in the cache directory (both cache and flushing) and if so, it MUST load the data from the files and send it to Sentry.

3. DoubleRotatingBuffer

SDKs SHOULD only consider implementing this option when options 1 or 2 are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or consistently store envelopes to disk during such terminations.

The telemetry buffer uses two buffers to minimize data loss in the event of an abnormal process termination:

• Crash-Safe List: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations.
• Async IO Cache: When a process terminates without the SDK being able to detect it, the crash-safe list loses all its elements. Therefore, the telemetry buffer uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread, which ensures minimal data loss when such terminations occur.

As the telemetry buffer MUST prevent data loss during flushing, it uses a double-buffering solution. The crash-safe list has two lists crash-safe-list-1 and crash-safe-list-2, and the async IO cache has two files async-io-cache-1 and async-io-cache-2. When crash-safe-list-1 is full, the telemetry buffer stores any new incoming items in crash-safe-list-2 until it successfully stores items from crash-safe-list-1 to disk as an envelope. Then it can delete items in crash-safe-list-1. The same applies to the async IO cache.

Telemetry Buffer Files

The SDK SHOULD store the telemetry buffer files in a folder that is a sibling of the envelopes or replay folder, named telemetry-buffer. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The telemetry-buffer folder MAY contain the following files:

• async-io-cache-1 and async-io-cache-2 - The async IO cache files.
• detected-termination-x - The file containing items from the crash-safe list from a previous detected abnormal termination.
• envelope-x - The envelope that the telemetry buffer is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where x is the an increasing index of the file starting from 0.

Receiving Items

The telemetry buffer has two lists crash-safe-list-1 and crash-safe-list-2 and two files async-io-cache-1 and async-io-cache-2. When it receives items, it performs the following steps:

1. Put the item into the crash-safe crash-safe-list-1 on the calling thread.
2. On a background thread, store the item in the async-io-cache-1.

Flushing

When the crash-safe-list1 exceeds the above described 1MiB in size or the timeout exceeds, the telemetry buffer performs the following flushing steps:

1. Store new incoming items to the crash-safe-list-2 and async-io-cache-2.
2. Put the items of crash-safe-list-1 into an envelope named envelope-x.
3. Delete the items in crash-safe-list-1 and async-io-cache-1.
4. Move the envelope-x to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry.

The telemetry buffer stores the envelope-x not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items crash-safe-list-1 and async-io-cache-1, the SDKs might send duplicate items.

Abnormal Process Termination

When SDKs detect an abnormal process termination, they MUST write the items in both crash-safe-list-1 and crash-safe-list-2 to the detected-termination-x file where x is the an increasing index of the file starting from 0.

When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items from the crash-safe lists, which we consider preferable to blocking the calling thread, which could be the main thread. However, the SDKs don't lose items from the async IO cache.

So the SDK MAY lose items due to undetectable abnormal process terminations that occur immediately after receiving an item, but it won't lose items due to detectable abnormal process terminations.

// SDKs keep this log
Sentry.logger.info("We might crash now.");

crash();

SDK Initialization

Whenever the SDKs initialize, they MUST check if there is any data in the telemetry buffer folder that needs to be recovered. They MUST perform the following steps when initializing:

1. Load all items from async-io-cache-1, async-io-cache-2 and detected-termination-x into memory if they exist. When the application terminates normally, these files don't exist.
2. Deduplicate the items based on the IDs of the items and store the deduplicated items in the envelope-x file.
3. Create new async-io-cache-1 and async-io-cache-2 files, and delete the detected-termination-x file.
4. Now the telemetry buffer can start receiving new items.
5. Move the envelope-x to the envelopes cache folder.

SDK Closes

Whenever the users closes the SDK or the application terminates normally, the telemetry buffer MUST perform the steps described in the Flushing section and the SDK MUST delete all items in the async-io-cache-1 and async-io-cache-2 files.

Miscellaneous

The telemetry buffer maintains its logic of batching multiple logs and spans together into a single envelope to avoid multiple HTTP requests.

Hybrid SDKs pass every log and span down to the native SDKs, which will put every log and span in their telemetry buffer and its cache when logs and spans are ready for sending, meaning after they go through beforeLog, integrations, processors, etc.