📜 Description

TL;DR: Reduces the time required to render a session replay frame from ~160ms to ~30-36ms (benchmarked on an iPhone 8)

• Implements a custom graphics image renderer.
• Adds a new experimental feature enableExperimentalViewRenderer to choose the view renderer (enables our graphics renderer)
• Adds a new experimental feature enableFastViewRendering to choose the view draw method (switches from UIKit to CoreGraphics rendering, might be incomplete)
• Adds an experimental graphics image renderer
• Adds a view renderer using the experimental graphics image renderer
• Adds a mask renderer using the experimental graphics image renderer

💡 Motivation and Context

See #4000 for more information for the main issue of frame performance.

To understand the changes, we need to look at the implementation of the SentryDefaultViewRenderer used by the SentryViewPhotographer:

@objcMembers
class SentryDefaultViewRenderer: NSObject, SentryViewRenderer {
    func render(view: UIView) -> UIImage {
        // START BLOCK - Setup
        let image = UIGraphicsImageRenderer(size: view.bounds.size).image { context in
            // START BLOCK - Draw
            view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
            // END BLOCK - Draw
        }
        // END BLOCK - Setup
        return image
    }
}

The rendering consists of two nested blocks:

• Setup: to create a rendering context which is then used to convert rendered bitmap data into an in image
• Draw: which is drawing the view hierarchy into the bitmap context

Both need to be analysed for their performance and potential improvement candidates.

In addition we need to reconsider how the resolutions and coordinate system works:

• View has a size of a width and height in points (i.e. 375pt x 667pt for iPhone 8)
• Windows have a scale at which the UI should be rendered (i.e. 2.0 for iPhone 8)
• Screens have a resolution of size * scale pixels (i.e. 750px x 1334px for iPhone 8)

Therefore I also analyzed the impact of creating a scaled vs unscaled image, because the difference can be seen visually not only due to changes in resolution, but also due to blurriness:

Example - iOS-Swift at scale 1.0:

Example - iOS-Swift at scale 2.0:

The drawHierarchy performance has also been discussed in the Apple Developer Forums, leading to ReplayKit as an alternative (TBD)

💚 How did you test it?

I tried to create unit tests to test the view photographer using a complex UI built in a Storyboard file. I was not able to get it working, because unit tests have no render server, therefore trying to render a view to an image fails with the following error:

Rendering a view (0x105223a80, UILayoutContainerView) that has not been committed to render server is not supported.

Instead I performed manual testing by adapting the SentryViewPhotographer to use this method, then looking at the values over time when running the sample iOS-Swift on an iPhone 8:

func image(view: UIView, onComplete: @escaping ScreenshotCallback) {
    let viewSize = view.bounds.size

    let startTime = DispatchTime.now().uptimeNanoseconds
    let startRedactTime = DispatchTime.now().uptimeNanoseconds
    let redact = redactBuilder.redactRegionsFor(view: view)
    let endRedactTime = DispatchTime.now().uptimeNanoseconds

    // The render method is synchronous and must be called on the main thread.
    // This is because the render method accesses the view hierarchy which is managed from the main thread.
    let startRenderTime = DispatchTime.now().uptimeNanoseconds
    let renderedScreenshot = renderer.render(view: view)
    let endRenderTime = DispatchTime.now().uptimeNanoseconds
    let endTime = DispatchTime.now().uptimeNanoseconds

    printSummary(
        redactDiff: endRedactTime - startRedactTime,
        renderDiff: endRenderTime - startRenderTime,
        totalDiff: endTime - startTime
    )

    dispatchQueue.dispatchAsync { [maskRenderer] in
        // The mask renderer does not need to be on the main thread.
        // Moving it to a background thread to avoid blocking the main thread, therefore reducing the performance
        // impact/lag of the user interface.
        let maskedScreenshot = maskRenderer.maskScreenshot(screenshot: renderedScreenshot, size: viewSize, masking: redact)
        onComplete(maskedScreenshot)
    }
}

private let sampleLimit = 120
private var redactDiffHistory = [UInt64]()
private var renderDiffHistory = [UInt64]()
private var totalDiffHistory = [UInt64]()

func printSummary(redactDiff: UInt64, renderDiff: UInt64, totalDiff: UInt64) {
    redactDiffHistory = (redactDiffHistory + [redactDiff]).suffix(sampleLimit)
    renderDiffHistory = (renderDiffHistory + [renderDiff]).suffix(sampleLimit)
    totalDiffHistory = (totalDiffHistory + [totalDiff]).suffix(sampleLimit)

    let redactDiffHistoryMin = redactDiffHistory.min() ?? 0
    let redactDiffHistoryMax = redactDiffHistory.max() ?? 0
    let redactDiffHistoryAverage = Double(redactDiffHistory.reduce(0, +)) / Double(max(redactDiffHistory.count, 1))
    let sortedRedactDiffHistory = redactDiffHistory.sorted()
    let redactDiffHistoryP50 = sortedRedactDiffHistory[redactDiffHistory.count / 2]
    let redactDiffHistoryP75 = sortedRedactDiffHistory[Int(Double(redactDiffHistory.count) * 0.75)]
    let redactDiffHistoryP95 = sortedRedactDiffHistory[Int(Double(redactDiffHistory.count) * 0.95)]
    let redactDiffHistoryLast = redactDiffHistory.last ?? 0

    let renderDiffHistoryMin = renderDiffHistory.min() ?? 0
    let renderDiffHistoryMax = renderDiffHistory.max() ?? 0
    let renderDiffHistoryAverage = Double(renderDiffHistory.reduce(0, +)) / Double(max(renderDiffHistory.count, 1))
    let sortedRenderDiffHistory = renderDiffHistory.sorted()
    let renderDiffHistoryP50 = sortedRenderDiffHistory[renderDiffHistory.count / 2]
    let renderDiffHistoryP75 = sortedRenderDiffHistory[Int(Double(renderDiffHistory.count) * 0.75)]
    let renderDiffHistoryP95 = sortedRenderDiffHistory[Int(Double(renderDiffHistory.count) * 0.95)]
    let renderDiffHistoryLast = renderDiffHistory.last ?? 0

    let totalDiffHistoryMin = totalDiffHistory.min() ?? 0
    let totalDiffHistoryMax = totalDiffHistory.max() ?? 0
    let totalDiffHistoryAverage = Double(totalDiffHistory.reduce(0, +)) / Double(max(totalDiffHistory.count, 1))
    let sortedTotalDiffHistory = totalDiffHistory.sorted()
    let totalDiffHistoryP50 = sortedTotalDiffHistory[totalDiffHistory.count / 2]
    let totalDiffHistoryP75 = sortedTotalDiffHistory[Int(Double(totalDiffHistory.count) * 0.75)]
    let totalDiffHistoryP95 = sortedTotalDiffHistory[Int(Double(totalDiffHistory.count) * 0.95)]
    let totalDiffHistoryLast = totalDiffHistory.last ?? 0
    
    func f(_ value: UInt64) -> String {
        String(format: "%8.4f ms", Double(value) / 1_000_000.0)
    }

    func f(_ value: Double) -> String {
        String(format: "%8.4f ms", Double(value) / 1_000_000.0)
    }

    let samples = String(format: "%4.i Samples", redactDiffHistory.count)
    print("| \(samples) | Redact      | Render      | Total       |")
    print("|--------------|-------------|-------------|-------------|")
    print("| Min          | \(f(redactDiffHistoryMin)) | \(f(renderDiffHistoryMin)) | \(f(totalDiffHistoryMin)) |")
    print("| Max          | \(f(redactDiffHistoryMax)) | \(f(renderDiffHistoryMax)) | \(f(totalDiffHistoryMax)) |")
    print("| Avg          | \(f(redactDiffHistoryAverage)) | \(f(renderDiffHistoryAverage)) | \(f(totalDiffHistoryAverage)) |")
    print("| p50          | \(f(redactDiffHistoryP50)) | \(f(renderDiffHistoryP50)) | \(f(totalDiffHistoryP50)) |")
    print("| p75          | \(f(redactDiffHistoryP75)) | \(f(renderDiffHistoryP75)) | \(f(totalDiffHistoryP75)) |")
    print("| p95          | \(f(redactDiffHistoryP95)) | \(f(renderDiffHistoryP95)) | \(f(totalDiffHistoryP95)) |")
    print("| Last         | \(f(redactDiffHistoryLast)) | \(f(renderDiffHistoryLast)) | \(f(totalDiffHistoryLast)) |")
}

Baseline Performance

These are the results of using the currently released SentryViewPhotographer.image(view:onComplete:). We are not measuring the maskRenderer.maskScreenshot(...) because it is run on the background thread and therefore not blocking the main thread.

If a screen uses 60 frames per seconds to render graphics, the maximum time used to render one frame should be less than 1 / 60 = 16.667 ms. If a frame render takes longer, the next frame will be not be rendered (= a dropped frame) to catch up the lost time. This results in a stuttering user interface and visible to the user.

Results at screen scale (i.e. 2.0 for iPhone 8):

let image = UIGraphicsImageRenderer(size: view.bounds.size).image { _ in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

The data shows that we have a significant frame delay with up to 161.5842ms = ~9 frames dropped every second.

Results at scale 1.0:

let image = UIGraphicsImageRenderer(size: view.bounds.size, format: .init(for: .init(displayScale: 1))).image { _ in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

No significant changes compared to the native screen scale.

Alternative Attempts

Using UIView.snapshotView(afterScreenUpdates:)

The first attempt to optimize the performance was using the method UIView.snapshotView(afterScreenUpdates:) to create a dummy view hierarchy as stated in the documentation, to see if the dummy view would render faster than the normal one:

This method very efficiently captures the current rendered appearance of a view and uses it to build a new snapshot view. You can use the returned view as a visual stand-in for the current view in your app. [..] Because the content is captured from the already rendered content, this method reflects the current visual appearance of the view and is not updated to reflect animations that are scheduled or in progress. However, calling this method is faster than trying to render the contents of the current view into a bitmap image yourself.

During testing the snapshot view never displayed any visual data when rendered into an UIImage.

An Apple Engineer mentioned in the Apple Developer Forum that a snapshot can not be rendered to an image due to security reasons.

Results:

Discarded this approach because not possible.

Reuse the UIGraphicsImageRenderer

The documentation mentions that this renderer uses a built-in cache, therefore recommends to reuse renderer instances:

After initializing an image renderer, you can use it to draw multiple images with the same configuration. An image renderer keeps a cache of Core Graphics contexts, so reusing the same renderer can be more efficient than creating new renderers.

But the UIGraphicsImageRenderer is set to a fixed size and in our case the size of the view could eventually change.

To reduce the memory footprint I decided to have a maximum of one cached renderer, discarding and re-creating it whenever the size changes. In the worst case it has to re-create the renderer every time, yielding the same performance as without the cache. In the best case the size never changes and the renderer is reused forever.

Results:

There is no significant change compared to the baseline.

Experimental View Renderer

UIKit is built on top of CoreGraphics and CoreAnimation also known as the QuartzCore.

The UIGraphicsImageRenderer has been introduced with iOS 10.0 to wrap around the CGContext provided by CoreGraphics, as the setup can be tedious and complicated.

The SentryGraphicsImageRenderer is a custom implementation creating a CGContext pixel buffer and also converting it into an UIImage afterwards.

Results at native scale (i.e. 2.0):

let scale = (view as? UIWindow ?? view.window)?.screen.scale ?? 1
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

• Significant decrease in render time compared to base line performance.

Results at 1.0 scale:

let scale = 1.0
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

• Significant decrease in render time compared to base line performance.
• Slower than native scale, probably due to downscaling of internal graphics data

Replacing view.drawHierarchy(in:afterScreenUpdates:) with view.layer.render(in:)

Instead of drawing the view using the drawHierarchy provided on the UIKit-level, we can directly call the rendering of the layer used by the UIView provided by CoreGraphics.

Results at native scale (i.e. 2.0):

let image = UIGraphicsImageRenderer(size: view.bounds.size).image { context in
    view.layer.render(in: context.cgContext)
}

• Significant decrease in render time compared to the base line performance.
• Faster than the SentryGraphicsImageRenderer + drawHierarchy

Results at 1.0 scale:

let image = UIGraphicsImageRenderer(size: view.bounds.size, format: .init(for: .init(displayScale: 1))).image { context in
    view.layer.render(in: context.cgContext)
}

• Slightly slower than native scale, probably due to downscaling of internal graphics data

Experimental Renderer + view.layer.render(in:)

Combining the previous two approaches.

Results at native scale (i.e. 2.0):

let scale = (view as? UIWindow ?? view.window)?.screen.scale ?? 1
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.layer.render(in: context.cgContext)
}

• Significant decrease in render time compared to the base line performance.
• Large improvements compared to experimental renderer with view.drawHierarchy(...)
• No improvements compared to UIGraphicsImageRenderer + view.layer.render(in:)

Results at 1.0 scale:

let scale = 1.0
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.layer.render(in: context.cgContext)
}

• Slower than native scale, probably due to downscaling of internal graphics data

Detailed Analysis of Implementation

Looking at one of the samples using drawViewHierarchyInRect in detail we can analyze the duration of the different calls in detail. In particular we notice that the most expensive calls are inside UIKit and CoreGraphics.

26.00 ms  100,0 % Sentry            |  SentryGraphicsImageRenderer.image(actions:)
25.00 ms   96,2 % Sentry            |   closure #1 in SentryExperimentalViewRenderer.render(view:)
25.00 ms   96,2 % UIKitCore         |    -[UIView drawViewHierarchyInRect:afterScreenUpdates:]
21.00 ms   80,8 % UIKitCore         |     -[UIImage drawInRect:blendMode:alpha:]
21.00 ms   80,8 % CoreGraphics      |      CGContextDrawImageWithOptions
20.00 ms   76,9 % CoreGraphics      |       ripc_DrawImage
15.00 ms   57,7 % CoreGraphics      |        ripc_AcquireRIPImageData
 5.00 ms   19,2 % CoreGraphics      |        RIPLayerBltImage
 1.00 ms    3,8 % libdispatch.dylib |       _dispatch_once_callout
 4.00 ms   15,4 % UIKitCore         |     _UIRenderViewImageAfterCommit
 1.00 ms    3,8 % Sentry            |   SentryGraphicsImageRenderer.Context.currentImage.getter
 1.00 ms    3,8 % CoreGraphics      |    CGBitmapContextCreateImage

It seems like UIView is backed by an UIImage from a previous render, then draws it again into the bitmap context. As this is is a private API, we can not access it directly. No further optimization possible.