Explore Spatial Scenes with Image Presentation Component

Joseph Simpson Updated on

We can pre or post-generate Spatial Scenes, depending on the needs of our app.

Overview

When converting photos to Spatial Scenes, we have two different paths we can take.

Pre-generate

We can choose to pre-generate the Spatial Scene before adding the component to an entity. This will let us present the image in our preferred viewing mode, but it can feel a bit slower. We provide an image URL to ImagePresentationComponent.Spatial3DImage and call generate() on the result. We can check availableViewingModes and set desiredViewingMode. Then we add the component to an entity.

let converted = try await ImagePresentationComponent.Spatial3DImage(contentsOf: url)
try await converted.generate()
var component = ImagePresentationComponent(spatial3DImage: converted)
if component.availableViewingModes.contains(.spatial3D) {
  component.desiredViewingMode = .spatial3D
}
entity.components.set(component)

This works well, but it can leave us with empty UI. You may want to consider showing a Progress View or loading indicator during generation. Let’s look at another option.

Post-generate

The set up is the same. We provide a URL and create an instance of Spatial3DImage. Then we create the component and add it to the entity. This will display the image in mono viewing mode. Next we call generate() on the image. If this succeeds, our component will be updated with spatial3D and spatial3DImmersive viewing modes. 

let converted = try await ImagePresentationComponent.Spatial3DImage(contentsOf: url)
var component = ImagePresentationComponent(spatial3DImage: converted)
entity.components.set(component)

try await converted.generate()

// Provide some UI to switch modes later
if component.availableViewingModes.contains(.spatial3D) {
  component.desiredViewingMode = .spatial3D
}

This option can provide a mono image as soon as visionOS can load it, making the UI feed more responsive. From here we can provide some controls to change viewing mode.

Dealing with Errors

The process to generate a Spatial Scene can throw errors. We can handle them just like any other error.

generationState = .loading
do {
    try await converted.generate()
    generationState = .success
} catch {
    generationState = .failure
    print("Generation failed: \(error.localizedDescription)")
    return
}

Video Demo

Example Code

struct Example130: View {
    @State var photoEntity = Entity()

    @State private var generationState: GenerationState = .empty
    @State private var currentMode = ImagePresentationComponent.ViewingMode.mono
    @State private var availableModes: Set<ImagePresentationComponent.ViewingMode> = []


    var body: some View {
        RealityView { content in
            photoEntity.setPosition([0, 1.6, -2.0], relativeTo: nil)
            photoEntity.scale = .init(repeating: 0.6)
            content.add(photoEntity)

            // Attach SwiftUI controls into the scene
            let controlMenu = Entity()
            let controlAttachment = ViewAttachmentComponent(
                rootView: ControlsPanel(
                    generationState: $generationState,
                    currentMode: $currentMode,
                    availableModes: $availableModes,
                    clearPhoto: { clearPhoto() },
                    loadPhotoPreGen: { Task { await loadPhotoPreGen(entity: photoEntity) } },
                    loadPhotoPostGen: { Task { await loadPhotoPostGen(entity: photoEntity) } }
                )
            )
            controlMenu.components.set(controlAttachment)
            controlMenu.setPosition([0, 1.2, -1.8], relativeTo: nil)
            content.add(controlMenu)

        }

        // Listen for changes to mode and update the component
        .onChange(of: currentMode, { _, newValue in
            photoEntity.components[ImagePresentationComponent.self]?.desiredViewingMode = newValue
        })
    }

    // Clear the current component
    func clearPhoto() {
        photoEntity.components.remove(ImagePresentationComponent.self)
        generationState = .empty
        currentMode = .mono
    }

    /// Load a regular (non-spatial) photo, then convert it to a Spatial Scene
    func loadPhotoPreGen(entity: Entity) async {
        guard let url = Bundle.main.url(forResource: "bell-01", withExtension: "jpeg") else { return }
        do {
            let converted = try await ImagePresentationComponent.Spatial3DImage(contentsOf: url)

            generationState = .loading

            do {
                try await converted.generate()
                var component = ImagePresentationComponent(spatial3DImage: converted)
                availableModes = component.availableViewingModes

                if availableModes.contains(.spatial3D) {
                    component.desiredViewingMode = .spatial3D
                    currentMode = .spatial3D
                }
                entity.components.set(component)
                generationState = .success
            } catch {
                let component = ImagePresentationComponent(spatial3DImage: converted)
                entity.components.set(component)
                availableModes = component.availableViewingModes
                currentMode = .mono
                generationState = .failure
                print("Generation failed: \(error.localizedDescription)")
                return
            }

        } catch {
            print("Failed to load image: \(error)")
        }
    }

    /// Load a regular (non-spatial) photo, but do not generate it yet.
    func loadPhotoPostGen(entity: Entity) async {
        guard let url = Bundle.main.url(forResource: "bell-01", withExtension: "jpeg") else { return }
        do {
            let converted = try await ImagePresentationComponent.Spatial3DImage(contentsOf: url)
            let component = ImagePresentationComponent(spatial3DImage: converted)
            entity.components.set(component)

            generationState = .loading
            do {
                try await converted.generate()
                availableModes = component.availableViewingModes
                currentMode = .mono
                generationState = .success
            } catch {
                availableModes = component.availableViewingModes
                currentMode = .mono
                generationState = .failure
                print("Generation failed: \(error.localizedDescription)")
                return
            }

        } catch {
            print("Failed to load image: \(error)")
        }
    }
}

fileprivate enum GenerationState {
    case empty
    case loading
    case success
    case failure
}

// Moving the control panel to a view
fileprivate struct ControlsPanel: View {
    @Binding var generationState: GenerationState
    @Binding var currentMode: ImagePresentationComponent.ViewingMode
    @Binding var availableModes: Set<ImagePresentationComponent.ViewingMode>

    let clearPhoto: () -> Void
    let loadPhotoPreGen: () -> Void
    let loadPhotoPostGen: () -> Void

    var body: some View {
        VStack(spacing: 12) {
            HStack {
                switch generationState {
                case .empty:
                    Text("No photo loaded.")
                case .loading:
                    Text("Generating Spatial Scene...")
                case .success:
                    Text("Spatial Scene generated.")
                case .failure:
                    Text("Failed to generate Spatial Scene.")
                }
            }

            HStack(spacing: 12) {

                Button(action: {
                    clearPhoto()
                }, label: {
                    Text("Clear")
                })

                Button(action: {
                    loadPhotoPreGen()
                }, label: {
                    Text("Pre Gen")
                })

                Button(action: {
                    loadPhotoPostGen()
                }, label: {
                    Text("Post Gen")
                })
            }
            .controlSize(.extraLarge)

            HStack {
                Button(action: {
                    currentMode = .mono
                }, label: {
                    Text("mono")
                })
                .disabled(!availableModes.contains(.mono))

                Button(action: {
                    currentMode = .spatial3D
                }, label: {
                    Text("spatial3D")
                })
                .disabled(!availableModes.contains(.spatial3D))

                Button(action: {
                    currentMode = .spatial3DImmersive
                }, label: {
                    Text("spatial3DImmersive")
                })
                .disabled(!availableModes.contains(.spatial3DImmersive))

            }
            .controlSize(.small)
            .padding()

        }
        .padding()
        .background(.black)
        .clipShape(.capsule)
    }
}

Support our work so we can continue to bring you new examples and articles.

Support our Work

Download the Xcode project with this and many more examples from Step Into Vision.
Some examples are provided as standalone Xcode projects. You can find those here.

Follow Step Into Vision

Questions or feedback?