ARKit PlaneDetectionProvider: visualize detected planes
Converting anchor geometry into a meshes we can render.
Overview
Let’s dive into PlaneDetectionProvider. This provider gives us access to anchors that we can use incorporate into our scenes. In this example, let’s unpack how to render some meshes for each of the detected anchors.
This post shows how to render the complex shapes that ARKit detects. If you would like to render simple shapes based on the extent of the anchor, see creating simple planes from anchors.
We need a place to store our session, and an array of anchors.
@State var session = ARKitSession()
@State private var planeAnchors: [UUID: Entity] = [:]
@State private var planeColors: [UUID: Color] = [:] //just here so we can use the same random color when an anchor is updatedWe can configure and run the session. When we detect a new anchor, or when one is updated, we’ll call createPlaneEntity to generate a new mesh. When an anchor is removed, we’ll remove it from the array and call removeFromParent on the related entity. This will remove the entity because it is a child of the RealityKit root. If you are grouping your plane entities under another object, you may have to do a bit more work to remove them.
func setupAndRunPlaneDetection() async throws {
let planeData = PlaneDetectionProvider(alignments: [.horizontal, .vertical, .slanted])
if PlaneDetectionProvider.isSupported {
do {
try await session.run([planeData])
for await update in planeData.anchorUpdates {
switch update.event {
case .added, .updated:
let anchor = update.anchor
if planeColors[anchor.id] == nil {
planeColors[anchor.id] = generatePastelColor()
}
let planeEntity = createPlaneEntity(for: anchor, color: planeColors[anchor.id]!)
planeAnchors[anchor.id] = planeEntity
case .removed:
let anchor = update.anchor
planeAnchors.removeValue(forKey: anchor.id)
planeColors.removeValue(forKey: anchor.id)
}
}
} catch {
print("ARKit session error \(error)")
}
}
}
createPlaneEntity will create a mesh with an assigned color for each plane anchor. We name each entity for the anchor id. We also set the transform of each entity to anchor.originFromAnchorTransform. The hard part comes next, in createMeshResource
private func createPlaneEntity(for anchor: PlaneAnchor, color: Color) -> Entity {
let entity = Entity()
entity.name = "Plane \(anchor.id)"
entity.setTransformMatrix(anchor.originFromAnchorTransform, relativeTo: nil)
var material = PhysicallyBasedMaterial()
material.baseColor.tint = UIColor(color)
if let meshResource = createMeshResource(anchor: anchor) {
entity.components.set(ModelComponent(mesh: meshResource, materials: [material]))
}
return entity
}Apple has an example called Placing content on detected planes. It took me longer than I care to admit, but I eventually figured out what they were doing in that project. In initial attempt, I tried to use the extent of the geometry to create some simple plane meshes.
// ❌ don't do this
let mesh = MeshResource.generatePlane(
width: anchor.geometry.extent.width,
depth: anchor.geometry.extent.height
)This didn’t work well. The planes I rendered didn’t line up where I expected them to be. You can read more about the attempt on the developer forums. In reality, extent is much line a bounding box for an n-gon, but the n-gon may not be shaped as we expect. I think it is helpful to think of plane detection more like surface detect. This provider detects flat surfaces, not necessarily rectangular planes.
Put another way, these anchors give is vertices and faces like this:
// vertices
GeometrySource(count: 18, format: MTLVertexFormat(rawValue: 30))
// faces
GeometryElement(count: 16, primitive: triangle)Which we need to convert into a format that can be used in RealityKit to create a shape
// converted vertices
[
SIMD3<Float>(-0.22344242, 0.0, 0.13505287),
SIMD3<Float>(0.051172107, 0.0, 0.13218012),
SIMD3<Float>(0.10902132, 0.0, 0.12598379),
SIMD3<Float>(0.11184792, 0.0, 0.124783024),
//...
]
// converted faces
[1, 4, 8, 11, 14, 15, 11, 13, 14, 11, 12, 13, 11, 15, 16, 10, 11, 16, 0, 10, 16, 4, 7, 8, 5, 6, 7, 4, 5, 7, 1, 2, 3, 1, 3, 4, 1, 8, 9, 1, 9, 10, 0, 1, 10, 0, 16, 17]The example project from Apple has this data conversion stashed away in a bunch of extensions on various classes and structs. I found that a bit of a pain to work with, so with the help of Cursor, I removed the extensions and places all the logic in this function
private func createMeshResource(anchor: PlaneAnchor) -> MeshResource? {
// Generate a mesh for the plane (for occlusion).
var meshResource: MeshResource? = nil
do {
var contents = MeshResource.Contents()
contents.instances = [MeshResource.Instance(id: "main", model: "model")]
var part = MeshResource.Part(id: "part", materialIndex: 0)
// Convert vertices to SIMD3<Float>
let vertices = anchor.geometry.meshVertices
var vertexArray: [SIMD3<Float>] = []
for i in 0..<vertices.count {
let vertex = vertices.buffer.contents().advanced(by: vertices.offset + vertices.stride * i).assumingMemoryBound(to: (Float, Float, Float).self).pointee
vertexArray.append(SIMD3<Float>(vertex.0, vertex.1, vertex.2))
}
part.positions = MeshBuffers.Positions(vertexArray)
print("vertices \(vertices)")
print("was converted to \(vertexArray)")
// Convert faces to UInt32
let faces = anchor.geometry.meshFaces
var faceArray: [UInt32] = []
let totalFaces = faces.count * faces.primitive.indexCount
for i in 0..<totalFaces {
let face = faces.buffer.contents().advanced(by: i * MemoryLayout<Int32>.size).assumingMemoryBound(to: Int32.self).pointee
faceArray.append(UInt32(face))
}
part.triangleIndices = MeshBuffer(faceArray)
print("faces \(faces)")
print("was converted to \(faceArray)")
contents.models = [MeshResource.Model(id: "model", parts: [part])]
meshResource = try MeshResource.generate(from: contents)
return meshResource
} catch {
print("Failed to create a mesh resource for a plane anchor: \(error).")
}
return nil
}Full Example Code
struct Example069: View {
@State var session = ARKitSession()
@State private var planeAnchors: [UUID: Entity] = [:]
@State private var planeColors: [UUID: Color] = [:]
var body: some View {
RealityView { content in
} update: { content in
for (_, entity) in planeAnchors {
if !content.entities.contains(entity) {
content.add(entity)
}
}
}
.task {
try! await setupAndRunPlaneDetection()
}
}
func setupAndRunPlaneDetection() async throws {
let planeData = PlaneDetectionProvider(alignments: [.horizontal, .vertical, .slanted])
if PlaneDetectionProvider.isSupported {
do {
try await session.run([planeData])
for await update in planeData.anchorUpdates {
switch update.event {
case .added, .updated:
let anchor = update.anchor
if planeColors[anchor.id] == nil {
planeColors[anchor.id] = generatePastelColor()
}
let planeEntity = createPlaneEntity(for: anchor, color: planeColors[anchor.id]!)
planeAnchors[anchor.id] = planeEntity
case .removed:
let anchor = update.anchor
if let entity = planeAnchors[anchor.id] {
entity.removeFromParent()
planeAnchors.removeValue(forKey: anchor.id)
planeColors.removeValue(forKey: anchor.id)
}
}
}
} catch {
print("ARKit session error \(error)")
}
}
}
private func generatePastelColor() -> Color {
let hue = Double.random(in: 0...1)
let saturation = Double.random(in: 0.2...0.4)
let brightness = Double.random(in: 0.8...1.0)
return Color(hue: hue, saturation: saturation, brightness: brightness)
}
private func createMeshResource(anchor: PlaneAnchor) -> MeshResource? {
// Generate a mesh for the plane (for occlusion).
var meshResource: MeshResource? = nil
do {
var contents = MeshResource.Contents()
contents.instances = [MeshResource.Instance(id: "main", model: "model")]
var part = MeshResource.Part(id: "part", materialIndex: 0)
// Convert vertices to SIMD3<Float>
let vertices = anchor.geometry.meshVertices
var vertexArray: [SIMD3<Float>] = []
for i in 0..<vertices.count {
let vertex = vertices.buffer.contents().advanced(by: vertices.offset + vertices.stride * i).assumingMemoryBound(to: (Float, Float, Float).self).pointee
vertexArray.append(SIMD3<Float>(vertex.0, vertex.1, vertex.2))
}
part.positions = MeshBuffers.Positions(vertexArray)
print("vertices \(vertices)")
print("was converted to \(vertexArray)")
// Convert faces to UInt32
let faces = anchor.geometry.meshFaces
var faceArray: [UInt32] = []
let totalFaces = faces.count * faces.primitive.indexCount
for i in 0..<totalFaces {
let face = faces.buffer.contents().advanced(by: i * MemoryLayout<Int32>.size).assumingMemoryBound(to: Int32.self).pointee
faceArray.append(UInt32(face))
}
part.triangleIndices = MeshBuffer(faceArray)
print("faces \(faces)")
print("was converted to \(faceArray)")
contents.models = [MeshResource.Model(id: "model", parts: [part])]
meshResource = try MeshResource.generate(from: contents)
return meshResource
} catch {
print("Failed to create a mesh resource for a plane anchor: \(error).")
}
return nil
}
private func createPlaneEntity(for anchor: PlaneAnchor, color: Color) -> Entity {
let entity = Entity()
entity.name = "Plane \(anchor.id)"
entity.setTransformMatrix(anchor.originFromAnchorTransform, relativeTo: nil)
var material = PhysicallyBasedMaterial()
material.baseColor.tint = UIColor(color)
if let meshResource = createMeshResource(anchor: anchor) {
entity.components.set(ModelComponent(mesh: meshResource, materials: [material]))
}
return entity
}
}Support our work so we can continue to bring you new examples and articles.
Download the Xcode project with this and many more examples from Step Into Vision.
Some examples are provided as standalone Xcode projects. You can find those here.
Special thanks to Gil on Bluesky for pointing me in the right direction for this. https://bsky.app/profile/gil-n.bsky.social