RealityKit Basics: Interaction
Using system gestures to interact with entities in RealityKit.
Overview
We can use standard SwiftUI gestures to interact with 3D content in our scenes. There are a few things we need to keep in mind.
- Each entity we want to interact with needs to have two components. We can add these in Reality Composer Pro, or directly in code.
- Input Component
- Collision Component
- When we create a gesture, we need to target it to one or more entities. We can use
targetedToEntity()to target a known entity- targetedToEntity(where:) to target entities based on a query. For example, only run this on entities with a certain component.
targetedToAnyEntity()to target any entity in our scene.
Learn more about gestures: SwiftUI Gestures with RealityKit Entities
Examples
Let’s start with a simple tap gesture. When we tap the earth entity we will scale it up slightly. We target the gesture to the earth entity so it won’t fire when we tap on anything else. We use an instance method on Entity to change the transform over time.
.gesture(TapGesture().targetedToEntity(earth)
.onEnded({ value in
var newTransform = Transform()
newTransform.scale = .init(repeating: 1.25)
earth.move(to: newTransform, relativeTo: earth, duration: 0.3)
})
)We can also double-tap the earth to reset the transform back to the original value. Note that we need to place this gesture higher in the call chain than the single tap. Otherwise, we will always fire the single tap.
.gesture(TapGesture(count: 2).targetedToEntity(earth)
.onEnded({ value in
earth.move(to: earthTransform, relativeTo: earth.parent!, duration: 0.3)
})
)When we tap on the moon, we can trigger a behavior to start a timeline. Behaviors and timelines are defined in Reality Composer Pro. In this case, we have a simple timeline that will spin and orbit the moon. We can trigger that timeline with the On Tap behavior. We’ll call the applyTapForBehaviors() instance method on Entity to trigger the behavior. The order of events is:
- Tap the moon
- Call
applyTapForBehaviors()to trigger On Tap behavior - On Tap behavior calls the timeline
- The timeline animates the moon over time
.gesture(TapGesture().targetedToEntity(moon)
.onEnded({ value in
// call applyTapForBehaviors to trigger the behavior on the moon entity
if value.entity.applyTapForBehaviors() {
print("timeline is running")
}
})
)Let’s look at one last example. We’ll use the drag gesture to rotate entities around their Y Axis. This example was adapted from Lab 014 – Building an Indirect Transform System.
fileprivate struct DragToRotate: ViewModifier {
@State var isDragging: Bool = false
@State var initialOrientation:simd_quatf = simd_quatf(
vector: .init(repeating: 0.0)
)
@State var rotation: Angle = .zero
func body(content: Content) -> some View {
content
.gesture(
DragGesture()
.targetedToAnyEntity()
.onChanged { value in
if !isDragging {
isDragging = true
initialOrientation = value.entity.transform.rotation
}
rotation.degrees += 0.01 * (value.velocity.width)
let rotationTransform = Transform(yaw: Float(rotation.radians))
value.entity.transform.rotation = initialOrientation * rotationTransform.rotation
}
.onEnded { value in
isDragging = false
initialOrientation = simd_quatf(
vector: .init(repeating: 0.0)
)
}
)
}
}When you’re ready to dive further into gestures, we have an entire series to get you started.
learn visionOS / System Gestures
Video Demo
Full Example Code
struct Example042: View {
@State var earth: Entity = Entity()
@State var earthTransform: Transform = Transform()
@State var moon: Entity = Entity()
var body: some View {
RealityView { content in
// Load a scene from the bundle
if let scene = try? await Entity(named: "RKBasicsLoading", in: realityKitContentBundle) {
content.add(scene)
scene.position.y = -0.4 // Move the entire scene down in the volume
// search the scene for an entity named "Earth"
if let earth = scene.findEntity(named: "Earth") {
self.earth = earth // capture the entity in state, used to target gestures
self.earthTransform = earth.transform // capture transform, used to reset
}
if let moon = scene.findEntity(named: "Moon") {
self.moon = moon // capture the entity in state, used to target gestures
}
}
}
// Example 2: Double tap the earth to reset transform
// Note that we need to place this higher in the call chain than the single tap.
.gesture(TapGesture(count: 2).targetedToEntity(earth)
.onEnded({ value in
earth.move(to: earthTransform, relativeTo: earth.parent!, duration: 0.3)
})
)
// Example 1: Tap the earth to scale it up
.gesture(TapGesture().targetedToEntity(earth)
.onEnded({ value in
var newTransform = Transform()
newTransform.scale = .init(repeating: 1.25)
earth.move(to: newTransform, relativeTo: earth, duration: 0.3)
})
)
// Example 3: Tap the moon to fire a timeline from Reality Composer Pro
.gesture(TapGesture().targetedToEntity(moon)
.onEnded({ value in
// call applyTapForBehaviors to trigger the behavior on the moon entity
if value.entity.applyTapForBehaviors() {
print("timeline is running")
}
})
)
// Example 4, part 2
.modifier(DragToRotate())
}
}
// Example 4, part 1
// We'll use a drag gesture to rotate the earth around the Y axis
// Adapted from Lab 014 - Building an Indirect Transform System
// https://stepinto.vision/labs/lab-014-building-an-indirect-transform-system/
fileprivate struct DragToRotate: ViewModifier {
@State var isDragging: Bool = false
@State var initialOrientation:simd_quatf = simd_quatf(
vector: .init(repeating: 0.0)
)
@State var rotation: Angle = .zero
func body(content: Content) -> some View {
content
.gesture(
DragGesture()
.targetedToAnyEntity()
.onChanged { value in
if !isDragging {
isDragging = true
initialOrientation = value.entity.transform.rotation
}
rotation.degrees += 0.01 * (value.velocity.width)
let rotationTransform = Transform(yaw: Float(rotation.radians))
value.entity.transform.rotation = initialOrientation * rotationTransform.rotation
}
.onEnded { value in
isDragging = false
initialOrientation = simd_quatf(
vector: .init(repeating: 0.0)
)
}
)
}
}Support our work so we can continue to bring you new examples and articles.
Download the Xcode project with this and many more examples from Step Into Vision.
Some examples are provided as standalone Xcode projects. You can find those here.
Follow Step Into Vision