Immersive VR Companion Application
Bridging real-time gyroscope head tracking, ML-based hand gesture control, and side-by-side stereoscopic rendering.

FLOW is a state-of-the-art Flutter-based immersive VR companion app. It transforms standard Android smartphones into interactive virtual reality headsets using cardboard-style goggles.

Unlike traditional mobile VR apps that offer passive viewing, FLOW combines Side-by-Side (SBS) stereoscopic video playback with dual-channel gyroscopic head tracking and ML-powered hand tracking (via front/back cameras). This allows users to see their hand skeletons overlaid in real-time on top of the video sphere and control playback through intuitive, hands-free head and hand gestures.

• 🕶️ Stereoscopic SBS Viewer: Horizontally tiled eye views with horizontal and vertical translations simulating a 3D VR environment.
• 🔄 Dual-Channel Gyroscopic Sensor Processing: Independent damping and sensitivity constants for viewport translation and gesture detection.
• 🖐️ MediaPipe Hand Tracking Pipeline: Stream frames from camera, detect 21 landmarks, and paint virtual glowing skeletons inside the VR view.
• 🧠 Gesture Recognition:
  • Head Gestures: Tilt head left (seek back), right (seek forward), up (play/pause), or down (toggle telemetry HUD).
  • Hand Gestures: Detect finger counts (2 = seek back, 3 = seek forward, 4 = play/pause, 5 = toggle HUD).
• 🪄 Air Touch: Spatial drawing experience using the device's camera and ML-based hand tracking. Users can draw in mid-air using a single finger, change brush colors with 2-4 fingers, and clear the canvas with an open palm.
• 🚀 Guided Device Validation: A three-step setup flow checking headset readiness, gyroscopic capability, and camera permissions before entering VR mode.
• 📺 Native Fallback Launch Path: Optional ExoPlayer implementation rendering spherical 360° OpenGL surfaces.
• 🔒 Pilot Authorization Gate: Restricts usage based on expiration dates and optional device whitelists.

Below is a preview of the animated bootstrapping and startup experience from the loader:

Before jumping into the VR engine, users go through a strict calibration and authorization wizard to ensure their device matches the hardware requirements:

1. VR Goggles Readiness	2. Gyroscope Calibration	3. Hand Tracking Verification
Manual step explaining cardboards headset assembly.	Invokes the Android sensor manager to verify the presence of rotation vector fallback sensors.	Requests runtime camera permissions and launches a test screen to verify real-time landmarks.

FLOW is built using a modern, scalable feature-first architecture with strict isolation between core libraries, state layers, screens, and Android native integrations.

graph TD
    A[main.dart Bootstrap] --> B(Pilot Gate Validation)
    B -->|Authorized| C[GoRouter Navigation Graph]
    B -->|Expired/Unauthorized| D[Blocking System Dialog]
    
    C --> E[Loader Screen]
    E -->|Redirect 2200ms| F[Shell Router Base]
    
    F --> G[Home Dashboard]
    F --> H[Setup Wizard]
    F --> I[Play Content Library]
    F --> J[Profile Stats]
    
    I -->|Play Selection| K[VrImmersiveScreen Engine]
    
    subgraph VrImmersiveScreen Pipeline
        K --> L[SBS Video Player]
        K --> M[Gyroscope Controller]
        K --> N[Camera Image Streamer]
        N --> O[MediaPipe ML Detector]
        O --> P[Hand Overlay Painter]
    end

The primary engine (vr_immersive_screen.dart) runs a loop executing multiple tracking calculations concurrently:

The stereoscopic depth feel is achieved by horizontal and vertical shifting. Left and right views receive a small disparity offset:

$$\Delta x_{\text{left}} = (\text{viewYaw} - 0.02) \times 300,\text{px}$$ $$\Delta x_{\text{right}} = (\text{viewYaw} + 0.02) \times 300,\text{px}$$ $$\Delta y_{\text{both}} = \text{viewPitch} \times 200,\text{px}$$

The translation is handled inside a custom nested ClipRect + OverflowBox widget hierarchy, which allows clipping oversized videos as a relative window frame.

To avoid conflicts between smooth camera navigation and strict gesture detection thresholds, events are parsed through two distinct filters:

• Control Channel (Gesture Gating): Optimized for damping accidental minor shakes. $$\text{yaw}t = (\text{yaw}{t-1} + \omega_z \times 0.04) \times 0.85$$ $$\text{pitch}t = (\text{pitch}{t-1} + \omega_x \times 0.04) \times 0.85$$
• View Channel (Smooth Camera): Higher sensitivity and gentler decay for natural look-around behavior. $$\text{viewYaw}t = (\text{viewYaw}{t-1} + \omega_z \times 0.06) \times 0.92$$ $$\text{viewPitch}t = (\text{viewPitch}{t-1} + \omega_x \times 0.06) \times 0.92$$

Landmarks returned by the model are mapped using vector thresholds to check extension status:

• Thumb Extension Rule: $$\text{isThumbExtended} \iff |x_{\text{tip}} - x_{\text{wrist}}| > |x_{\text{IP}} - x_{\text{wrist}}|$$
• Fingers Extension Rule (Index, Middle, Ring, Pinky): $$\text{isFingerExtended} \iff y_{\text{tip}} < y_{\text{PIP}}$$ (Since coordinate space starts at the top-left, a lower Y index represents a vertically higher point)

FLOW introduces Air Touch, an interactive spatial drawing experience using the device's camera and ML-based hand tracking.

• Real-time Camera Streaming: Captures low-resolution YUV420 image streams for optimized hand tracking performance.
• Dynamic ML Hand Detection: Uses hand_detection to process frames and map 21 landmarks into a normalized coordinate space, correcting for device orientation changes on the fly.
• Intuitive Gestures:
  • Draw (1 Finger): Extend your index finger to begin painting dynamic strokes.
  • Color Picker (2-4 Fingers): Hold 2, 3, or 4 fingers to switch brush colors seamlessly.
  • Clear Canvas (5 Fingers): Open your palm to wipe the drawing space.
• Custom Rendering: Uses an overlay painter pipeline to visualize both skeletal hand structures and user-generated strokes in real-time.

flow/
├── android/app/src/main/
│   ├── kotlin/com/example/flow/
│   │   ├── MainActivity.kt        # Method channel handlers & sensor verification
│   │   └── VRPlayerActivity.kt    # Native Android Media3 Spherical Video Player
│   ├── res/layout/
│   │   └── activity_vr_player.xml # spherical_gl_surface_view rendering surface
│   └── AndroidManifest.xml        # Permission locks & Activity registration
├── assets/
│   ├── fonts/                     # NASA, Geist, and GeistPixel fonts
│   └── images/                    # SVGs, animations, and app icons
├── lib/
│   ├── main.dart                  # App bootstrap and orientation locks
│   ├── core/
│   │   ├── constants/             # Centralized strings
│   │   ├── native/                # VRChannel method interface
│   │   ├── providers/             # Riverpod providers (pilot, theme, profile)
│   │   ├── router/                # App navigation configuration
│   │   └── theme/                 # AppColors, radii, spacings, and theme definitions
│   ├── features/
│   │   ├── home/                  # Dashboard and quick actions
│   │   ├── loader/                # Startup splash animation
│   │   ├── play/                  # Immersive video player engine
│   │   ├── profile/               # User analytics and metrics
│   │   ├── settings/              # Configurations and theme toggles
│   │   ├── setup/                 # Calibration and test screens
│   │   └── shell/                 # Bottom navigation shell route layout
│   └── shared/widgets/            # Reusable components (GlassPanel, PilotBadge)
└── pubspec.yaml                   # Flutter dependencies declaration

• Flutter SDK version ≥ 3.5.0
• Android SDK setup with Min SDK level 23 (Marshmallow)
• A physical Android device containing a hardware gyroscope sensor and a functional back camera.

1. Clone the Repository:

   git clone https://github.com/your-username/flow.git
   cd flow

2. Fetch Dependencies:

   flutter clean
   flutter pub get

3. Run Development Build:

   flutter run

4. Assemble Release APK:

   flutter build apk --release

• 🎨 Barrel Distortion Shaders: Simulating physical fish-eye lenses to compensate for cardboard lenses.
• 📐 Calibration UI: Add screen-space sliders inside the HUD to alter yaw/pitch sensitivities and hand overlay translation multipliers.
• 🕶️ 3D Stereoscopic Hand Projection: Translate 2.5D coordinates to complete 3D vectors mapping real depth distances.
• 🌐 Cloud Video Library: Stream user-uploaded content directories rather than hardcoded mock feeds.

Made with 🩶 for immersive exploration.