This document provides a high-level introduction to the mirai package: its purpose, core concepts, architectural design, and position within the R ecosystem. mirai is a minimalist async evaluation framework that enables asynchronous and parallel execution of R expressions across distributed computing resources.
For detailed information on specific topics:
Sources: DESCRIPTION13-20 README.md17-27 R/mirai-package.R1-11
mirai (ミライ, meaning "future" in Japanese) is an asynchronous evaluation framework for R that enables parallel computation locally or across distributed networks. The package implements a non-polling, event-driven architecture built on the nanonext package and NNG (Nanomsg Next Generation) messaging library, allowing it to scale from a single laptop to thousands of processes across high-performance computing clusters and cloud platforms.
The core workflow consists of two primary functions:
mirai() - submits R expressions for asynchronous evaluation, returning immediately with a mirai object README.md54-58daemons() - configures persistent background processes that receive and execute tasks README.md51Core Asynchronous Evaluation Workflow
Sources: README.md41-77 R/mirai-package.R1-11 DESCRIPTION13-20
mirai implements a hub-and-spoke architecture where the host R session listens at a URL and daemons dial in to connect. This inverted connection model enables true dynamic scaling—daemons can be added or removed at any time without requiring the host to know in advance how many workers will connect.
Hub Architecture with Heterogeneous Daemon Sources
The host uses listen() from nanonext to create the listening socket, while daemons use dial() to connect. This pattern allows mixing daemon types freely—local subprocesses, SSH-launched remote processes, HPC scheduler jobs, and HTTP API-launched instances all connect to the same URL.
Sources: README.md82-96 R/mirai-package.R14-21 R/mirai-package.R33-39
A compute profile is a named configuration that maintains its own daemon pool, allowing tasks to be directed to specific resource types. The .compute parameter in mirai() and related functions specifies which profile to use.
Compute Profiles Direct Tasks to Different Resource Pools
Each profile operates independently with its own configuration, including optional dispatcher process, TLS settings, custom serialization, and OpenTelemetry tracing context. The default profile name is "default" R/mirai-package.R71
Sources: README.md86-92 R/mirai-package.R71
daemons() can operate in two modes controlled by the dispatcher parameter:
Dispatcher Mode (dispatcher=TRUE, default): An intermediate dispatcher process manages FIFO task scheduling. As of version 2.6.0, the dispatcher() loop is re-implemented entirely in C code within nanonext for reduced overhead NEWS.md30 This mode enables features like task cancellation with stop_mirai() and optimal load balancing.
Direct Mode (dispatcher=FALSE): Tasks are sent directly from the host to daemons without an intermediary, reducing overhead for simple use cases but sacrificing advanced features like task cancellation.
Dispatcher Mode Provides Centralized Scheduling; Direct Mode Minimizes Overhead
For detailed information on dispatcher architecture, see Dispatcher Architecture.
Sources: NEWS.md30 NEWS.md256-257 NEWS.md302-306 R/mirai-package.R33-39
| Function | Purpose | Returns |
|---|---|---|
mirai() | Submit async task | mirai object (handle to result) |
daemons() | Configure daemon pool for compute profile | Logical (invisible) NEWS.md84-85 |
mirai_map() | Parallel map operation | mirai_map object (list of tasks) |
everywhere() | Broadcast expression to all daemons | mirai_map object |
call_mirai() | Block and wait for result | Result or error value |
unresolved() | Check if task is complete | Logical |
stop_mirai() | Cancel ongoing task | Logical (success/failure) |
race_mirai() | Wait for the next mirai to resolve | Integer index NEWS.md24-26 |
| Function | Purpose | Returns |
|---|---|---|
launch_local() | Launch local daemons manually | Count launched |
launch_remote() | Launch remote daemons or return shell commands | Character vector or count |
| Function | Purpose | Returns |
|---|---|---|
ssh_config() | SSH deployment configuration | Configuration object |
cluster_config() | HPC scheduler configuration | Configuration object |
http_config() | HTTP API deployment configuration | Configuration object NEWS.md20 |
serial_config() | Custom serialization configuration | Configuration list |
host_url() | Construct host URL from hostname | Character vector (URLs) |
local_url() | Construct local IPC URL | Character (URL) |
Component Dependencies Showing nanonext Foundation
Sources: R/mirai-package.R33-39 R/mirai-package.R1-11
Unlike traditional polling-based systems, mirai uses a fully event-driven architecture based on NNG's asynchronous I/O primitives. This design consumes zero CPU when idle and responds to task completion with microsecond latency.
The key architectural elements:
mirai() uses send() in asynchronous mode and returns immediately R/mirai-package.R33-39recv_aio() to set up callbacks that fire on data arrival R/mirai-package.R33-39cv() objects from nanonext provide signaling without polling R/mirai-package.R33-39as.promise() method converts mirai objects to promises that resolve via event-driven callbacks NEWS.md107Sources: README.md22 README.md101-104 R/mirai-package.R33-39
mirai supports multiple transport protocols through NNG:
| Transport | URL Scheme | Use Case | Platform Notes |
|---|---|---|---|
| Abstract Unix sockets | abstract:// | Local IPC | Linux only (default) R/mirai-package.R52 |
| Unix domain sockets | ipc:///tmp/ | Local IPC | POSIX systems (default on macOS) R/mirai-package.R60 |
| Named pipes | ipc:// | Local IPC | Windows (default) R/mirai-package.R56 |
| TCP | tcp:// | Network communication | All platforms |
| TLS over TCP | tls+tcp:// | Secure network | All platforms |
| WebSocket | ws:// | Web-based | All platforms |
| WebSocket Secure | wss:// | Secure web-based | All platforms |
Platform-specific defaults are set in .onLoad R/mirai-package.R45-63
Sources: R/mirai-package.R45-63 R/mirai-package.R14-21
When using dispatcher mode, tasks are queued in strict first-in-first-out order and dispatched to available daemons. The stop_mirai() function can cancel tasks that have not yet started execution NEWS.md100
Task Cancellation Before Dispatch
Sources: NEWS.md100 NEWS.md258 README.md8
mirai provides comprehensive OpenTelemetry support for observability across distributed processes. When tracing is enabled, mirai emits spans for asynchronous operations and distributed computation R/mirai-package.R22-27
For complete documentation, see OpenTelemetry Tracing.
Sources: R/mirai-package.R22-27 NEWS.md91-92
The serial_config() function registers serialization hooks for handling objects that cannot be serialized by R's default mechanism, such as torch tensors, Arrow tables, or Polars dataframes README.md140-156
For implementation details, see Custom Serialization.
Sources: NEWS.md105-106 README.md112 README.md140-156
mirai serves as a convergence point for asynchronous and parallel computing across R:
mirai is the first official alternative communications backend for R's parallel package README.md126-128 Users can create a cluster with parallel::makeCluster(type = "MIRAI").ExtendedTask support README.md134-136purrr and provides infrastructure for tidymodels README.md130-144targets pipelines via the crew package README.md158-160For a complete survey of integrations, see Ecosystem Integration.
Sources: README.md123-161 DESCRIPTION14-15
This overview introduced mirai's core concepts. For practical usage:
mirai running.For architecture details:
Sources: README.md79-80
Refresh this wiki