Skip to content

feat: add standalone mode for Docker-free local development#2012

Merged
jlaneve merged 36 commits intomainfrom
feat/standalone-mode
Mar 3, 2026
Merged

feat: add standalone mode for Docker-free local development#2012
jlaneve merged 36 commits intomainfrom
feat/standalone-mode

Conversation

@jlaneve
Copy link
Contributor

@jlaneve jlaneve commented Feb 13, 2026

Summary

Adds a standalone mode for running Airflow locally without Docker, unified into the existing astro dev command tree via mode-based dispatch.

How it works

  • A dev.mode project config ("docker" or "standalone") controls which backend is used
  • --standalone / --docker persistent flags allow per-call overrides
  • All existing commands (start, stop, logs, ps, kill, restart) work in both modes
  • run, bash, pytest, parse, object import, object export all work in standalone mode
  • logs supports component filtering (--scheduler, --api-server, etc.) in both modes
  • build, upgrade-test, and compose-export return clear "not available" errors in standalone mode

Key changes

  • Config: dev.mode config with "docker" default, validated to accept only "docker" or "standalone"
  • Flags: --standalone/--docker mutually exclusive persistent flags on the dev root command
  • Dispatch: resolveDevMode() resolves mode (flag → config priority), resolveHandlerInit() returns the right handler
  • Hooks: ConfigureContainerRuntime assigns a noOpContainerRuntime in standalone mode, so downstream hooks (EnsureRuntime, SetRuntimeIfExists, KillPreRunHook, KillPostRunHook) work without nil-pointer guards or mode checks
  • Start/Restart: --foreground/-f and --port/-p flags added for standalone mode
  • Grouped help: start and restart use a custom usage template that organizes flags into Common / Docker Mode / Standalone Mode sections
  • Log filtering: astro dev logs component flags (--scheduler, --api-server, --triggerer, --dag-processor, --webserver) filter standalone log lines by prefix. --webserver maps to api-server since AF3 standalone has no separate webserver. Unfiltered shows all lines including standalone orchestrator messages.
  • Port conflict detection: Pre-flight TCP check before starting — airflow standalone doesn't crash when the port is taken (only the api-server subprocess fails), so without this check the health check would pass against whichever service already occupies the port
  • Custom port: Uses AIRFLOW__API__PORT (AF3 config key) instead of the deprecated AIRFLOW__WEBSERVER__WEB_SERVER_PORT, reads from both --port flag and api-server.port config
  • Command dispatch: build and upgrade-test now route through resolveHandlerInit() instead of hardcoded containerHandlerInit, so they properly error in standalone mode instead of silently running Docker operations
  • Removed: The separate astro dev local command tree is replaced by the unified approach

Standalone command implementations

Command Standalone behavior
astro dev run <cmd> Executes command in the venv environment (e.g. airflow dags list)
astro dev bash Opens interactive shell with venv env (AIRFLOW_HOME, PATH set)
astro dev object import Imports connections/variables/pools from airflow_settings.yaml via venv
astro dev object export Exports to settings file or .env via venv
astro dev pytest Runs pytest on DAGs with file path resolution and args passthrough
astro dev parse Validates DAGs by running the default integrity test
astro dev logs Streams log file with optional component filtering
astro dev build Returns clear "not available in standalone mode" error
astro dev compose-export Returns clear "not available in standalone mode" error
astro dev upgrade-test Returns clear "not available in standalone mode" error

Usage

Setting the mode via config:

# Set standalone as the default mode for a project
$ astro config set dev.mode standalone
Setting dev.mode to standalone successfully

# Check the current mode
$ astro config get dev.mode
dev.mode: standalone

# Switch back to Docker mode
$ astro config set dev.mode docker
Setting dev.mode to docker successfully

# Invalid values are rejected
$ astro config set dev.mode invalid
Error: invalid value for dev.mode: dev.mode must be "docker" or "standalone", got "invalid"

Using per-command flags:

# Override mode for a single command
astro dev start --standalone
astro dev start --standalone --foreground --port 9090
astro dev logs --standalone -f
astro dev stop --standalone
astro dev kill --standalone

Standalone-specific commands:

# Run airflow CLI commands
astro dev run dags list
astro dev run connections list
astro dev run version

# Open an interactive shell with venv environment
astro dev bash

# Import/export airflow objects
astro dev object import
astro dev object export
astro dev object export --env-export

# Run tests
astro dev pytest
astro dev pytest dags/test_my_dag.py
astro dev pytest --args "-v --tb=short"
astro dev parse

# Filter logs by component
astro dev logs -f --scheduler
astro dev logs --api-server
astro dev logs --triggerer --dag-processor
astro dev logs --webserver  # maps to api-server in standalone

What the flags look like

$ astro dev start --help

Start a local Airflow environment. In Docker mode (default), this spins up
containers for each Airflow component. In standalone mode, this runs Airflow
directly on your machine without Docker.

Usage:
  astro dev start [flags]

Flags:
  -d, --deployment-id string   ID of the Deployment to retrieve environment connections from
  -e, --env string             Location of file containing environment variables (default ".env")
  -h, --help                   help for start
  -s, --settings-file string   Settings file from which to import airflow objects (default "airflow_settings.yaml")
      --wait duration          Duration to wait for webserver to get healthy. (default 1m0s)
  -w, --workspace-id string    ID of the Workspace to retrieve environment connections from.

Docker Mode Flags:
      --build-secrets strings   Mimics docker build --secret flag.
      --compose-file string     Location of a custom compose file to use for starting Airflow
  -i, --image-name string       Name of a custom built image to start airflow with
  -n, --no-browser              Don't bring up the browser once the Webserver is healthy
      --no-cache                Do not use cache when building container image

Standalone Mode Flags:
  -f, --foreground    Run in the foreground
  -p, --port string   Port for the Airflow webserver

Global Flags:
      --docker             Run in Docker mode
      --standalone         Run in standalone mode (without Docker)
      --verbosity string   Log level (debug, info, warn, error, fatal, panic (default "warning")

Known limitation

astro dev run --standalone does not work because the run command uses DisableFlagParsing (pre-existing cobra behavior to pass args through to airflow). Use astro config set dev.mode standalone instead, which works correctly for all commands including run.

Test plan

  • Unit tests for all new standalone commands and helpers
  • Unit test for port conflict detection (TestStandaloneStart_PortInUse)
  • Existing Docker-mode tests still pass
  • E2E: fresh project init → start → run/parse/pytest/import/export/logs → restart → stop
  • E2E: port conflict between two standalone projects blocked immediately
  • E2E: port conflict with non-Airflow process blocked immediately
  • E2E: custom port via astro config set api-server.port works correctly
  • E2E: build, upgrade-test, compose-export show proper errors in standalone mode

🤖 Generated with Claude Code

@coveralls-official
Copy link

coveralls-official bot commented Feb 13, 2026

Pull Request Test Coverage Report for Build 5a0db6db-bca6-478c-bbbd-cd9fc16e1472

Details

  • 723 of 958 (75.47%) changed or added relevant lines in 11 files are covered.
  • 43 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.5%) to 35.682%

Changes Missing Coverage Covered Lines Changed/Added Lines %
airflow/mocks/ContainerHandler.go 0 4 0.0%
settings/settings.go 0 5 0.0%
cmd/airflow_hooks.go 1 8 12.5%
airflow_versions/airflow_versions.go 0 28 0.0%
cmd/airflow_help.go 7 35 20.0%
airflow/standalone.go 575 738 77.91%
Files with Coverage Reduction New Missed Lines %
pkg/telemetry/telemetry.go 43 34.27%
Totals Coverage Status
Change from base Build d6e0c989-4e72-4da5-a7d0-4566338f1974: 0.5%
Covered Lines: 23786
Relevant Lines: 66661

💛 - Coveralls

…pment

Add a new `astro dev standalone` command that runs Airflow locally
without Docker, using `airflow standalone` and `uv` for dependency
management. This provides a dramatically faster dev loop for Airflow 3
projects.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve force-pushed the feat/standalone-mode branch from d95fbe6 to 1e42a1e Compare February 13, 2026 19:22
pre-commit-ci bot and others added 4 commits February 13, 2026 19:22
Standalone mode now works with both Airflow 2 (runtime 4.0.0+) and
Airflow 3 (runtime 3.x). The health check endpoint, image registry,
and settings version are determined dynamically based on the detected
Airflow major version.

Changes:
- Accept airflowMajor "2" or "3" (was "3" only)
- Health check: /health + webserver (AF2) vs /api/v2/monitor/health + api-server (AF3)
- Image registry: quay.io/astronomer/astro-runtime (AF2) vs astrocrpublic.azurecr.io/runtime (AF3)
- Settings version passed dynamically (was hardcoded 3)
- Kill/reset cleans up both AF2 and AF3 credential files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Default `astro dev standalone` to background mode — the CLI starts the
airflow process, writes a PID file, waits for the health check, prints
status, and returns.  A `--foreground` flag preserves the previous
stream-to-terminal behaviour.

New subcommands:
  - `astro dev standalone stop`  — SIGTERM the process group, clean up PID file
  - `astro dev standalone logs [-f]` — dump or tail the log file

Also wires `reset` to stop a running process before cleaning up files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve force-pushed the feat/standalone-mode branch from 9683aac to 2593b32 Compare February 13, 2026 21:25
@tayloramurphy
Copy link
Contributor

tayloramurphy commented Feb 16, 2026

@jlaneve I know this mirrors the airflow standalone command but would an shorter alias be useful too? standalone is just a lot to type. lite or native could be alternates.

Edit: no longer relevant now that it's a flag / config setting.

…install

- Remove airflowMajor field and AF2 code paths (standalone is AF3-only)
- Replace Docker-based constraint extraction with HTTP fetch from
  pip.astronomer.io/runtime-constraints
- Implement 2-step install: first install airflow with full constraints,
  then install user requirements with only airflow/task-sdk version locks
- Add parsePackageVersionFromConstraints helper for task-sdk version
- Remove runtimeImageName, execDockerRun, constraintsFileInImage
- Simplify healthEndpoint to always return AF3 endpoint

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve force-pushed the feat/standalone-mode branch from b1a21b5 to e91a7ad Compare February 17, 2026 19:06
jlaneve and others added 3 commits February 18, 2026 21:03
…an project root

- Fix CDN base URLs: constraintsBaseURL → cdn.astronomer.io/runtime-constraints
- Add freezeBaseURL (cdn.astronomer.io/runtime-freeze) for full 191-package list
- getConstraints() now fetches and caches both constraints + freeze files;
  returns freeze path for use as pip -c arg in step 1 install
- Move "already running" check to top of Start() before any install work
- Add ensureCredentials() to seed passwords file with admin:admin on first run
- Add readCredentials() to display username/password in startup output
- Redirect AIRFLOW_HOME → .astro/standalone/ so airflow.cfg, airflow.db,
  and logs/ all live there instead of cluttering the project root
- Set AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_PASSWORDS_FILE to .astro/standalone/
- Update Kill() to clean up .venv/ and .astro/standalone/ only
- Update tests: freeze file routing in fetch mock, AIRFLOW_HOME assertions,
  new TestStandaloneEnsureCredentials, TestStandaloneReadCredentials tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Standalone.Build() stub (errStandaloneNotSupported) to satisfy
  the updated ContainerHandler interface which gained a Build method in main
- Resolve conflict in cmd/airflow_test.go: keep both TestAirflowStandalone
  and new TestAirflowBuild from main, preserving all subtests from both

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rename the command from 'standalone' to 'local' for a more intuitive UX.
The internal Go types (Standalone, StandaloneHandlerInit) are unchanged.

- cmd/airflow.go: Use:"standalone" → Use:"local", all function/var names
- cmd/airflow_hooks.go: EnsureStandaloneRuntime → EnsureLocalRuntime
- cmd/airflow_test.go: update all test names and assertions
- airflow/standalone.go: add Build() stub (satisfies updated ContainerHandler)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jlaneve jlaneve changed the title feat: add astro dev standalone for Docker-free local development feat: add astro dev local for Docker-free local development Feb 19, 2026
jlaneve and others added 11 commits February 18, 2026 21:29
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse the optional -python-X.Y suffix from runtime image tags
(e.g. 3.1-12-python-3.11) instead of hardcoding Python 3.12.
Falls back to 3.12 (the default for all Runtime 3.x images)
when no suffix is present.

Cache filenames now include the Python version to avoid stale
lookups when switching between Python versions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lution

- Fix standaloneExecAirflowCommand using non-existent .venv/bin/bash;
  use system bash instead (venv PATH is already set in env)
- Fix .env file overriding standalone-critical vars (AIRFLOW_HOME, etc.);
  .env is now applied first, then critical settings override on top
- Add 3-tier Python version resolution: Dockerfile tag → runtime JSON → fallback 3.12
- Add DefaultPythonVersion/PythonVersions fields to RuntimeVersionMetadata
- Add GetDefaultPythonVersion() to look up Python version from updates.astronomer.io

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s subcommand

- Remove redundant "already running" check in startBackground (Start already guards it)
- Strip matching quotes from .env values to match Docker Compose behavior
- Add --port flag and config.CFG.APIServerPort fallback for custom webserver port
- Add `astro dev local ps` subcommand to check process status
- Make readCredentials deterministic by preferring the admin user

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove comments that restate the code, duplicate doc comments from
elsewhere, or are stale context from incremental development. Keep
numbered steps in Start() and comments that explain non-obvious "why".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
config.CFG.APIServerPort.GetString() returns "0" (not "") when the port
is not configured in .astro/config.yaml. This caused the health check to
hit localhost:0 and time out. Add explicit check for "0" in webserverPort().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Settings import runs airflow CLI commands (connections list, variables
list) that require the database to be initialized. Previously settings
were applied before `airflow standalone` was started, causing the DB to
not exist yet. Move applySettings to after the health check passes in
both foreground and background modes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve marked this pull request as ready for review February 24, 2026 19:58
@jlaneve jlaneve requested a review from a team as a code owner February 24, 2026 19:58
Replace the separate `astro dev local` command tree with a unified
mode-based approach. A `dev.mode` project config ("docker" or
"standalone") controls which backend is used, with `--standalone` and
`--docker` persistent flags for per-call overrides. All existing
commands (start, stop, logs, ps, kill, restart) now work in both modes.

- Add `dev.mode` config with "docker" default and validator
- Add `--standalone`/`--docker` mutually exclusive persistent flags
- Add resolveDevMode/resolveHandlerInit for mode-based handler dispatch
- Update all pre-run hooks to skip Docker in standalone mode
- Add --foreground/-f and --port/-p flags to start and restart
- Remove `astro dev local` command tree entirely
- Update user-facing strings from `astro dev local` to `astro dev`
- Update command descriptions to be mode-agnostic
- build/upgrade-test always use Docker regardless of mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the separate `astro dev local` command tree with a unified
mode-based approach. A `dev.mode` project config ("docker" or
"standalone") controls which backend is used, with `--standalone` and
`--docker` persistent flags for per-call overrides. All existing
commands (start, stop, logs, ps, kill, restart) now work in both modes.

- Add `dev.mode` config with "docker" default and validator
- Add `--standalone`/`--docker` mutually exclusive persistent flags
- Add resolveDevMode/resolveHandlerInit for mode-based handler dispatch
- Update all pre-run hooks to skip Docker in standalone mode
- Add --foreground/-f and --port/-p flags to start and restart
- Remove `astro dev local` command tree entirely
- Update user-facing strings from `astro dev local` to `astro dev`
- Update command descriptions to be mode-agnostic
- build/upgrade-test always use Docker regardless of mode

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve changed the title feat: add astro dev local for Docker-free local development feat: add standalone mode for Docker-free local development Feb 24, 2026
jlaneve and others added 6 commits February 24, 2026 16:24
Instead of scattering isStandaloneMode() checks in every hook,
assign a noOpContainerRuntime in standalone mode so containerRuntime
is never nil. Downstream hooks work unchanged — the no-op methods
just return nil. Eliminates a nil-pointer landmine for future contributors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a custom usage template that splits flags into Common, Docker Mode,
and Standalone Mode sections using cobra flag annotations. Applied to
the start and restart commands which have mode-specific flags. Other
commands keep the default template.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, parse, settings)

Implement 6 ContainerHandler methods on the Standalone struct that
previously returned errStandaloneNotSupported:

- Run: executes arbitrary commands in the venv environment
- Bash: opens an interactive shell with venv env (AIRFLOW_HOME, PATH)
- ImportSettings: imports connections/variables/pools from settings file
- ExportSettings: exports to settings file or .env file
- Pytest: runs pytest on DAGs with file path resolution and args passthrough
- Parse: validates DAGs by running the default integrity test

Add ensureVenv() helper to validate the venv exists before running
commands, and resolveInEnvPath() to look up binaries in the env's PATH
(needed because exec.Command resolves using the parent process's PATH,
not cmd.Env).

Update error messages for Build, ComposeExport, and UpgradeTest to be
more descriptive about why they're unavailable in standalone mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map the existing --scheduler, --triggerer, --api-server, --webserver,
and --dag-processor flags to standalone log prefixes so users can filter
log output by component. The --webserver flag maps to api-server since
AF3 standalone has no separate webserver process.

When no flags are passed, all lines are shown (including standalone
orchestrator startup messages). When any component filter is active,
only matching lines appear.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… mode dispatch

Pre-flight check dials localhost:<port> before starting to prevent
silently connecting to the wrong Airflow instance. Also routes build
and upgrade-test commands through resolveHandlerInit() so they properly
error in standalone mode, and fixes the custom port env var to use the
AF3 config key (AIRFLOW__API__PORT).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Contributor

@jeremybeard jeremybeard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example_astronauts DAG from the init project doesn't seem to want to progress for me, does it work for you? Another example DAG of mine finished normally though.

Image

Also wondering, are we hard limited in this mode to a concurrency of one? It can make getting through large DAGs very slow.

jlaneve and others added 4 commits February 25, 2026 15:02
Move standalone mode implementation and tests to standalone.go and
standalone_test.go to better reflect their contents.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Floating tags like "3.1" (without the -Z patch) are now resolved to
the latest pinned version (e.g., "3.1-12") via the runtime versions
JSON from updates.astronomer.io. Unrecognizable tags (custom images)
get a clear error explaining that a pinned runtime image is required.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match Docker mode behavior by setting SIMPLE_AUTH_MANAGER_ALL_ADMINS=True
instead of requiring login with seeded credentials.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match Docker image build log behavior — hide uv install output by
default and only show it when --verbose / debug logging is active.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve self-assigned this Feb 25, 2026
jlaneve and others added 4 commits February 25, 2026 15:16
# Conflicts:
#	config/config.go
#	config/types.go
The LocalExecutor communicates with the api-server via the execution API.
Without this, custom ports cause Connection refused errors because the
executor defaults to localhost:8080 while the api-server is elsewhere.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ions in standalone mode

Add an experimental notice when starting standalone mode. Fix a goroutine
leak in startForeground where the signal handler blocked forever on normal
exit. Replace concrete type assertions (containerHandler.(*Standalone))
with a SetStartOpts interface method so the cmd layer no longer depends on
the Standalone concrete type.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Python's _scproxy calls SCDynamicStoreCopyProxies which is not fork-safe.
When Airflow's LocalExecutor forks, this can spin at 100% CPU indefinitely.
Setting NO_PROXY=* tells Python to skip _scproxy entirely. We only do this
when no proxy is configured (env vars, .env file, or macOS system settings)
so corporate proxy users aren't affected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s struct

Fold the 8 positional Start parameters and the separate SetStartOpts
call into a single StartOptions struct, reducing the interface surface
and eliminating the two-step setup pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jlaneve jlaneve force-pushed the feat/standalone-mode branch from aadd5b9 to b7d7123 Compare March 2, 2026 17:21
@jlaneve jlaneve merged commit fac9949 into main Mar 3, 2026
5 of 6 checks passed
@jlaneve jlaneve deleted the feat/standalone-mode branch March 3, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants