Overview

Relevant source files

• README.md
• go.mod
• go.sum

Purpose and Scope

This page introduces go-taskflow, a task-parallel programming framework for Go that enables developers to build and execute complex workflows with concurrent task execution and dependency management. This overview covers the framework's purpose, key features, high-level architecture, and component relationships.

For installation instructions and your first TaskFlow, see Getting Started. For detailed documentation of individual components, see Core Components.

Sources: README.md1-17

What is go-taskflow?

go-taskflow is a general-purpose task-parallel programming framework designed for Go, inspired by taskflow-cpp It provides abstractions for defining directed acyclic graphs (DAGs) of tasks with dependencies, then executing those tasks concurrently while respecting their relationships.

The framework leverages Go's native goroutines and channels to provide efficient concurrent execution without requiring external dependencies. It is designed for scenarios where work can be decomposed into discrete tasks with clear dependencies, such as data pipelines, workflow automation, and parallel algorithms.

Module: github.com/noneback/go-taskflow (Go 1.21.6+)

Sources: README.md14-15 go.mod1-3

Key Features

go-taskflow provides the following capabilities:

Feature	Description	Reference
Static Tasks	Simple function-based tasks that execute once	Core Components
Subflows	Nested TaskFlows for modular workflow organization	Subflows
Conditional Tasks	Dynamic branching based on runtime conditions	Conditional Tasks
Cyclic Flows	Loop patterns for iterative workflows	Cyclic Workflows
Priority Scheduling	HIGH/NORMAL/LOW priority levels for task ordering	Priority Scheduling
Graph Visualization	DOT format output for debugging task dependencies	Task Graph Visualization
Performance Profiling	Flamegraph generation for execution analysis	Performance Profiling
Native Concurrency	Goroutine-based execution with configurable worker pools	Executor

Sources: README.md16-28

High-Level Architecture

go-taskflow is organized into four primary layers:

API Layer

The user-facing API consists of three main abstractions defined in taskflow.go task.go and flow.go:

• TaskFlow: Container for tasks and their dependencies, represents the complete workflow graph
• Task: Individual work unit with a user-defined function
• Subflow: Nested TaskFlow created dynamically during execution
• Condition: Branching node that returns an integer to select the next task path

Graph Representation Layer

The API layer is backed by internal structures in graph.go and node.go:

• eGraph: Internal execution graph structure that manages the DAG topology
• innerNode: Internal node representation with state machine and scheduling logic
• flowBuilder: Factory for creating different node types (static, subflow, condition)

Execution Engine

The engine in executor.go orchestrates concurrent execution:

• Executor: Coordinates task scheduling and manages the worker pool
• Copool: Goroutine pool for concurrent task execution (utils/copool.go)
• Queue: Thread-safe work queue for task scheduling (utils/queue.go)

Observability Layer

Non-intrusive monitoring tools:

• Visualizer: Generates DOT format graphs (visualizer.go)
• Profiler: Tracks execution metrics and generates flamegraphs (profiler.go)

Sources: README.md59-77 executor.go17-26 graph.go10-18 node.go26-36 utils/copool.go16-27

Core Components and Code Entities

Diagram: Core component relationships showing code entity mappings

This diagram maps the conceptual components to their actual code entities in the repository. Each component is annotated with its source file location.

Sources: taskflow.go22-67 task.go1-50 flow.go1-100 executor.go17-26 graph.go10-18 node.go26-36

Task Execution Lifecycle

Diagram: Task execution lifecycle from definition to completion with code entity references

This sequence diagram shows how user code interacts with the framework's components during the complete lifecycle of task execution. Method names correspond to actual functions in the codebase.

Sources: taskflow.go22-67 executor.go40-44 executor.go189-197 graph.go55-66 node.go62-69

Concurrency Model

go-taskflow implements a work-stealing scheduler with the following characteristics:

Worker Pool Management

• The Executor creates a fixed-size goroutine pool (Copool) based on the concurrency parameter
• Workers idle on a condition variable (sync.Cond) when no work is available
• The scheduler enqueues ready tasks into a thread-safe Queue

Dependency Tracking

• Each innerNode maintains an atomic joinCounter for incoming dependencies
• When a task completes, it decrements its successors' counters using atomic operations
• When a counter reaches zero, the node becomes ready and is enqueued

Panic Handling

• Each worker goroutine includes defer/recover logic to catch panics
• A panic sets the graph's canceled atomic flag, aborting remaining work
• The Executor waits for all in-flight tasks to complete before returning

Sources: executor.go17-26 executor.go46-60 executor.go80-88 node.go34 node.go43-49 graph.go13-17 utils/copool.go65-77

Use Cases

The framework is designed for scenarios including:

Use Case	Description	Example
Data Pipelines	Multi-stage data processing with dependencies between stages	ETL workflows, data transformations
AI Agent Workflows	Sequential and parallel agent execution with decision points	Multi-agent systems, workflow automation
Parallel Algorithms	CPU-intensive tasks that can be decomposed into independent subtasks	Parallel merge sort, matrix operations
Build Systems	Dependency-driven compilation and packaging	Build orchestration, deployment pipelines

Sources: README.md30-34

Key Constraints and Behaviors

Graph Immutability During Execution

Once Executor.Run() is called, the TaskFlow enters a frozen state. Any attempt to modify the graph (add tasks, change dependencies) will result in undefined behavior. Use TaskFlow.Reset() to clear execution state for re-execution.

Error Handling Philosophy

go-taskflow follows Go's error-as-values philosophy. The framework only handles unrecovered panics by canceling the entire graph. Users are responsible for handling errors within their task functions. If graceful degradation is required, wrap task logic in panic recovery handlers.

Conditional Task Semantics

Condition nodes return an integer that selects which successor to execute. If the returned value exceeds the number of successors, no successor executes. This enables both branching and looping patterns.

Sources: README.md155-174 taskflow.go17-20

Getting Started

For a quick introduction to building your first TaskFlow, see the Quick Start Tutorial. The examples directory contains complete working examples demonstrating various patterns.

To visualize your TaskFlows and profile their execution, see Visualization and Profiling.

Sources: README.md45-135