{"id":4963,"date":"2026-01-26T01:51:08","date_gmt":"2026-01-26T06:51:08","guid":{"rendered":"https:\/\/chubes.net\/?documentation=database-schema"},"modified":"2026-04-27T00:09:32","modified_gmt":"2026-04-27T04:09:32","slug":"database-schema","status":"publish","type":"documentation","link":"https:\/\/chubes.net\/docs\/data-machine\/core-system\/database-schema\/","title":{"rendered":"Database Schema"},"content":{"rendered":"<p>Data Machine uses eight core tables for managing pipelines, flows, jobs, agents, access control, deduplication tracking, chat sessions, and centralized logging.<\/p><h2 class=\"wp-block-heading\">Core Tables<\/h2><h3 class=\"wp-block-heading\"><code>wp_datamachine_pipelines<\/code><\/h3><p><strong>Purpose<\/strong>: Reusable workflow templates<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_pipelines (\n    pipeline_id bigint(20) unsigned NOT NULL AUTO_INCREMENT,\n    user_id bigint(20) unsigned NOT NULL DEFAULT 0,\n    agent_id bigint(20) unsigned DEFAULT NULL,\n    pipeline_name varchar(255) NOT NULL,\n    pipeline_config longtext NULL,\n    created_at datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    updated_at datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n    PRIMARY KEY (pipeline_id),\n    KEY user_id (user_id),\n    KEY agent_id (agent_id),\n    KEY pipeline_name (pipeline_name),\n    KEY created_at (created_at),\n    KEY updated_at (updated_at)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>pipeline_id<\/code> &#8211; Auto-increment primary key<\/li><li><code>user_id<\/code> &#8211; WordPress user who created the pipeline<\/li><li><code>agent_id<\/code> &#8211; Agent this pipeline belongs to (multi-agent scoping, @since v0.36.1)<\/li><li><code>pipeline_name<\/code> &#8211; Human-readable pipeline name<\/li><li><code>pipeline_config<\/code> &#8211; JSON configuration containing step definitions<\/li><li><code>created_at<\/code> &#8211; Creation timestamp<\/li><li><code>updated_at<\/code> &#8211; Last modification timestamp<\/li><\/ul><h3 class=\"wp-block-heading\"><code>wp_datamachine_flows<\/code><\/h3><p><strong>Purpose<\/strong>: Scheduled instances of pipelines with specific configurations<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_flows (\n    flow_id bigint(20) unsigned NOT NULL AUTO_INCREMENT,\n    pipeline_id bigint(20) unsigned NOT NULL,\n    user_id bigint(20) unsigned NOT NULL DEFAULT 0,\n    agent_id bigint(20) unsigned DEFAULT NULL,\n    flow_name varchar(255) NOT NULL,\n    flow_config longtext NOT NULL,\n    scheduling_config longtext NOT NULL,\n    PRIMARY KEY (flow_id),\n    KEY pipeline_id (pipeline_id),\n    KEY user_id (user_id),\n    KEY agent_id (agent_id)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>flow_id<\/code> &#8211; Auto-increment primary key<\/li><li><code>pipeline_id<\/code> &#8211; Reference to parent pipeline<\/li><li><code>user_id<\/code> &#8211; WordPress user who created the flow<\/li><li><code>agent_id<\/code> &#8211; Agent this flow belongs to (multi-agent scoping, @since v0.36.1)<\/li><li><code>flow_name<\/code> &#8211; Instance-specific name<\/li><li><code>flow_config<\/code> &#8211; JSON configuration with flow-specific settings<\/li><li><code>scheduling_config<\/code> &#8211; Scheduling rules and automation settings<\/li><\/ul><h3 class=\"wp-block-heading\"><code>wp_datamachine_jobs<\/code><\/h3><p><strong>Purpose<\/strong>: Individual execution records<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_jobs (\n    job_id bigint(20) unsigned NOT NULL AUTO_INCREMENT,\n    user_id bigint(20) unsigned NOT NULL DEFAULT 0,\n    agent_id bigint(20) unsigned DEFAULT NULL,\n    pipeline_id varchar(20) NULL DEFAULT NULL,\n    flow_id varchar(20) NULL DEFAULT NULL,\n    source varchar(50) NOT NULL DEFAULT &#039;pipeline&#039;,\n    label varchar(255) NULL DEFAULT NULL,\n    parent_job_id bigint(20) unsigned NULL DEFAULT NULL,\n    status varchar(255) NOT NULL,\n    engine_data longtext NULL,\n    created_at datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    completed_at datetime NULL DEFAULT NULL,\n    PRIMARY KEY (job_id),\n    KEY status (status),\n    KEY pipeline_id (pipeline_id),\n    KEY flow_id (flow_id),\n    KEY source (source),\n    KEY parent_job_id (parent_job_id),\n    KEY user_id (user_id),\n    KEY agent_id (agent_id)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>job_id<\/code> &#8211; Auto-increment primary key<\/li><li><code>user_id<\/code> &#8211; WordPress user who triggered the job<\/li><li><code>agent_id<\/code> &#8211; Agent this job belongs to (multi-agent scoping, @since v0.36.1)<\/li><li><code>pipeline_id<\/code> &#8211; Reference to source pipeline, or <code>'direct'<\/code> for ephemeral execution mode<\/li><li><code>flow_id<\/code> &#8211; Reference to flow that created this job, or <code>'direct'<\/code> for ephemeral execution mode<\/li><li><code>source<\/code> &#8211; Execution source: <code>'pipeline'<\/code> (standard), <code>'system'<\/code> (system tasks), or <code>'direct'<\/code> (ephemeral)<\/li><li><code>label<\/code> &#8211; Human-readable label for the job (used by system tasks)<\/li><li><code>parent_job_id<\/code> &#8211; Reference to parent job for batch execution (child jobs link back to parent)<\/li><li><code>status<\/code> &#8211; Current execution status (varchar(255) supports compound statuses like <code>agent_skipped - reason<\/code>)<\/li><li><code>engine_data<\/code> &#8211; Engine parameters (source_url, image_url, effects for undo) stored by handlers for downstream use<\/li><li><code>created_at<\/code> &#8211; Job creation timestamp<\/li><li><code>completed_at<\/code> &#8211; Completion timestamp<\/li><\/ul><p><strong>Note:<\/strong> <code>pipeline_id<\/code> and <code>flow_id<\/code> are <code>varchar(20)<\/code> to support both numeric IDs and the <code>'direct'<\/code> sentinel value for ephemeral workflows.<\/p><h3 class=\"wp-block-heading\"><code>wp_datamachine_processed_items<\/code><\/h3><p><strong>Purpose<\/strong>: Deduplication tracking to prevent duplicate processing<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_processed_items (\n    id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,\n    flow_step_id VARCHAR(255) NOT NULL,\n    source_type VARCHAR(50) NOT NULL,\n    item_identifier VARCHAR(255) NOT NULL,\n    job_id BIGINT(20) UNSIGNED NOT NULL,\n    processed_timestamp DATETIME DEFAULT CURRENT_TIMESTAMP NOT NULL,\n    PRIMARY KEY (id),\n    UNIQUE KEY `flow_source_item` (flow_step_id, source_type, item_identifier(191)),\n    KEY `flow_step_id` (flow_step_id),\n    KEY `source_type` (source_type),\n    KEY `job_id` (job_id),\n    KEY `flow_source_ts` (flow_step_id, source_type, processed_timestamp)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>id<\/code> &#8211; Auto-increment primary key<\/li><li><code>flow_step_id<\/code> &#8211; Composite identifier: <code>{pipeline_step_id}_{flow_id}<\/code><\/li><li><code>source_type<\/code> &#8211; Handler type (rss, wordpress_local, reddit, etc.)<\/li><li><code>item_identifier<\/code> &#8211; Unique identifier within source type<\/li><li><code>job_id<\/code> &#8211; Job that processed this item<\/li><li><code>processed_timestamp<\/code> &#8211; Processing timestamp<\/li><\/ul><p><strong>Indexes<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>flow_source_item<\/code> (UNIQUE) \u2014 point lookups + dedupe constraint.<\/li><li><code>flow_step_id<\/code>, <code>source_type<\/code>, <code>job_id<\/code> \u2014 bulk deletes and filtered audits.<\/li><li><code>flow_source_ts<\/code> (since 0.71.0) \u2014 covers time-windowed range scans used by <code>find_stale()<\/code> \/ <code>has_been_processed_within()<\/code>. <code>ProcessedItems::ensure_flow_source_ts_index()<\/code> backfills the index on existing installs since <code>dbDelta<\/code> does not reliably add indexes to populated tables.<\/li><\/ul><h3 class=\"wp-block-heading\"><code>wp_datamachine_chat_sessions<\/code><\/h3><p><strong>Purpose<\/strong>: Persistent conversation state for chat API with multi-turn conversation support<\/p><p><strong>Implementation<\/strong>: <code>inc\/Core\/Database\/Chat\/Chat.php<\/code><\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_chat_sessions (\n    session_id VARCHAR(50) NOT NULL,\n    user_id BIGINT(20) UNSIGNED NOT NULL,\n    agent_id BIGINT(20) UNSIGNED NULL,\n    title VARCHAR(100) NULL,\n    messages LONGTEXT NOT NULL COMMENT &#039;JSON array of conversation messages&#039;,\n    metadata LONGTEXT NULL COMMENT &#039;JSON object for session metadata&#039;,\n    provider VARCHAR(50) NULL COMMENT &#039;AI provider (anthropic, openai, etc)&#039;,\n    model VARCHAR(100) NULL COMMENT &#039;AI model identifier&#039;,\n    context VARCHAR(20) NOT NULL DEFAULT &#039;chat&#039;,\n    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n    expires_at DATETIME NULL COMMENT &#039;Auto-cleanup timestamp&#039;,\n    PRIMARY KEY (session_id),\n    KEY user_id (user_id),\n    KEY agent_id (agent_id),\n    KEY context (context),\n    KEY user_context (user_id, context),\n    KEY created_at (created_at),\n    KEY updated_at (updated_at),\n    KEY expires_at (expires_at)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>session_id<\/code> &#8211; UUID4 session identifier (primary key)<\/li><li><code>user_id<\/code> &#8211; WordPress user ID (user-scoped isolation)<\/li><li><code>agent_id<\/code> &#8211; Agent this session belongs to (multi-agent scoping, @since v0.36.1)<\/li><li><code>title<\/code> &#8211; Auto-generated session title<\/li><li><code>messages<\/code> &#8211; JSON array of conversation messages (chronological ordering)<\/li><li><code>metadata<\/code> &#8211; JSON object with message_count, last_activity timestamps<\/li><li><code>provider<\/code> &#8211; AI provider used for session (optional, tracked for continuity)<\/li><li><code>model<\/code> &#8211; AI model used for session (optional, tracked for continuity)<\/li><li><code>context<\/code> &#8211; Session context: <code>'chat'<\/code> (default) or <code>'pipeline'<\/code><\/li><li><code>created_at<\/code> &#8211; Session creation timestamp<\/li><li><code>updated_at<\/code> &#8211; Last activity timestamp<\/li><li><code>expires_at<\/code> &#8211; Expiration timestamp (24-hour default timeout)<\/li><\/ul><p><strong>Session Management<\/strong>:<\/p><ul class=\"wp-block-list\"><li>User-scoped and agent-scoped session isolation<\/li><li>Automatic session creation on first message<\/li><li>Session expiration with cleanup mechanism<\/li><li>Metadata tracking for message count and activity timestamps<\/li><\/ul><h3 class=\"wp-block-heading\"><code>wp_datamachine_agents<\/code><\/h3><p><strong>Purpose<\/strong>: Agent registry for multi-agent architecture (@since v0.36.1)<\/p><p><strong>Implementation<\/strong>: <code>inc\/Core\/Database\/Agents\/Agents.php<\/code><\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_agents (\n    agent_id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,\n    agent_slug VARCHAR(200) NOT NULL,\n    agent_name VARCHAR(200) NOT NULL,\n    owner_id BIGINT(20) UNSIGNED NOT NULL,\n    agent_config LONGTEXT NULL,\n    status VARCHAR(20) NOT NULL DEFAULT &#039;active&#039;,\n    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n    PRIMARY KEY (agent_id),\n    UNIQUE KEY agent_slug (agent_slug),\n    KEY owner_id (owner_id),\n    KEY status (status)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>agent_id<\/code> &#8211; Auto-increment primary key<\/li><li><code>agent_slug<\/code> &#8211; Unique identifier for filesystem directory naming and CLI reference<\/li><li><code>agent_name<\/code> &#8211; Human-readable display name<\/li><li><code>owner_id<\/code> &#8211; WordPress user ID of the agent&#8217;s creator\/owner<\/li><li><code>agent_config<\/code> &#8211; JSON configuration (AI provider preferences, model settings)<\/li><li><code>status<\/code> &#8211; Agent status: <code>'active'<\/code> or <code>'inactive'<\/code><\/li><li><code>created_at<\/code> &#8211; Creation timestamp<\/li><li><code>updated_at<\/code> &#8211; Last modification timestamp<\/li><\/ul><h3 class=\"wp-block-heading\"><code>wp_datamachine_agent_access<\/code><\/h3><p><strong>Purpose<\/strong>: Role-based access control for multi-agent resource sharing (@since v0.36.1)<\/p><p><strong>Implementation<\/strong>: <code>inc\/Core\/Database\/Agents\/AgentAccess.php<\/code><\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_agent_access (\n    id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,\n    agent_id BIGINT(20) UNSIGNED NOT NULL,\n    user_id BIGINT(20) UNSIGNED NOT NULL,\n    role VARCHAR(20) NOT NULL DEFAULT &#039;viewer&#039;,\n    granted_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    PRIMARY KEY (id),\n    UNIQUE KEY agent_user (agent_id, user_id),\n    KEY agent_id (agent_id),\n    KEY user_id (user_id),\n    KEY role (role)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>id<\/code> &#8211; Auto-increment primary key<\/li><li><code>agent_id<\/code> &#8211; Reference to the agent<\/li><li><code>user_id<\/code> &#8211; WordPress user who has access<\/li><li><code>role<\/code> &#8211; Access level: <code>'viewer'<\/code> (read), <code>'operator'<\/code> (read + execute), <code>'admin'<\/code> (full control)<\/li><li><code>granted_at<\/code> &#8211; When access was granted<\/li><\/ul><p><strong>Role Hierarchy<\/strong>: <code>viewer<\/code> (0) &lt; <code>operator<\/code> (1) &lt; <code>admin<\/code> (2). Higher roles inherit all lower-level permissions.<\/p><h3 class=\"wp-block-heading\"><code>wp_datamachine_logs<\/code><\/h3><p><strong>Purpose<\/strong>: Centralized system logs for all agent activity (@since v0.4.0)<\/p><p><strong>Implementation<\/strong>: <code>inc\/Core\/Database\/Logs\/LogRepository.php<\/code><\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">sql<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-sql\"><code class=\"language-sql\">CREATE TABLE wp_datamachine_logs (\n    id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,\n    agent_id BIGINT(20) UNSIGNED DEFAULT NULL,\n    user_id BIGINT(20) UNSIGNED DEFAULT NULL,\n    level VARCHAR(20) NOT NULL DEFAULT &#039;info&#039;,\n    message TEXT NOT NULL,\n    context LONGTEXT DEFAULT NULL,\n    created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,\n    PRIMARY KEY (id),\n    KEY idx_agent_time (agent_id, created_at),\n    KEY idx_level_time (level, created_at),\n    KEY idx_created_at (created_at)\n);<\/code><\/pre><\/div><p><strong>Fields<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>id<\/code> &#8211; Auto-increment primary key<\/li><li><code>agent_id<\/code> &#8211; Agent context for the log entry (nullable)<\/li><li><code>user_id<\/code> &#8211; WordPress user context (nullable)<\/li><li><code>level<\/code> &#8211; Log severity: <code>'debug'<\/code>, <code>'info'<\/code>, <code>'warning'<\/code>, <code>'error'<\/code>, <code>'critical'<\/code><\/li><li><code>message<\/code> &#8211; Human-readable log message<\/li><li><code>context<\/code> &#8211; JSON context data (job_id, flow_id, stack traces, etc.)<\/li><li><code>created_at<\/code> &#8211; Log timestamp<\/li><\/ul><h2 class=\"wp-block-heading\">Relationships<\/h2><h3 class=\"wp-block-heading\">Primary Relationships<\/h3><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\"><\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code\"><code>Agent (1) \u2500\u2500\u2192 Pipeline (many)\n           \u2500\u2500\u2192 Flow (many)\n           \u2500\u2500\u2192 Job (many)\n           \u2500\u2500\u2192 ChatSession (many)\n\nAgentAccess: Agent (1) \u2190\u2192 User (many)  [role-based]\n\nPipeline (1) \u2192 Flow (many) \u2192 Job (many)\n                \u2193\n            ProcessedItems (many)\n\nJob (1) \u2192 ChildJob (many)  [via parent_job_id, batch execution]\n\nUser (1) \u2192 ChatSession (many)<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Key Identifiers<\/h3><p><strong>Pipeline Step ID<\/strong>: UUID4 for cross-flow step referencing<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$pipeline_step_id = &#039;xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx&#039;;<\/code><\/pre><\/div><p><strong>Flow Step ID<\/strong>: Composite identifier for flow-specific tracking<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$flow_step_id = $pipeline_step_id . &#039;_&#039; . $flow_id;<\/code><\/pre><\/div><p><strong>Agent Slug<\/strong>: Unique string identifier for filesystem and CLI<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$agent_slug = &#039;my-agent&#039;; \/\/ maps to agents\/{slug}\/ directory<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Database Operations<\/h2><h3 class=\"wp-block-heading\">Pipeline Operations<\/h3><p><strong>Create Pipeline<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$pipeline_id = $db_pipelines-&gt;create_pipeline([\n    &#039;pipeline_name&#039; =&gt; &#039;RSS to Twitter&#039;,\n    &#039;pipeline_config&#039; =&gt; $config_json\n]);<\/code><\/pre><\/div><p><strong>Get Pipeline Config<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$config = $db_pipelines-&gt;get_pipeline_config($pipeline_id);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Flow Operations<\/h3><p><strong>Create Flow<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$flow_id = $db_flows-&gt;create_flow([\n    &#039;pipeline_id&#039; =&gt; $pipeline_id,\n    &#039;flow_name&#039; =&gt; &#039;Morning Posts&#039;,\n    &#039;flow_config&#039; =&gt; $flow_config_json\n]);<\/code><\/pre><\/div><p><strong>Get Flow Config<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$config = apply_filters(&#039;datamachine_get_flow_config&#039;, [], $flow_id);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Job Operations<\/h3><p><strong>Create Job<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$job_id = $db_jobs-&gt;create_job([\n    &#039;pipeline_id&#039; =&gt; $pipeline_id,\n    &#039;flow_id&#039; =&gt; $flow_id\n]);<\/code><\/pre><\/div><p><strong>Create Child Job<\/strong> (batch execution):<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$child_job_id = $db_jobs-&gt;create_job([\n    &#039;pipeline_id&#039; =&gt; $pipeline_id,\n    &#039;flow_id&#039; =&gt; $flow_id,\n    &#039;parent_job_id&#039; =&gt; $parent_job_id\n]);<\/code><\/pre><\/div><p><strong>Fail Job<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Abilities API\n$ability = wp_get_ability( &#039;datamachine\/fail-job&#039; );\n$ability-&gt;execute( [ &#039;job_id&#039; =&gt; $job_id, &#039;reason&#039; =&gt; &#039;Processing failed: timeout exceeded&#039; ] );\n\n\/\/ Action Hook (for extensibility)\ndo_action(&#039;datamachine_update_job_status&#039;, $job_id, &#039;failed&#039;, &#039;Processing failed: timeout exceeded&#039;);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Processed Items<\/h3><p><strong>Mark Item Processed<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$context-&gt;markItemProcessed($item_id);<\/code><\/pre><\/div><p><strong>Check If Processed<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$is_processed = $context-&gt;isItemProcessed($item_id);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Chat Session Operations<\/h3><p><strong>Create Session<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">use DataMachineCoreDatabaseChatChat as ChatDatabase;\n\n$chat_db = new ChatDatabase();\n$session_id = $chat_db-&gt;create_session($user_id, [\n    &#039;started_at&#039; =&gt; current_time(&#039;mysql&#039;),\n    &#039;message_count&#039; =&gt; 0\n]);<\/code><\/pre><\/div><p><strong>Get Session<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$session = $chat_db-&gt;get_session($session_id);\n\/\/ Returns: [&#039;session_id&#039;, &#039;user_id&#039;, &#039;agent_id&#039;, &#039;title&#039;, &#039;messages&#039;, &#039;metadata&#039;,\n\/\/           &#039;provider&#039;, &#039;model&#039;, &#039;context&#039;, &#039;created_at&#039;, &#039;updated_at&#039;, &#039;expires_at&#039;]<\/code><\/pre><\/div><p><strong>Update Session<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$chat_db-&gt;update_session(\n    $session_id,\n    $messages,  \/\/ Complete messages array\n    $metadata,  \/\/ Updated metadata\n    $provider,  \/\/ AI provider\n    $model      \/\/ AI model\n);<\/code><\/pre><\/div><p><strong>Cleanup Expired Sessions<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$deleted_count = $chat_db-&gt;cleanup_expired_sessions();<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Agent Operations<\/h3><p><strong>Create Agent<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$result = wp_execute_ability(&#039;datamachine\/create-agent&#039;, [\n    &#039;slug&#039; =&gt; &#039;my-agent&#039;,\n    &#039;name&#039; =&gt; &#039;My Agent&#039;,\n]);<\/code><\/pre><\/div><p><strong>Check Access<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$agent_access = new DataMachineCoreDatabaseAgentsAgentAccess();\n$can_access = $agent_access-&gt;user_can_access($agent_id, $user_id, &#039;operator&#039;);<\/code><\/pre><\/div><p><strong>Grant Access<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$agent_access-&gt;grant_access($agent_id, $user_id, &#039;operator&#039;);<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Configuration Storage<\/h2><h3 class=\"wp-block-heading\">Pipeline Config Structure<\/h3><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">json<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-json\"><code class=\"language-json\">{\n    &quot;step_uuid_1&quot;: {\n        &quot;step_type&quot;: &quot;fetch&quot;,\n        &quot;handler&quot;: &quot;rss&quot;,\n        &quot;execution_order&quot;: 0,\n        &quot;system_prompt&quot;: &quot;AI instructions...&quot;,\n        &quot;handler_config&quot;: {\n            &quot;rss_url&quot;: &quot;https:\/\/example.com\/feed.xml&quot;\n        }\n    },\n    &quot;step_uuid_2&quot;: {\n        &quot;step_type&quot;: &quot;publish&quot;,\n        &quot;handler&quot;: &quot;twitter&quot;,\n        &quot;execution_order&quot;: 1,\n        &quot;handler_config&quot;: {\n            &quot;twitter_include_source&quot;: true\n        }\n    }\n}<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Flow Config Structure<\/h3><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">json<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-json\"><code class=\"language-json\">{\n    &quot;step_uuid_1_123&quot;: {\n        &quot;prompt_queue&quot;: [\n            {\n                &quot;prompt&quot;: &quot;Custom prompt for this flow instance...&quot;,\n                &quot;added_at&quot;: &quot;2026-04-26T00:00:00+00:00&quot;\n            }\n        ],\n        &quot;queue_mode&quot;: &quot;static&quot;,\n        &quot;execution_order&quot;: 0\n    },\n    &quot;step_uuid_2_123&quot;: {\n        &quot;execution_order&quot;: 1\n    }\n}<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Data Access Patterns<\/h2><h3 class=\"wp-block-heading\">Service Discovery<\/h3><p>All database operations use direct repository instantiation:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$db_pipelines      = new DataMachineCoreDatabasePipelinesPipelines();\n$db_flows          = new DataMachineCoreDatabaseFlowsFlows();\n$db_jobs           = new DataMachineCoreDatabaseJobsJobs();\n$db_processed_items = new DataMachineCoreDatabaseProcessedItemsProcessedItems();\n$db_agents         = new DataMachineCoreDatabaseAgentsAgents();\n$db_agent_access   = new DataMachineCoreDatabaseAgentsAgentAccess();\n$db_logs           = new DataMachineCoreDatabaseLogsLogRepository();\n$db_chat           = new DataMachineCoreDatabaseChatChat();<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Transactional Operations<\/h3><p>Database operations maintain referential integrity through foreign key constraints and cascading deletes.<\/p><p><strong>Pipeline Deletion<\/strong>: Automatically removes associated flows, jobs, and processed items\n<strong>Flow Deletion<\/strong>: Automatically removes associated jobs and processed items\n<strong>Job Deletion<\/strong>: Sets processed items job_id to NULL\n<strong>Agent Deletion<\/strong>: Removes agent_access grants; optionally removes filesystem directory<\/p><h3 class=\"wp-block-heading\">Multi-Agent Scoping<\/h3><p>All major tables carry <code>user_id<\/code> and <code>agent_id<\/code> columns for resource isolation:<\/p><ul class=\"wp-block-list\"><li>Queries filter by <code>agent_id<\/code> when in multi-agent context<\/li><li>Single-agent mode uses <code>agent_id = NULL<\/code> or <code>0<\/code> (valid state)<\/li><li>Migration (<code>datamachine_backfill_agent_ids<\/code>) backfills existing resources when agents are created<\/li><\/ul><h2 class=\"wp-block-heading\">Indexing Strategy<\/h2><h3 class=\"wp-block-heading\">Performance Indexes<\/h3><ul class=\"wp-block-list\"><li><strong>Pipeline Name<\/strong> &#8211; Fast pipeline lookups by name<\/li><li><strong>Flow Pipeline ID<\/strong> &#8211; Efficient flow-to-pipeline joins<\/li><li><strong>Job Status<\/strong> &#8211; Quick job status filtering<\/li><li><strong>Job Parent ID<\/strong> &#8211; Fast batch child-job lookups<\/li><li><strong>Job Source<\/strong> &#8211; Filter by execution source (pipeline, system, direct)<\/li><li><strong>Processed Items Composite<\/strong> &#8211; Fast deduplication checks<\/li><li><strong>Timestamp Indexes<\/strong> &#8211; Chronological queries and cleanup<\/li><li><strong>Agent Slug (unique)<\/strong> &#8211; Fast agent lookups by slug<\/li><li><strong>Agent\/Time Composite<\/strong> &#8211; Log filtering by agent and time range<\/li><li><strong>Level\/Time Composite<\/strong> &#8211; Log filtering by severity<\/li><\/ul><h3 class=\"wp-block-heading\">Query Optimization<\/h3><ul class=\"wp-block-list\"><li><strong>Prepared Statements<\/strong> &#8211; All queries use wpdb::prepare()<\/li><li><strong>Selective Columns<\/strong> &#8211; Only required columns retrieved<\/li><li><strong>Proper Limits<\/strong> &#8211; Pagination for large result sets<\/li><li><strong>Index Hints<\/strong> &#8211; Strategic use of composite indexes<\/li><\/ul><h2 class=\"wp-block-heading\">Migration History<\/h2><h3 class=\"wp-block-heading\">Key Schema Migrations<\/h3><figure class=\"wp-block-table\"><table><thead><tr><th>Version<\/th><th>Migration<\/th><th>Description<\/th><\/tr><\/thead><tbody><tr><td>v0.4.0<\/td><td><code>LogRepository::create_table()<\/code><\/td><td>Added centralized logs table<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>datamachine_migrate_to_layered_architecture()<\/code><\/td><td>Created agents table, migrated filesystem to three-layer dirs<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>AgentAccess::create_table()<\/code><\/td><td>Added role-based access control<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>Pipelines::migrate_columns()<\/code><\/td><td>Added <code>user_id<\/code>, <code>agent_id<\/code> to pipelines<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>Flows::migrate_columns()<\/code><\/td><td>Added <code>user_id<\/code>, <code>agent_id<\/code> to flows<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>Jobs::migrate_columns()<\/code><\/td><td>Added <code>user_id<\/code>, <code>agent_id<\/code>, <code>source<\/code>, <code>label<\/code>, <code>parent_job_id<\/code> to jobs<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>Chat::ensure_agent_id_column()<\/code><\/td><td>Added <code>agent_id<\/code>, <code>title<\/code>, <code>context<\/code> to chat_sessions<\/td><\/tr><tr><td>v0.36.1<\/td><td><code>datamachine_backfill_agent_ids()<\/code><\/td><td>Backfilled agent_id on existing resources<\/td><\/tr><\/tbody><\/table><\/figure><p>Migrations run automatically via <code>datamachine_maybe_run_migrations()<\/code> on <code>init<\/code> (priority 5) whenever code version exceeds stored DB version.<\/p>","protected":false},"excerpt":{"rendered":"<p>Data Machine uses eight core tables for managing pipelines, flows, jobs, agents, access control, deduplication tracking, chat sessions, and centralized logging. Core Tables wp_datamachine_pipelines Purpose: Reusable workflow templates CREATE TABLE&#8230;<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"footnotes":""},"tags":[],"project":[506],"project_type":[484],"class_list":["post-4963","documentation","type-documentation","status-publish","hentry","project-core-system","project_type-wordpress-plugins"],"project_info":{"id":487,"name":"Data Machine","slug":"data-machine"},"project_type_info":{"id":484,"name":"WordPress Plugins","slug":"wordpress-plugins"},"_links":{"self":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4963","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation"}],"about":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/types\/documentation"}],"version-history":[{"count":7,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4963\/revisions"}],"predecessor-version":[{"id":11423,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4963\/revisions\/11423"}],"wp:attachment":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/media?parent=4963"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/tags?post=4963"},{"taxonomy":"project","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project?post=4963"},{"taxonomy":"project_type","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project_type?post=4963"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}