{"id":4955,"date":"2026-01-26T01:51:07","date_gmt":"2026-01-26T06:51:07","guid":{"rendered":"https:\/\/chubes.net\/?documentation=data-machine-architecture"},"modified":"2026-04-27T00:09:31","modified_gmt":"2026-04-27T04:09:31","slug":"data-machine-architecture","status":"publish","type":"documentation","link":"https:\/\/chubes.net\/docs\/data-machine\/data-machine-architecture\/","title":{"rendered":"Data Machine Architecture"},"content":{"rendered":"<p>Data Machine is an AI-first WordPress plugin that uses a Pipeline+Flow architecture for automated content processing and publication. It provides multi-provider AI integration with tool-first design patterns, centered around a reliability-first <strong>Single Item Execution Model<\/strong>, with <strong>multi-agent support<\/strong> and a <strong>layered memory system<\/strong>.<\/p><h2 class=\"wp-block-heading\">Core Components<\/h2><h3 class=\"wp-block-heading\">Pipeline+Flow System<\/h3><ul class=\"wp-block-list\"><li><strong>Pipelines<\/strong>: Reusable templates containing step configurations<\/li><li><strong>Flows<\/strong>: Configured instances of pipelines with scheduling<\/li><li><strong>Jobs<\/strong>: Individual executions of flows with status tracking, each processing exactly one item. Support parent-child relationships for batch execution via <code>parent_job_id<\/code>.<\/li><\/ul><h3 class=\"wp-block-heading\">Execution Engine<\/h3><p>Services layer architecture with direct method calls for optimal performance. The engine implements a four-action execution cycle that processes exactly one item per job to ensure maximum reliability and isolation.<\/p><h3 class=\"wp-block-heading\">Database Schema<\/h3><p>Eight core tables:<\/p><figure class=\"wp-block-table\"><table><thead><tr><th>Table<\/th><th>Purpose<\/th><\/tr><\/thead><tbody><tr><td><code>wp_datamachine_pipelines<\/code><\/td><td>Pipeline templates (reusable), with <code>user_id<\/code> and <code>agent_id<\/code><\/td><\/tr><tr><td><code>wp_datamachine_flows<\/code><\/td><td>Flow instances (scheduled + configured), with <code>user_id<\/code> and <code>agent_id<\/code><\/td><\/tr><tr><td><code>wp_datamachine_jobs<\/code><\/td><td>Job execution records, with <code>user_id<\/code>, <code>agent_id<\/code>, <code>parent_job_id<\/code>, <code>source<\/code>, <code>label<\/code><\/td><\/tr><tr><td><code>wp_datamachine_processed_items<\/code><\/td><td>Deduplication tracking per execution<\/td><\/tr><tr><td><code>wp_datamachine_chat_sessions<\/code><\/td><td>Persistent conversation state, with <code>agent_id<\/code>, <code>title<\/code>, <code>context<\/code><\/td><\/tr><tr><td><code>wp_datamachine_agents<\/code><\/td><td>Agent registry (slug, name, owner, config, status)<\/td><\/tr><tr><td><code>wp_datamachine_agent_access<\/code><\/td><td>Role-based access control (viewer, operator, admin)<\/td><\/tr><tr><td><code>wp_datamachine_logs<\/code><\/td><td>Centralized system logs with agent scoping<\/td><\/tr><\/tbody><\/table><\/figure><p>See <a href=\"https:\/\/chubes.net\/docs\/data-machine\/core-system\/database-schema\/\">Database Schema<\/a> for full table definitions and relationships.<\/p><h3 class=\"wp-block-heading\">Multi-Agent Architecture<\/h3><p>Data Machine supports <strong>multiple agents on a single WordPress installation<\/strong> (@since v0.36.1):<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\"><\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code\"><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502                WordPress Site                    \u2502\n\u2502                                                  \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510      \u2502\n\u2502  \u2502 Agent A  \u2502  \u2502 Agent B  \u2502  \u2502 Agent C  \u2502      \u2502\n\u2502  \u2502          \u2502  \u2502          \u2502  \u2502          \u2502      \u2502\n\u2502  \u2502 SOUL.md  \u2502  \u2502 SOUL.md  \u2502  \u2502 SOUL.md  \u2502      \u2502\n\u2502  \u2502 MEMORY.md\u2502  \u2502 MEMORY.md\u2502  \u2502 MEMORY.md\u2502      \u2502\n\u2502  \u2502 daily\/   \u2502  \u2502 daily\/   \u2502  \u2502 daily\/   \u2502      \u2502\n\u2502  \u2502          \u2502  \u2502          \u2502  \u2502          \u2502      \u2502\n\u2502  \u2502 pipelines\u2502  \u2502 pipelines\u2502  \u2502 pipelines\u2502      \u2502\n\u2502  \u2502 flows    \u2502  \u2502 flows    \u2502  \u2502 flows    \u2502      \u2502\n\u2502  \u2502 jobs     \u2502  \u2502 jobs     \u2502  \u2502 jobs     \u2502      \u2502\n\u2502  \u2502 chat     \u2502  \u2502 chat     \u2502  \u2502 chat     \u2502      \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518      \u2502\n\u2502                                                  \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510   \u2502\n\u2502  \u2502 Shared Layer: SITE.md, RULES.md          \u2502   \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518   \u2502\n\u2502                                                  \u2502\n\u2502  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510  \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510                     \u2502\n\u2502  \u2502 User 1   \u2502  \u2502 User 2   \u2502                     \u2502\n\u2502  \u2502 USER.md  \u2502  \u2502 USER.md  \u2502                     \u2502\n\u2502  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518  \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518                     \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518<\/code><\/pre><\/div><p><strong>Key components:<\/strong><\/p><ul class=\"wp-block-list\"><li><strong>Agent Registry<\/strong> (<code>datamachine_agents<\/code>): Each agent has a unique slug, owner, and configuration<\/li><li><strong>Access Control<\/strong> (<code>datamachine_agent_access<\/code>): Role-based sharing (viewer &lt; operator &lt; admin)<\/li><li><strong>Resource Scoping<\/strong>: All pipelines, flows, jobs, and chat sessions carry <code>agent_id<\/code><\/li><li><strong>Filesystem Isolation<\/strong>: Each agent gets <code>agents\/{slug}\/<\/code> for identity files and daily memory<\/li><li><strong>Permission Helper<\/strong>: <code>PermissionHelper<\/code> resolves agent context and enforces access checks<\/li><\/ul><h3 class=\"wp-block-heading\">Layered Memory Architecture<\/h3><p>Agent memory is organized in a <strong>three-layer directory system<\/strong> below Data Machine&#8217;s files root in WordPress uploads:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\"><\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code\"><code>datamachine-files\/\n\u251c\u2500\u2500 shared\/              # Site-wide (all agents)\n\u2502   \u251c\u2500\u2500 SITE.md\n\u2502   \u2514\u2500\u2500 RULES.md\n\u251c\u2500\u2500 agents\/              # Per-agent identity\n\u2502   \u251c\u2500\u2500 agent-a\/\n\u2502   \u2502   \u251c\u2500\u2500 SOUL.md\n\u2502   \u2502   \u251c\u2500\u2500 MEMORY.md\n\u2502   \u2502   \u2514\u2500\u2500 daily\/\n\u2502   \u2502       \u2514\u2500\u2500 2026\/\n\u2502   \u2502           \u2514\u2500\u2500 03\/\n\u2502   \u2502               \u251c\u2500\u2500 15.md\n\u2502   \u2502               \u2514\u2500\u2500 16.md\n\u2502   \u2514\u2500\u2500 agent-b\/\n\u2502       \u251c\u2500\u2500 SOUL.md\n\u2502       \u2514\u2500\u2500 MEMORY.md\n\u251c\u2500\u2500 users\/               # Per-user preferences\n\u2502   \u251c\u2500\u2500 1\/\n\u2502   \u2502   \u251c\u2500\u2500 USER.md\n\u2502   \u2502   \u2514\u2500\u2500 MEMORY.md\n\u2502   \u2514\u2500\u2500 2\/\n\u2502       \u2514\u2500\u2500 USER.md\n\u2514\u2500\u2500 pipeline-{id}\/       # Pipeline-scoped files\n    \u251c\u2500\u2500 context\/\n    \u2514\u2500\u2500 flow-{id}\/<\/code><\/pre><\/div><p><strong>CoreMemoryFilesDirective<\/strong> (Priority 20) loads files from layers in order:<\/p><ol class=\"wp-block-list\"><li>SITE.md \u2192 RULES.md from the shared layer<\/li><li>SOUL.md \u2192 MEMORY.md from the agent layer<\/li><li>USER.md \u2192 MEMORY.md from the user layer<\/li><li>Custom files from <code>MemoryFileRegistry<\/code> (extensions)<\/li><\/ol><p>See <a href=\"https:\/\/chubes.net\/docs\/data-machine\/core-system\/wordpress-as-persistent-memory-for-ai-agents\/\">WordPress as Agent Memory<\/a> for full memory documentation.<\/p><h3 class=\"wp-block-heading\">Daily Memory System<\/h3><p>Temporal knowledge management via date-organized files:<\/p><ul class=\"wp-block-list\"><li><strong>DailyMemory<\/strong>: File operations at <code>agents\/{slug}\/daily\/YYYY\/MM\/DD.md<\/code><\/li><li><strong>DailyMemoryTask<\/strong>: System task with two phases:<ul class=\"wp-block-list\"><li>Phase 1: Synthesizes daily activity (jobs, chat) into daily file<\/li><li>Phase 2: Prunes MEMORY.md when &gt; 8KB, archiving session content to daily file<\/li><\/ul><\/li><li><strong>DailyMemorySelectorDirective<\/strong> (Priority 46): Injects daily memory into pipeline AI requests with configurable selection modes (recent days, specific dates, date range, months). Capped at 100KB total.<\/li><li><strong>DailyMemoryAbilities<\/strong>: CRUD + search via Abilities API with multi-agent scoping<\/li><\/ul><h3 class=\"wp-block-heading\">System Tasks Framework<\/h3><p>Background AI operations that run outside the normal pipeline model:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\"><\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code\"><code>\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502     SystemTask      \u2502  (abstract base)\n\u2502                     \u2502\n\u2502 execute()           \u2502  \u2190 Task-specific logic\n\u2502 completeJob()       \u2502  \u2190 Mark done + store engine_data\n\u2502 failJob()           \u2502  \u2190 Record failure\n\u2502 reschedule()        \u2502  \u2190 Retry with backoff (max 24)\n\u2502 supportsUndo()      \u2502  \u2190 Opt-in undo support\n\u2502 undo()              \u2502  \u2190 Reverse recorded effects\n\u2502 getPromptDefs()     \u2502  \u2190 Editable AI prompts\n\u2502 resolveSystemModel()\u2502  \u2190 Agent-aware model selection\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n         \u25b2\n         \u2502 extends\n    \u250c\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n    \u2502                                          \u2502\nImageGenerationTask  AltTextTask  DailyMemoryTask\nImageOptimizationTask  InternalLinkingTask\nGitHubIssueTask  MetaDescriptionTask<\/code><\/pre><\/div><p><strong>Undo System<\/strong>: Tasks that record effects in <code>engine_data<\/code> can be reversed:<\/p><ul class=\"wp-block-list\"><li>Phase 1: Synthesizes daily activity (jobs, chat) into daily file<\/li><li>Phase 2: Prunes MEMORY.md when &gt; 8KB, archiving session content to daily file<\/li><\/ul><h3 class=\"wp-block-heading\">Workspace System<\/h3><p>Secure file management outside the web root for agent operations. <strong>Moved to data-machine-code extension.<\/strong><\/p><ul class=\"wp-block-list\"><li>Phase 1: Synthesizes daily activity (jobs, chat) into daily file<\/li><li>Phase 2: Prunes MEMORY.md when &gt; 8KB, archiving session content to daily file<\/li><\/ul><h3 class=\"wp-block-heading\">Engine Data Architecture<\/h3><p><strong>Clean Data Separation<\/strong>: AI agents receive clean data packets without URLs while handlers access engine parameters via centralized filter pattern.<\/p><p><strong>Enhanced Database Storage + Filter Access<\/strong>: Fetch handlers store engine parameters (source_url, image_url) in database; steps retrieve via centralized <code>datamachine_engine_data<\/code> filter with storage\/retrieval mode detection for unified access.<\/p><p><strong>Core Pattern<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Fetch handlers store via centralized filter (array storage)\nif ($job_id) {\n    apply_filters(&#039;datamachine_engine_data&#039;, null, $job_id, [\n        &#039;source_url&#039; =&gt; $source_url,\n        &#039;image_url&#039; =&gt; $image_url\n    ]);\n}\n\n\/\/ Steps retrieve via centralized filter (EngineData.php)\n$engine_data = apply_filters(&#039;datamachine_engine_data&#039;, [], $job_id);\n$source_url = $engine_data[&#039;source_url&#039;] ?? null;\n$image_url = $engine_data[&#039;image_url&#039;] ?? null;<\/code><\/pre><\/div><p><strong>Benefits<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>post_content_modified<\/code> \u2192 restore WordPress revision<\/li><li><code>post_meta_set<\/code> \u2192 restore previous value<\/li><li><code>attachment_created<\/code> \u2192 delete attachment<\/li><li><code>featured_image_set<\/code> \u2192 restore\/remove thumbnail<\/li><\/ul><h3 class=\"wp-block-heading\">Abilities-First Architecture (@since v0.11.7)<\/h3><p><strong>Performance Revolution<\/strong>: Complete replacement of the older filter-based action and service-manager layers with direct ability classes. REST endpoints, WP-CLI commands, and chat tools all delegate to WordPress Abilities API registrations under <code>inc\/Abilities\/<\/code>.<\/p><p><strong>Ability domains<\/strong> (business logic):<\/p><ul class=\"wp-block-list\"><li><strong>Location<\/strong>: <code>\/var\/lib\/datamachine\/workspace\/<\/code> (or <code>DATAMACHINE_WORKSPACE_PATH<\/code>)<\/li><li><strong>Git-aware<\/strong>: Clone, status, pull, add, commit, push, log, diff<\/li><li><strong>File ops<\/strong>: Read (with pagination), write, edit (find-replace), list directory<\/li><li><strong>Security<\/strong>: Outside web root; mutating ops are CLI-only (not REST-exposed)<\/li><li><strong>CLI<\/strong>: <code>wp datamachine-code workspace {path,list,clone,remove,show,read,ls,write,edit,git}<\/code><\/li><\/ul><p><strong>Coding workspace note<\/strong>: Git-aware workspace and GitHub coding operations moved to the <code>data-machine-code<\/code> extension plugin. Data Machine core no longer registers <code>WorkspaceAbilities<\/code> or GitHub issue abilities.<\/p><p><strong>Benefits<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>Clean AI Data<\/strong>: AI processes content without URLs for better model performance<\/li><li><strong>Centralized Access<\/strong>: Single filter interface for all engine data retrieval<\/li><li><strong>Filter Consistency<\/strong>: Maintains architectural pattern of filter-based service discovery<\/li><li><strong>Flexible Storage<\/strong>: Steps access only what they need via filter call<\/li><\/ul><h3 class=\"wp-block-heading\">Step Types<\/h3><ul class=\"wp-block-list\"><li><strong>Flow abilities<\/strong> &#8211; Flow CRUD, duplication, pause\/resume, scheduling, webhooks, and queue management<\/li><li><strong>Pipeline abilities<\/strong> &#8211; Pipeline CRUD, import\/export, and pipeline-step template management<\/li><li><strong>Flow step abilities<\/strong> &#8211; Individual flow step configuration and handler management<\/li><li><strong>Job abilities<\/strong> &#8211; Workflow execution, retry\/fail\/delete\/recovery, flow health, and summaries<\/li><li><strong>Processed item abilities<\/strong> &#8211; Deduplication tracking across workflows<\/li><li><strong>Agent abilities<\/strong> &#8211; Agent CRUD, access grants, tokens, remote calls, memory, and daily memory<\/li><li><strong>File abilities<\/strong> &#8211; Agent files, flow uploads, cleanup, and memory scaffolding<\/li><\/ul><h3 class=\"wp-block-heading\">Directive System<\/h3><p>Priority-ordered context injection into every AI request:<\/p><figure class=\"wp-block-table\"><table><thead><tr><th>Priority<\/th><th>Directive<\/th><th>Context<\/th><th>Purpose<\/th><\/tr><\/thead><tbody><tr><td>10<\/td><td><code>PipelineCoreDirective<\/code><\/td><td>Pipeline<\/td><td>Base pipeline identity<\/td><\/tr><tr><td>15<\/td><td><code>ChatAgentDirective<\/code><\/td><td>Chat<\/td><td>Chat agent instructions<\/td><\/tr><tr><td>20<\/td><td><code>CoreMemoryFilesDirective<\/code><\/td><td>All<\/td><td>Layer files + custom registry<\/td><\/tr><tr><td>20<\/td><td><code>SystemAgentDirective<\/code><\/td><td>System<\/td><td>System task identity<\/td><\/tr><tr><td>40<\/td><td><code>PipelineMemoryFilesDirective<\/code><\/td><td>Pipeline<\/td><td>Per-pipeline memory files<\/td><\/tr><tr><td>45<\/td><td><code>ChatPipelinesDirective<\/code><\/td><td>Chat<\/td><td>Pipeline\/flow context<\/td><\/tr><tr><td>45<\/td><td><code>FlowMemoryFilesDirective<\/code><\/td><td>Pipeline<\/td><td>Per-flow memory files<\/td><\/tr><tr><td>46<\/td><td><code>DailyMemorySelectorDirective<\/code><\/td><td>Pipeline<\/td><td>Selected daily memory<\/td><\/tr><tr><td>50<\/td><td><code>PipelineSystemPromptDirective<\/code><\/td><td>Pipeline<\/td><td>Workflow instructions<\/td><\/tr><tr><td>80<\/td><td><code>SiteContextDirective<\/code><\/td><td>All<\/td><td>WordPress metadata (filterable)<\/td><\/tr><\/tbody><\/table><\/figure><p>Directives implement <code>DirectiveInterface<\/code> and return arrays of typed outputs:<\/p><ul class=\"wp-block-list\"><li><strong>3x Performance Improvement<\/strong>: Direct method calls eliminate filter indirection<\/li><li><strong>Centralized Business Logic<\/strong>: Consistent validation and error handling<\/li><li><strong>Reduced Database Queries<\/strong>: Optimized data access patterns<\/li><li><strong>Clean Architecture<\/strong>: Single responsibility per ability class<\/li><li><strong>Backward Compatibility<\/strong>: Maintains WordPress hook integration<\/li><\/ul><h3 class=\"wp-block-heading\">Authentication System<\/h3><p><strong>Base Authentication Provider Architecture<\/strong> (@since v0.2.6): Complete inheritance system with centralized option storage and validation across all authentication providers.<\/p><p><strong>Base Classes<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>Fetch<\/strong>: Data retrieval with clean content processing (core handlers include Files, RSS, Email, WordPress Local, WordPress Media, and WordPress API; extension plugins can register more)<\/li><li><strong>AI<\/strong>: Content processing with multi-provider support (OpenAI, Anthropic, Google, Grok)<\/li><li><strong>Publish<\/strong>: Content distribution with modular handler architecture (core handlers include WordPress and Email; extension plugins can register social destinations)<\/li><li><strong>Upsert<\/strong>: Content modification (WordPress posts\/pages)<\/li><li><strong>System Task<\/strong>: Execute system tasks within pipeline flows<\/li><li><strong>Agent Ping<\/strong>: Outbound webhook notifications to external agents<\/li><li><strong>Webhook Gate<\/strong>: Wait for inbound webhook before proceeding<\/li><\/ul><p><strong>OAuth Handlers<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>system_text<\/code> \u2014 plain text content<\/li><li><code>system_json<\/code> \u2014 labeled structured data<\/li><li><code>system_file<\/code> \u2014 file path with MIME type<\/li><\/ul><p><strong>Authentication Providers<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>BaseAuthProvider<\/strong> (<code>\/inc\/Core\/OAuth\/BaseAuthProvider.php<\/code>): Abstract base for all authentication providers with unified option storage, callback URL generation, and authentication state checking<\/li><li><strong>BaseOAuth1Provider<\/strong> (<code>\/inc\/Core\/OAuth\/BaseOAuth1Provider.php<\/code>): Base for extension-provided OAuth 1.0a providers<\/li><li><strong>BaseOAuth2Provider<\/strong> (<code>\/inc\/Core\/OAuth\/BaseOAuth2Provider.php<\/code>): Base for core and extension OAuth 2.0 providers<\/li><\/ul><p><strong>OAuth2 Flow<\/strong>:<\/p><ol class=\"wp-block-list\"><li>Create state nonce for CSRF protection<\/li><li>Build authorization URL with parameters<\/li><li>Handle callback: verify state, exchange code for token, retrieve account details, store credentials<\/li><\/ol><p><strong>OAuth1 Flow<\/strong>:<\/p><ol class=\"wp-block-list\"><li>Get request token<\/li><li>Build authorization URL<\/li><li>Handle callback: validate parameters, exchange for access token, store credentials<\/li><\/ol><p><strong>Benefits<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>OAuth1Handler<\/strong> (<code>\/inc\/Core\/OAuth\/OAuth1Handler.php<\/code>): Three-legged OAuth 1.0a flow implementation<\/li><li><strong>OAuth2Handler<\/strong> (<code>\/inc\/Core\/OAuth\/OAuth2Handler.php<\/code>): Authorization code flow implementation<\/li><\/ul><h3 class=\"wp-block-heading\">Universal Engine Architecture<\/h3><p>Data Machine v0.2.0 introduced a universal Engine layer (<code>\/inc\/Engine\/AI\/<\/code>) that serves both Pipeline and Chat agents with shared AI infrastructure:<\/p><p><strong>Core Engine Components<\/strong>:<\/p><ul class=\"wp-block-list\"><li>Core ships base classes plus concrete providers next to the handlers that need them, such as Email auth.<\/li><li>Extension plugins register their own providers through <code>datamachine_auth_providers<\/code>.<\/li><\/ul><p><strong>Tool Categories<\/strong>:<\/p><ul class=\"wp-block-list\"><li>Eliminates duplicated storage logic across all providers (~60% code reduction per provider)<\/li><li>Standardized error handling and logging<\/li><li>Unified security implementation<\/li><li>Easy integration of new providers via base class extension<\/li><\/ul><h3 class=\"wp-block-heading\">Filter-Based Discovery<\/h3><p>All components self-register via WordPress filters:<\/p><ul class=\"wp-block-list\"><li><strong>AIConversationLoop<\/strong>: Multi-turn conversation execution with tool calling, completion detection, and state management<\/li><li><strong>ToolExecutor<\/strong>: Universal tool discovery, enablement validation, and execution across agent types<\/li><li><strong>ToolParameters<\/strong>: Centralized parameter building for AI tools with data packet integration<\/li><li><strong>ConversationManager<\/strong>: Message formatting and conversation state management<\/li><li><strong>RequestBuilder<\/strong>: AI request construction with directive application and tool restructuring<\/li><li><strong>ToolResultFinder<\/strong>: Utility for finding tool execution results in data packets<\/li><\/ul><h3 class=\"wp-block-heading\">Modular Component Architecture (@since v0.2.1)<\/h3><p>Data Machine v0.2.1 introduced modular component systems for enhanced code organization and maintainability:<\/p><p><strong>FilesRepository Components<\/strong> (<code>\/inc\/Core\/FilesRepository\/<\/code>):<\/p><ul class=\"wp-block-list\"><li>Handler-specific tools for publish\/update operations<\/li><li>Global tools for search and analysis (GoogleSearch, LocalSearch, WebFetch, WordPressPostReader)<\/li><li>Coding workspace tools live in the <code>data-machine-code<\/code> extension plugin<\/li><li>Agent memory tools (AgentMemory, AgentDailyMemory) for runtime memory access<\/li><li>Chat-only tools for workflow building (@since v0.4.3):<ul class=\"wp-block-list\"><li>AddPipelineStep, ApiQuery, AuthenticateHandler, ConfigureFlowSteps, ConfigurePipelineStep, CopyFlow, CreateFlow, CreatePipeline, CreateTaxonomyTerm, ExecuteWorkflowTool, GetHandlerDefaults, ManageLogs, ReadLogs, RunFlow, SearchTaxonomyTerms, SetHandlerDefaults, UpdateFlow<\/li><\/ul><\/li><li>Automatic tool discovery and three-layer enablement system<\/li><\/ul><p><strong>WordPress Shared Components<\/strong> (<code>\/inc\/Core\/WordPress\/<\/code>):<\/p><ul class=\"wp-block-list\"><li>AddPipelineStep, ApiQuery, AuthenticateHandler, ConfigureFlowSteps, ConfigurePipelineStep, CopyFlow, CreateFlow, CreatePipeline, CreateTaxonomyTerm, ExecuteWorkflowTool, GetHandlerDefaults, ManageLogs, ReadLogs, RunFlow, SearchTaxonomyTerms, SetHandlerDefaults, UpdateFlow<\/li><\/ul><p><strong>EngineData<\/strong> (<code>\/inc\/Core\/EngineData.php<\/code>):<\/p><ul class=\"wp-block-list\"><li>AddPipelineStep, ApiQuery, AuthenticateHandler, ConfigureFlowSteps, ConfigurePipelineStep, CopyFlow, CreateFlow, CreatePipeline, CreateTaxonomyTerm, ExecuteWorkflowTool, GetHandlerDefaults, ManageLogs, ReadLogs, RunFlow, SearchTaxonomyTerms, SetHandlerDefaults, UpdateFlow<\/li><\/ul><p><strong>Engine Components<\/strong> (<code>\/inc\/Engine\/<\/code>):<\/p><ul class=\"wp-block-list\"><li><code>datamachine_handlers<\/code> &#8211; Register fetch\/publish\/upsert handlers<\/li><li><code>datamachine_tools<\/code> &#8211; Register AI tools and capabilities (unified static + runtime handler tool registry)<\/li><li><code>datamachine_auth_providers<\/code> &#8211; Register authentication providers<\/li><li><code>datamachine_step_types<\/code> &#8211; Register custom step types<\/li><li><code>datamachine_directives<\/code> &#8211; Register AI context directives<\/li><li><code>datamachine_get_oauth1_handler<\/code> &#8211; OAuth 1.0a handler service discovery<\/li><li><code>datamachine_get_oauth2_handler<\/code> &#8211; OAuth 2.0 handler service discovery<\/li><\/ul><p><strong>Benefits<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>DirectoryManager<\/strong> &#8211; Directory creation, path management, and three-layer resolution<\/li><li><strong>FileStorage<\/strong> &#8211; File operations and flow-isolated storage<\/li><li><strong>FileCleanup<\/strong> &#8211; Retention policy enforcement and cleanup<\/li><li><strong>ImageValidator<\/strong> &#8211; Image validation and metadata extraction<\/li><li><strong>VideoValidator<\/strong> &#8211; Video file validation<\/li><li><strong>RemoteFileDownloader<\/strong> &#8211; Remote file downloading with validation<\/li><li><strong>FileRetrieval<\/strong> &#8211; Data retrieval from file storage<\/li><li><strong>DailyMemory<\/strong> &#8211; Daily memory file operations (read, write, append, search, list)<\/li><\/ul><p>For detailed documentation:<\/p><ul class=\"wp-block-list\"><li><strong>TaxonomyHandler<\/strong> &#8211; Taxonomy selection and term creation (skip, AI-decided, pre-selected modes)<\/li><li><strong>WordPressSettingsHandler<\/strong> &#8211; Shared WordPress settings fields<\/li><li><strong>WordPressFilters<\/strong> &#8211; Service discovery registration<\/li><\/ul><h3 class=\"wp-block-heading\">Centralized Handler Filter System<\/h3><p><strong>Unified Cross-Cutting Functionality<\/strong>: The engine provides centralized filters for shared functionality across multiple handlers, eliminating code duplication and ensuring consistency.<\/p><p><strong>Core Centralized Filters<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>Consolidated Operations<\/strong> &#8211; Featured image attachment, source URL attribution, and engine data access (@since v0.2.1, enhanced v0.2.6)<\/li><li><strong>Unified Interface<\/strong> &#8211; Single class for all engine data operations (replaces FeaturedImageHandler and SourceUrlHandler in v0.2.6)<\/li><\/ul><p><strong>Implementation<\/strong>:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Timeframe parsing example\n$cutoff_timestamp = apply_filters(&#039;datamachine_timeframe_limit&#039;, null, &#039;24_hours&#039;);\n$date_query = $cutoff_timestamp ? [&#039;after&#039; =&gt; gmdate(&#039;Y-m-d H:i:s&#039;, $cutoff_timestamp)] : [];\n\n\/\/ Keyword matching example\n$matches = apply_filters(&#039;datamachine_keyword_search_match&#039;, true, $content, $search_keywords);\nif (!$matches) continue; \/\/ Skip non-matching items\n\n\/\/ Data packet creation example\n$data = apply_filters(&#039;datamachine_data_packet&#039;, $data, $packet_data, $flow_step_id, $step_type);<\/code><\/pre><\/div><p><strong>Benefits<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>StepNavigator<\/strong> &#8211; Centralized step navigation logic for execution flow<\/li><\/ul><h3 class=\"wp-block-heading\">WordPress Publish Handler Architecture<\/h3><p><strong>Modular Component System<\/strong>: The WordPress publish handler uses specialized processing modules for enhanced maintainability and extensibility.<\/p><p><strong>Core Components<\/strong>:<\/p><ul class=\"wp-block-list\"><li><strong>Code Deduplication<\/strong>: Eliminates repetitive functionality across handlers<\/li><li><strong>Single Responsibility<\/strong>: Each component has focused purpose<\/li><li><strong>Maintainability<\/strong>: Centralized logic simplifies updates<\/li><li><strong>Extensibility<\/strong>: Easy to add new functionality via composition<\/li><\/ul><p><strong>Configuration Hierarchy<\/strong>: System-wide defaults ALWAYS override handler-specific configuration when set, providing consistent behavior across all WordPress publish operations.<\/p><p><strong>Features<\/strong>:<\/p><ul class=\"wp-block-list\"><li>FilesRepository Components<\/li><li>WordPress Shared Components<\/li><li>EngineData<\/li><li>StepNavigator<\/li><\/ul><h3 class=\"wp-block-heading\">File Management<\/h3><p>Flow-isolated UUID storage with automatic cleanup:<\/p><ul class=\"wp-block-list\"><li><strong><code>datamachine_timeframe_limit<\/code><\/strong>: Shared timeframe parsing with discovery\/conversion modes<ul class=\"wp-block-list\"><li>Discovery mode: Returns available timeframe options for UI dropdowns<\/li><li>Conversion mode: Returns Unix timestamp for specified timeframe<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><\/li><li><strong><code>datamachine_keyword_search_match<\/code><\/strong>: Universal keyword matching with OR logic<ul class=\"wp-block-list\"><li>Case-insensitive Unicode-safe matching<\/li><li>Comma-separated keyword support<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><\/li><li><strong><code>datamachine_data_packet<\/code><\/strong>: Standardized data packet creation and structure<ul class=\"wp-block-list\"><li>Ensures type and timestamp fields are present<\/li><li>Maintains chronological ordering via array_unshift()<\/li><li>Used by: All step types for consistent data flow<\/li><\/ul><\/li><\/ul><h3 class=\"wp-block-heading\">HTTP Client<\/h3><p>The centralized <code>HttpClient<\/code> class (<code>\/inc\/Core\/HttpClient.php<\/code>) standardizes all outbound requests for fetch and publish handlers. It wraps the native WordPress HTTP helpers while:<\/p><ul class=\"wp-block-list\"><li>Discovery mode: Returns available timeframe options for UI dropdowns<\/li><li>Conversion mode: Returns Unix timestamp for specified timeframe<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><p>See <a href=\"https:\/\/chubes.net\/docs\/data-machine\/core-system\/http-client\/\">HTTP Client<\/a> for implementation details and usage guidance.<\/p><h3 class=\"wp-block-heading\">Admin Interface<\/h3><p><strong>Modern React Architecture<\/strong>: The entire Data Machine admin interface (Pipelines, Logs, Settings, Jobs, and Agents) uses a complete React implementation with zero jQuery or AJAX dependencies.<\/p><p><strong>React Implementation<\/strong>:<\/p><ul class=\"wp-block-list\"><li>Case-insensitive Unicode-safe matching<\/li><li>Comma-separated keyword support<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><p><strong>Component Architecture<\/strong>:<\/p><ul class=\"wp-block-list\"><li>Ensures type and timestamp fields are present<\/li><li>Maintains chronological ordering via array_unshift()<\/li><li>Used by: All step types for consistent data flow<\/li><\/ul><p><strong>Complete REST API Integration<\/strong>:\nAll admin pages now use REST API architecture with zero jQuery\/AJAX dependencies.<\/p><p><strong>Security Model<\/strong>: All admin operations require <code>manage_options<\/code> capability with WordPress nonce validation.<\/p><h3 class=\"wp-block-heading\">Extension Framework<\/h3><p>Complete extension system for custom handlers and tools:<\/p><ul class=\"wp-block-list\"><li>Discovery mode: Returns available timeframe options for UI dropdowns<\/li><li>Conversion mode: Returns Unix timestamp for specified timeframe<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><h2 class=\"wp-block-heading\">Key Features<\/h2><h3 class=\"wp-block-heading\">AI Integration<\/h3><ul class=\"wp-block-list\"><li>Case-insensitive Unicode-safe matching<\/li><li>Comma-separated keyword support<\/li><li>Used by: RSS, WordPress Local, WordPress Media, WordPress API, and extension fetch handlers<\/li><\/ul><h3 class=\"wp-block-heading\">Data Processing<\/h3><ul class=\"wp-block-list\"><li>Ensures type and timestamp fields are present<\/li><li>Maintains chronological ordering via array_unshift()<\/li><li>Used by: All step types for consistent data flow<\/li><\/ul><h3 class=\"wp-block-heading\">Scheduling<\/h3><ul class=\"wp-block-list\"><li><strong>Code Consistency<\/strong>: Identical behavior across all handlers using shared filters<\/li><li><strong>Maintainability<\/strong>: Single implementation location for shared functionality<\/li><li><strong>Extensibility<\/strong>: New handlers automatically inherit shared capabilities<\/li><li><strong>Performance<\/strong>: Optimized implementations used across all handlers<\/li><\/ul><h3 class=\"wp-block-heading\">Security<\/h3><ul class=\"wp-block-list\"><li><strong>EngineData<\/strong>: Consolidated featured image attachment and source URL attribution with configuration hierarchy (system defaults override handler config) (@since v0.2.1, enhanced v0.2.6)<\/li><li><strong>TaxonomyHandler<\/strong>: Configuration-based taxonomy processing with three selection modes (skip, AI-decided, pre-selected)<\/li><li><strong>Direct Integration<\/strong>: WordPress handlers use EngineData and TaxonomyHandler directly for single source of truth data access<\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p>Data Machine is an AI-first WordPress plugin that uses a Pipeline+Flow architecture for automated content processing and publication. It provides multi-provider AI integration with tool-first design patterns, centered around a&#8230;<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"footnotes":""},"tags":[],"project":[487],"project_type":[484],"class_list":["post-4955","documentation","type-documentation","status-publish","hentry","project-data-machine","project_type-wordpress-plugins"],"project_info":{"id":487,"name":"Data Machine","slug":"data-machine"},"project_type_info":{"id":484,"name":"WordPress Plugins","slug":"wordpress-plugins"},"_links":{"self":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4955","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation"}],"about":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/types\/documentation"}],"version-history":[{"count":10,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4955\/revisions"}],"predecessor-version":[{"id":11418,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4955\/revisions\/11418"}],"wp:attachment":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/media?parent=4955"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/tags?post=4955"},{"taxonomy":"project","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project?post=4955"},{"taxonomy":"project_type","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project_type?post=4955"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}