{"id":4968,"date":"2026-01-26T01:51:09","date_gmt":"2026-01-26T06:51:09","guid":{"rendered":"https:\/\/chubes.net\/?documentation=fetchhandler-base-class"},"modified":"2026-04-20T00:09:36","modified_gmt":"2026-04-20T04:09:36","slug":"fetchhandler-base-class","status":"publish","type":"documentation","link":"https:\/\/chubes.net\/docs\/data-machine\/core-system\/fetchhandler-base-class\/","title":{"rendered":"FetchHandler Base Class"},"content":{"rendered":"<h2 class=\"wp-block-heading\">Overview<\/h2><p>The <code>FetchHandler<\/code> class (<code>\/inc\/Core\/Steps\/Fetch\/Handlers\/FetchHandler.php<\/code>) is the abstract base class for all fetch handlers in the Data Machine system. Introduced in version 0.2.1, it provides standardized functionality for data fetching operations including deduplication, engine data storage, filtering, and logging.<\/p><h2 class=\"wp-block-heading\">Architecture<\/h2><p><strong>Location<\/strong>: <code>\/inc\/Core\/Steps\/Fetch\/Handlers\/FetchHandler.php<\/code>\n<strong>Inheritance<\/strong>: Abstract base class extending <code>Step<\/code>\n<strong>Since<\/strong>: 0.2.1<\/p><h2 class=\"wp-block-heading\">Core Functionality<\/h2><h3 class=\"wp-block-heading\">Single Item Execution Model<\/h3><p>All fetch handlers implement the <strong>Single Item Execution Model<\/strong>, processing exactly one item per job execution. This ensures that failures are isolated to individual items and prevents batch processing timeouts.<\/p><h3 class=\"wp-block-heading\">Deduplication Management<\/h3><p>Automatic deduplication tracking to prevent processing the same items multiple times:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Check if item was already processed\nif ($this-&gt;isItemProcessed($item_id, $flow_step_id)) {\n    return $this-&gt;emptyResponse();\n}\n\n\/\/ Mark item as processed\n$this-&gt;markItemProcessed($item_id, $flow_step_id, $job_id);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Engine Data Storage<\/h3><p>Store handler-specific parameters for downstream handlers:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">$this-&gt;storeEngineData($job_id, [\n    &#039;source_url&#039; =&gt; $source_url,\n    &#039;image_url&#039; =&gt; $image_url\n]);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Standardized Responses<\/h3><p>Consistent response methods for success and error cases:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Success response with data packets\nreturn $this-&gt;successResponse([$dataPacket]);\n\n\/\/ Empty response (no new items)\nreturn $this-&gt;emptyResponse();\n\n\/\/ Error response\nreturn $this-&gt;errorResponse(&#039;Error message&#039;, [&#039;details&#039; =&gt; $details]);<\/code><\/pre><\/div><h3 class=\"wp-block-heading\">Exclude Keywords Filtering (@since v0.3.1)<\/h3><p>Filter content based on negative keywords to exclude unwanted items:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">\/\/ Check if content contains any exclude keywords\n$exclude_keywords = $config[&#039;exclude_keywords&#039;] ?? &#039;&#039;;\nif (!empty($exclude_keywords) &amp;&amp; $this-&gt;applyExcludeKeywords($content, $exclude_keywords)) {\n    \/\/ Content contains excluded keywords, skip this item\n    continue;\n}<\/code><\/pre><\/div><p>The <code>applyExcludeKeywords()<\/code> method returns <code>true<\/code> if any exclude keyword is found in the text (case-insensitive), indicating the item should be filtered out.<\/p><h2 class=\"wp-block-heading\">Required Implementation<\/h2><p>All fetch handlers must implement the <code>executeFetch()<\/code> method:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">abstract protected function executeFetch(\n    int $pipeline_id,\n    array $config,\n    ?string $flow_step_id,\n    int $flow_id,\n    ?string $job_id\n): array;<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Standard Implementation Pattern<\/h2><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">use DataMachineCoreStepsFetchHandlersFetchHandler;\n\nclass MyFetchHandler extends FetchHandler {\n    public function __construct() {\n        parent::__construct(&#039;my_handler&#039;);\n    }\n\n    protected function executeFetch(\n        int $pipeline_id,\n        array $config,\n        ?string $flow_step_id,\n        int $flow_id,\n        ?string $job_id\n    ): array {\n        \/\/ Check deduplication\n        if ($this-&gt;isItemProcessed($item_id, $flow_step_id)) {\n            return $this-&gt;emptyResponse();\n        }\n\n        \/\/ Fetch data from source\n        $fetched_data = $this-&gt;fetch_from_source($config);\n\n        \/\/ Mark as processed\n        $this-&gt;markItemProcessed($item_id, $flow_step_id, $job_id);\n\n        \/\/ Store engine data for downstream handlers\n        $this-&gt;storeEngineData($job_id, [\n            &#039;source_url&#039; =&gt; $source_url,\n            &#039;image_url&#039; =&gt; $image_url\n        ]);\n\n        \/\/ Create standardized data packet\n        $dataPacket = new DataMachineCoreDataPacket(\n            [&#039;content_string&#039; =&gt; $content_string, &#039;file_info&#039; =&gt; null],\n            [&#039;source_type&#039; =&gt; &#039;my_handler&#039;, &#039;item_identifier_to_log&#039; =&gt; $item_id],\n            &#039;fetch&#039;\n        );\n\n        return $this-&gt;successResponse([$dataPacket-&gt;addTo([])]);\n    }\n}<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Engine Data Parameters<\/h2><p>Fetch handlers should store relevant parameters for publish\/upsert handlers:<\/p><figure class=\"wp-block-table\"><table><thead><tr><th>Parameter<\/th><th>Description<\/th><th>Used By<\/th><\/tr><\/thead><tbody><tr><td><code>source_url<\/code><\/td><td>Source URL of the content<\/td><td>Update handlers, logging<\/td><\/tr><tr><td><code>image_url<\/code><\/td><td>URL of associated image<\/td><td>Publish handlers with image support<\/td><\/tr><\/tbody><\/table><\/figure><h2 class=\"wp-block-heading\">Handler-Specific Engine Parameters<\/h2><p>Different fetch handlers store different engine parameters:<\/p><ul class=\"wp-block-list\"><li><strong>Reddit<\/strong>: <code>source_url<\/code> (post URL), <code>image_url<\/code> (stored image URL)<\/li><li><strong>WordPress Local<\/strong>: <code>source_url<\/code> (permalink), <code>image_url<\/code> (featured image URL)<\/li><li><strong>WordPress API<\/strong>: <code>source_url<\/code> (post link), <code>image_url<\/code> (featured image URL)<\/li><li><strong>WordPress Media<\/strong>: <code>source_url<\/code> (parent post permalink), <code>image_url<\/code> (media URL)<\/li><li><strong>RSS<\/strong>: <code>source_url<\/code> (item link), <code>image_url<\/code> (enclosure URL)<\/li><li><strong>Universal Web Scraper<\/strong>: <code>source_url<\/code> (page URL), <code>image_url<\/code> (detected image)<\/li><li><strong>Google Sheets<\/strong>: <code>source_url<\/code> (empty), <code>image_url<\/code> (empty)<\/li><li><strong>Files<\/strong>: <code>image_url<\/code> (public URL for images only)<\/li><\/ul><h2 class=\"wp-block-heading\">File Handling<\/h2><p>For file-based fetch handlers, use the FilesRepository components:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">use DataMachineCoreFilesRepositoryFileStorage;\n\n$file_storage = new FileStorage();\n$stored_path = $file_storage-&gt;store_file($file_content, $filename, $job_id);<\/code><\/pre><\/div><h2 class=\"wp-block-heading\">Key Methods (@since v0.3.1)<\/h2><h3 class=\"wp-block-heading\">applyExcludeKeywords()<\/h3><p>Filter content based on negative keywords:<\/p><div class=\"code-block-wrapper\"><div class=\"code-block-header\"><span class=\"code-block-language\">php<\/span><button class=\"code-copy-btn\" aria-label=\"Copy code\"><svg><use href=\"https:\/\/chubes.net\/wp-content\/themes\/chubes\/assets\/icons\/chubes.svg#icon-copy\"><\/use><\/svg><\/button><\/div><pre data-chubes-enhanced class=\"wp-block-code language-php\"><code class=\"language-php\">protected function applyExcludeKeywords(string $text, string $exclude_keywords): bool<\/code><\/pre><\/div><p><strong>Parameters<\/strong>:<\/p><ul class=\"wp-block-list\"><li><code>$text<\/code>: Text content to search<\/li><li><code>$exclude_keywords<\/code>: Comma-separated list of keywords to exclude<\/li><\/ul><p><strong>Returns<\/strong>: <code>true<\/code> if any exclude keyword is found (item should be filtered out), <code>false<\/code> otherwise<\/p><p><strong>Features<\/strong>:<\/p><ul class=\"wp-block-list\"><li>Case-insensitive matching<\/li><li>Unicode-safe via <code>mb_stripos()<\/code><\/li><li>Handles comma-separated keyword lists<\/li><li>Returns false for empty keyword lists<\/li><\/ul><h2 class=\"wp-block-heading\">Benefits<\/h2><ul class=\"wp-block-list\"><li><strong>Deduplication<\/strong>: Automatic prevention of duplicate processing<\/li><li><strong>Consistency<\/strong>: Standardized response patterns across all fetch handlers<\/li><li><strong>Engine Integration<\/strong>: Seamless data flow to downstream handlers<\/li><li><strong>Error Handling<\/strong>: Centralized error response formatting<\/li><li><strong>Maintainability<\/strong>: Reduced code duplication and consistent patterns<\/li><li><strong>Negative Filtering<\/strong> (@since v0.3.1): Built-in exclude keyword filtering<\/li><\/ul><h2 class=\"wp-block-heading\">Implementations<\/h2><p>All fetch handlers extend this base class:<\/p><ul class=\"wp-block-list\"><li>RSS Handler<\/li><li>Reddit Handler<\/li><li>Universal Web Scraper Handler<\/li><li>WordPress Local Handler<\/li><li>WordPress Media Handler<\/li><li>WordPress API Handler<\/li><li>Google Sheets Handler<\/li><li>Files Handler<\/li><\/ul><p>See Fetch Handlers Overview for comparison.<\/p>","protected":false},"excerpt":{"rendered":"<p>Overview The FetchHandler class (\/inc\/Core\/Steps\/Fetch\/Handlers\/FetchHandler.php) is the abstract base class for all fetch handlers in the Data Machine system. Introduced in version 0.2.1, it provides standardized functionality for data fetching&#8230;<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"footnotes":""},"tags":[],"project":[506],"project_type":[484],"class_list":["post-4968","documentation","type-documentation","status-publish","hentry","project-core-system","project_type-wordpress-plugins"],"project_info":{"id":487,"name":"Data Machine","slug":"data-machine"},"project_type_info":{"id":484,"name":"WordPress Plugins","slug":"wordpress-plugins"},"_links":{"self":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4968","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation"}],"about":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/types\/documentation"}],"version-history":[{"count":4,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4968\/revisions"}],"predecessor-version":[{"id":11288,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/documentation\/4968\/revisions\/11288"}],"wp:attachment":[{"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/media?parent=4968"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/tags?post=4968"},{"taxonomy":"project","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project?post=4968"},{"taxonomy":"project_type","embeddable":true,"href":"https:\/\/chubes.net\/wp-json\/wp\/v2\/project_type?post=4968"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}