rocAL Serialize PR 3 : Introduce PipelineOperator class by fiona-gladwin · Pull Request #401 · ROCm/rocAL

fiona-gladwin · 2025-10-22T12:13:42Z

Motivation

Introduces PipelineOperator class to represent pipeline operators with required data
Tracks the operators in MasterGraph with unique naming
Implements argument collection for nodes through template-based mechanisms

Technical Details

The PipelineOperator class is introduced to track the operators in MasterGraph with unique naming.
Adds support to the BrightnessNode and ImageLoaderNode to store the args and operator details in the pipeline.
Introduce variadic templated function in the Node class to set all the arguments for the node

Test Plan

No new tests added - Must pass existing tests

NOTE:
Contains file changes from previous PR's. To be merged after PR #400

Remove the usage of API enums in internal node files

Add support to register all internal enums

Creates an argument instance for each arg passed in the Node Handles different argument types i.e parameters, vectors, enums and predefined types

…and ImageLoaderNode

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Create single constructor with check for different data types in Argument class

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

into fg/ser_argument

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

LakshmiKumar23 · 2025-11-10T23:35:32Z

@fiona-gladwin please resolve merge conflicts

rrawther · 2025-11-11T01:21:58Z

@fiona-gladwin : please address the comments

rocAL/source/loaders/image/node_image_loader.cpp

into fg/ser1_pipe_op

Copilot

Pull Request Overview

This pull request introduces the PipelineOperator class to track pipeline operators with unique naming and metadata, along with supporting infrastructure to collect and store operator arguments. The changes extend serialization capabilities for the rocAL pipeline.

Key Changes

Introduces the PipelineOperator class to represent operators in the pipeline with operator name, module category, arguments, and associated node reference
Adds a new Tensor constructor that accepts a name parameter for unique identification
Implements set_node_arguments template method in the Node class using variadic templates and fold expressions to collect operator arguments
Updates BrightnessNode and ImageLoaderNode to store arguments using the new infrastructure
Tracks operators in MasterGraph with unique naming using _op_idx and _tensor_idx counters

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
rocAL/include/pipeline/pipeline_operator.h	New class representing pipeline operators with name, module type, arguments, and node reference
rocAL/include/pipeline/node.h	Adds `node_name()` virtual method and `set_node_arguments` variadic template for argument collection
rocAL/include/pipeline/master_graph.h	Adds operator and tensor tracking with unique IDs; updates `add_node` templates to register operators
rocAL/include/pipeline/tensor.h	Adds new constructor accepting name parameter and `tensor_name()` getter method
rocAL/include/loaders/image/node_image_loader.h	Implements `node_name()` override returning "ImageLoaderNode"
rocAL/include/augmentations/color_augmentations/node_brightness.h	Implements `node_name()` override returning "BrightnessNode"
rocAL/source/pipeline/tensor.cpp	Implements new named constructor that initializes `_tensor_name`
rocAL/source/pipeline/master_graph.cpp	Updates all `Tensor` constructor calls to use named version; adds LabelReader operator tracking
rocAL/source/loaders/image/node_image_loader.cpp	Collects and stores 23 loader arguments using `set_node_arguments`
rocAL/source/augmentations/color_augmentations/node_brightness.cpp	Stores alpha and beta arguments for both float and FloatParam versions
CHANGELOG.md	Documents new `PipelineOperator` class and operator tracking support

Comments suppressed due to low confidence (1)

rocAL/include/pipeline/tensor.h:415

The assignment operator does not clear existing contents before copying, which could lead to memory leaks if the TensorList already contains tensors. Additionally, it should return *this to follow standard assignment operator conventions:

void operator=(TensorList& other) {
    if (this != &other) {
        release();  // Free existing tensors
        _tensor_list.clear();
        _tensor_data_size.clear();
        _tensor_roi_size.clear();
        for (unsigned idx = 0; idx < other.size(); idx++) {
            auto* new_tensor = new Tensor(other[idx]->info(), other[idx]->tensor_name() + "_copy");
            if (new_tensor->create_from_handle(other[idx]->context()) != 0)
                THROW("Cannot create the tensor from handle")
            this->push_back(new_tensor);
        }
    }
}

Or better yet, return a reference:

TensorList& operator=(TensorList& other) { ... return *this; }

    void operator=(TensorList& other) {
        for (unsigned idx = 0; idx < other.size(); idx++) {
            auto* new_tensor = new Tensor(other[idx]->info(), other[idx]->tensor_name() + "_copy");
            if (new_tensor->create_from_handle(other[idx]->context()) != 0)
                THROW("Cannot create the tensor from handle")
            this->push_back(new_tensor);
        }
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rocAL/include/pipeline/node.h

rocAL/include/pipeline/pipeline_operator.h

Copilot

Pull Request Overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rocAL/include/pipeline/node.h

rocAL/include/pipeline/pipeline_operator.h

rocAL/include/pipeline/master_graph.h

rocAL/source/loaders/image/node_image_loader.cpp

fiona-gladwin · 2025-11-18T18:21:02Z

@rrawther @LakshmiKumar23 addressed all review comments.

* Docs - updated the toc (#406) * rocAL Serialize PR 1- Introduce EnumRegistry (#398) * Use enums present in commons.h Remove the usage of API enums in internal node files * Remove unused include rocal_api_types.h * Introduce EnumRegistry Add support to register all internal enums * Remove redundant static cast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove semicolon for enum macro * Modify name of macro * Change enums to scoped enums in commons.h * Minor fix * Remove unused includes * Add compiler keywords * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CMakeLists version for rocAL --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Fiona Gladwin <121928245+fgladwin@users.noreply.github.com> Co-authored-by: Sundar Rajan Vaithiyanathan <99159823+SundarRajan28@users.noreply.github.com> Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> * HIP - Set HIP device in output routine thread (#407) * Adding hipSetDevice in output routine thread * Remove whitespace Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Sundar Rajan Vaithiyanathan <svaithiy@amd.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> * rocAL Serialize PR 2 : Introduce Argument class (#400) * Use enums present in commons.h Remove the usage of API enums in internal node files * Remove unused include rocal_api_types.h * Introduce EnumRegistry Add support to register all internal enums * Introduce Argument class Creates an argument instance for each arg passed in the Node Handles different argument types i.e parameters, vectors, enums and predefined types * Remove redundant static cast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove semicolon for enum macro * Modify name of macro * Change enums to scoped enums in commons.h * Code reorganization Create single constructor with check for different data types in Argument class * Minor fix * Remove unused includes * Add compiler keywords * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Minor change * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Minor change * Fix shared pointer fetching * Minor fix * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CMakeLists version for rocAL * Update CHANGELOG * Minor change * Update CHANGELOG.md Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com> * Add const correctness to get args list --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Fiona Gladwin <121928245+fgladwin@users.noreply.github.com> Co-authored-by: Sundar Rajan Vaithiyanathan <99159823+SundarRajan28@users.noreply.github.com> Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com> Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> * Docs - removed opencl and added hip (#408) Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> * Enable Lintian Support rocAL (#403) * fix lintian errors * remove overrides install, fix formatting * fix readme error * removing docs/README.md --------- Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> * Docs - Bump rocm-docs-core[api_reference] from 1.27.0 to 1.29.0 in /docs/sphinx (#416) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.27.0 to 1.29.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.27.0...v1.29.0) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.29.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Docs - readding the readme (#417) Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> * Bug fixes - HIP (#409) * Fix bug in CropResize node * Use HIP mem for fused crop rocjpeg decoder * Fix bug with ROI updation in numpy loader * Resolve review comments --------- Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> Co-authored-by: Fiona-MCW <70996026+fiona-gladwin@users.noreply.github.com> * Docs - Bump rocm-docs-core[api_reference] from 1.29.0 to 1.30.0 in /docs/sphinx (#418) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.29.0 to 1.30.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.29.0...v1.30.0) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.30.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Docs - Bump rocm-docs-core[api_reference] from 1.30.0 to 1.30.1 in /docs/sphinx (#423) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.30.0 to 1.30.1. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.30.0...v1.30.1) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.30.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * rocAL Serialize PR 3 : Introduce PipelineOperator class (#401) * Use enums present in commons.h Remove the usage of API enums in internal node files * Remove unused include rocal_api_types.h * Introduce EnumRegistry Add support to register all internal enums * Introduce Argument class Creates an argument instance for each arg passed in the Node Handles different argument types i.e parameters, vectors, enums and predefined types * Introduce PipelineOperator class * Add support to create and return node name and tensor name * Add support in MasterGraph to store details of PipelineOperators * Add support to store the argument details in the node for brightness and ImageLoaderNode * Remove redundant static cast Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove semicolon for enum macro * Modify name of macro * Change enums to scoped enums in commons.h * Code reorganization Create single constructor with check for different data types in Argument class * Minor fix * Remove unused includes * Add compiler keywords * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Minor change * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update rocAL/include/pipeline/argument.h Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Minor change * Fix shared pointer fetching * Minor fix * Minor fix * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Update CMakeLists version for rocAL * Update CHANGELOG * Update CHANGELOG.md * Minor change * Minor changes * Update CHANGELOG.md Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com> * Add const qualifier * Add const correctness to get args list * Minor change * Add const correctness to get args list * Update CHANGELOG * Generate UID for tensors from MasterGraph * Minor change * Resolve copilot review comments * Minor change * Minor changes --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Fiona Gladwin <121928245+fgladwin@users.noreply.github.com> Co-authored-by: Sundar Rajan Vaithiyanathan <99159823+SundarRajan28@users.noreply.github.com> Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com> Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> * CMakeLists - Update GPU targets (#421) Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> * multi channel dataloader - example/sample (#415) * Update numpy reader example * Add new example and remove changes from other example * Update comment * Remove unused import Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Resolve review comments --------- Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> Co-authored-by: Fiona-MCW <70996026+fiona-gladwin@users.noreply.github.com> Co-authored-by: Kiriti Gowda <kiritigowda@gmail.com> Co-authored-by: Sundar Rajan Vaithiyanathan <svaithiy@amd.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Docs - Update CHANGELOG.md (#424) * Update CHANGELOG.md Updates for 7.2 * Changelog - bugfixes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com> Co-authored-by: Fiona-MCW <70996026+fiona-gladwin@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Fiona Gladwin <121928245+fgladwin@users.noreply.github.com> Co-authored-by: Sundar Rajan Vaithiyanathan <99159823+SundarRajan28@users.noreply.github.com> Co-authored-by: Sundar Rajan Vaithiyanathan <svaithiy@amd.com> Co-authored-by: Lakshmi Kumar <lakshmi.kumar@amd.com> Co-authored-by: jonatluu <jonatluu@amd.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fiona-gladwin and others added 30 commits September 22, 2025 02:30

Use enums present in commons.h

c4e55ca

Remove the usage of API enums in internal node files

Remove unused include rocal_api_types.h

22af55b

Introduce EnumRegistry

752a734

Add support to register all internal enums

Introduce Argument class

6d45638

Creates an argument instance for each arg passed in the Node Handles different argument types i.e parameters, vectors, enums and predefined types

Introduce PipelineOperator class

90736c8

Add support to create and return node name and tensor name

fc55deb

Add support in MasterGraph to store details of PipelineOperators

49e7025

Add support to store the argument details in the node for brightness …

25c9d91

…and ImageLoaderNode

Remove redundant static cast

cba503f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Remove semicolon for enum macro

604ce6a

Modify name of macro

ef53616

Change enums to scoped enums in commons.h

bb665e5

Merge branch 'fg/enum_register' into fg/ser_argument

1f11c53

Code reorganization

d9d527e

Create single constructor with check for different data types in Argument class

Minor fix

2651138

Remove unused includes

68dc5d3

Add compiler keywords

da2ae16

Merge branch 'fg/enum_change' into fg/enum_register

db854ce

Merge branch 'fg/enum_register' into fg/ser_argument

ca0a736

Update rocAL/include/pipeline/argument.h

a1979c2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update rocAL/include/pipeline/argument.h

48730ba

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'develop' into fg/enum_change

d05dd31

Merge branch 'develop' into fg/enum_register

478e019

Merge branch 'fg/ser_argument' of https://github.com/fiona-gladwin/rocAL

df36a72

into fg/ser_argument

Minor change

6b97bcb

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

a8a5955

Update rocAL/include/pipeline/argument.h

8f361bb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update rocAL/include/pipeline/argument.h

c6212ca

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Minor change

57dfa73

Fix shared pointer fetching

55640dd

fiona-gladwin and others added 3 commits November 11, 2025 10:28

Merge branch 'develop' into fg/ser1_pipe_op

dda7fd8

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

d0fd61f

Merge branch 'develop' into fg/ser1_pipe_op

90f7226

LakshmiKumar23 reviewed Nov 11, 2025

View reviewed changes

rocAL/source/loaders/image/node_image_loader.cpp Show resolved Hide resolved

fiona-gladwin and others added 5 commits November 12, 2025 00:19

Generate UID for tensors from MasterGraph

c1ab697

Merge branch 'fg/ser1_pipe_op' of https://github.com/fiona-gladwin/rocAL

ac72c30

into fg/ser1_pipe_op

Minor change

4109ed4

Merge branch 'develop' into fg/ser1_pipe_op

c16bd3e

Merge branch 'develop' into fg/ser1_pipe_op

c60e86b

kiritigowda requested a review from Copilot November 17, 2025 18:13

Copilot started reviewing on behalf of kiritigowda November 17, 2025 18:14 View session

Copilot finished reviewing on behalf of kiritigowda November 17, 2025 18:19

Copilot AI reviewed Nov 17, 2025

View reviewed changes

rocAL/include/pipeline/node.h Outdated Show resolved Hide resolved

rocAL/include/pipeline/pipeline_operator.h Outdated Show resolved Hide resolved

rocAL/include/pipeline/pipeline_operator.h Outdated Show resolved Hide resolved

rocAL/include/pipeline/pipeline_operator.h Show resolved Hide resolved

Resolve copilot review comments

798e1a1

fgladwin requested a review from Copilot November 18, 2025 06:42

Copilot started reviewing on behalf of fgladwin November 18, 2025 06:44 View session

Minor change

d2aff1b

Copilot finished reviewing on behalf of fgladwin November 18, 2025 06:46

Copilot AI reviewed Nov 18, 2025

View reviewed changes

Minor changes

5100f7c

Merge branch 'develop' into fg/ser1_pipe_op

77fa35e

LakshmiKumar23 approved these changes Nov 26, 2025

View reviewed changes

Merge branch 'develop' into fg/ser1_pipe_op

6ef53ca

rrawther approved these changes Dec 2, 2025

View reviewed changes

kiritigowda approved these changes Dec 4, 2025

View reviewed changes

kiritigowda merged commit 753c9f0 into ROCm:develop Dec 4, 2025
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rocAL Serialize PR 3 : Introduce PipelineOperator class #401

rocAL Serialize PR 3 : Introduce PipelineOperator class #401
kiritigowda merged 88 commits intoROCm:developfrom
fiona-gladwin:fg/ser1_pipe_op

fiona-gladwin commented Oct 22, 2025 •

edited

Loading

Uh oh!

LakshmiKumar23 commented Nov 10, 2025

Uh oh!

rrawther commented Nov 11, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fiona-gladwin commented Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

fiona-gladwin commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Uh oh!

LakshmiKumar23 commented Nov 10, 2025

Uh oh!

rrawther commented Nov 11, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fiona-gladwin commented Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

fiona-gladwin commented Oct 22, 2025 •

edited

Loading