[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) by sven1977 · Pull Request #41074 · ray-project/ray

sven1977 · 2023-11-10T14:17:29Z

Preparatory PR: Make EnvRunners use (enhanced) Connector API (#1: mostly cleanups and small fixes)

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 · 2023-11-14T11:35:03Z

rllib/algorithms/algorithm.py

        for agent_id, ob in observations.items():
            worker = self.workers.local_worker()
-            preprocessed = worker.preprocessors[policy_id].transform(ob)
+            if worker.preprocessors.get(policy_id) is not None:


This is a bug fix.

sven1977 · 2023-11-14T11:35:34Z

rllib/algorithms/algorithm_config.py

        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Now that we are aiming for a the EnvRunner API as the default, we should rename/clarify some of these config settings and methods.

Please consider loading a checkpoint here? Are these renaming backward compatible?

Is there even a story around this? Like can people even move from rllib 2+ to 3?

sven1977 · 2023-11-14T11:35:56Z

rllib/core/models/torch/encoder.py

            bias=config.use_bias,
        )

+        self.state_in_out_spec = {


Simplified (repetitive) code.

make this private attribute?

sven1977 · 2023-11-14T11:37:01Z

rllib/env/multi_agent_episode.py

        return self._getattr_by_index("observations", indices, global_ts)

-    def get_actions(
+    def get_infos(


Reordered:

obs, infos (<- env.reset data)

action, reward, terminated/truncated (<- other env.step results)

extra model outs

sven1977 · 2023-11-14T11:37:17Z

rllib/env/single_agent_env_runner.py

-        gym.register(
-            "custom-env-v0",
-            partial(
+        if (


sven1977 · 2023-11-14T11:37:43Z

rllib/evaluation/worker_set.py

        if local_worker and self.local_worker() is not None:
            local_result = [func(self.local_worker())]

+        if not self.__worker_manager.actor_ids():


Shortcut for local-worker only case.

sven1977 · 2023-11-14T11:37:56Z

rllib/tuned_examples/appo/multi-agent-cartpole-crashing-restart-env-appo.yaml

        restart_failed_sub_environments: true

-        # Switch on evaluation workers being managed by AsyncRequestsManager object.
+        # Switch on asynchronous handling of evaluation workers.


AsyncRequestsManager doesn't exist anymore.

sven1977 · 2023-11-14T11:38:20Z

rllib/utils/spaces/space_utils.py

    return input_


+@DeveloperAPI


Very useful new utility. Inverse of already existing unbatch utility.

kouroshHakha · 2023-11-16T04:26:32Z

rllib/algorithms/algorithm_config.py

        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Please consider loading a checkpoint here? Are these renaming backward compatible?

kouroshHakha · 2023-11-16T04:28:05Z

rllib/algorithms/algorithm_config.py

        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Is there even a story around this? Like can people even move from rllib 2+ to 3?

kouroshHakha · 2023-11-16T04:32:01Z

rllib/core/models/torch/encoder.py

            bias=config.use_bias,
        )

+        self.state_in_out_spec = {


make this private attribute?

kouroshHakha · 2023-11-16T05:16:32Z

rllib/utils/spaces/space_utils.py



+@DeveloperAPI
+def batch(list_of_structs, individual_items_already_have_batch_1: bool = False):


data types please (for input and output)

can we have unittest of this ?

done and done

also enhanced the docstring to make the example and explanations more clear.

kouroshHakha · 2023-11-16T05:21:32Z

rllib/utils/spaces/space_utils.py

+            flat = [[] for _ in range(len(flattened_item))]
+        for i, value in enumerate(flattened_item):
+            flat[i].append(value)
+


add:

if item is None: raise ValueError("Input list_of_structs does not contain valid structs.")

kouroshHakha · 2023-11-16T05:21:57Z

rllib/utils/spaces/space_utils.py

+        in this struct represents the batch for a single component
+        (in case struct is tuple/dict). Alternatively, a simple batch of
+        primitives (non tuple/dict) might be returned.
+    """


add

if not list_of_structs: raise ValueError("Input list_of_structs is empty.")

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 · 2023-11-16T11:16:36Z

Thanks for the review @kouroshHakha ! Waiting for tests to pass ...

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…anups

…ay-project#41074)

…41074) (#41212)

…ay-project#41074) (ray-project#41212)

…ay-project#41074)

…ay-project#41074) (ray-project#41212)

wip

017dcfc

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from ArturNiederfahrenhorst, avnishn, kouroshHakha, maxpumperla and smorad as code owners November 10, 2023 14:17

sven1977 assigned kouroshHakha Nov 10, 2023

sven1977 commented Nov 14, 2023

View reviewed changes

rllib/env/single_agent_env_runner.py Outdated

gym.register(

"custom-env-v0",

partial(

if (

Copy link
Copy Markdown

Contributor Author

sven1977 Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug fix.

sven1977 commented Nov 14, 2023

View reviewed changes

sven1977 mentioned this pull request Nov 14, 2023

[RLlib] New ConnectorV2 API #02: SingleAgentEpisode enhancements. #41075

Merged

8 tasks

kouroshHakha approved these changes Nov 16, 2023

View reviewed changes

wip

768b88c

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 added 2 commits November 16, 2023 12:27

wip

1b7d1cc

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge branch 'master' into env_runner_support_connectors_01_minor_cle…

aed527f

…anups

sven1977 merged commit ca29fec into ray-project:master Nov 17, 2023

sven1977 deleted the env_runner_support_connectors_01_minor_cleanups branch November 17, 2023 11:29

rickyyx mentioned this pull request Nov 22, 2023

[ci][core] Perf regression on tasks_per_second, pgs_per_second #41338

Closed

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023

[RLlib] New ConnectorV2 API #1: Some preparatory cleanups and fixes. (r…

f608186

…ay-project#41074)

sven1977 added a commit that referenced this pull request Dec 21, 2023

[RLlib] New ConnectorV2 API #3: Introduce actual ConnectorV2 API. (#…

bd555a0

…41074) (#41212)

vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024

[RLlib] New ConnectorV2 API #3: Introduce actual ConnectorV2 API. (r…

ad4e256

…ay-project#41074) (ray-project#41212)

simonsays1980 pushed a commit to simonsays1980/ray that referenced this pull request Dec 17, 2025

[RLlib] New ConnectorV2 API #1: Some preparatory cleanups and fixes. (r…

1b9ff72

…ay-project#41074)

simonsays1980 pushed a commit to simonsays1980/ray that referenced this pull request Dec 17, 2025

[RLlib] New ConnectorV2 API #3: Introduce actual ConnectorV2 API. (r…

66cf2eb

…ay-project#41074) (ray-project#41212)



		@DeveloperAPI
		def batch(list_of_structs, individual_items_already_have_batch_1: bool = False):

Conversation

sven1977 commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sven1977 commented Nov 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sven1977 commented Nov 10, 2023 •

edited

Loading