[Feature] Simplified multiagent config with asymmetric observation spaces

[mgerstgrasser]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In #16565 and #17046 an improvement was made to allow RLLib to determine the observation and action space from the environment in multiagent environments. This only works when the spaces are the same for all agents though. I want to propose extending this to allow for different spaces for each agent. This could be implemented as a relatively small change, by letting env.observation_space and env.action_space be multiagent dicts holding the respective spaces for each agent.

Use case

Similar to the original PRs mentioned, this would simplify configuration in multiagent settings, and obviate the need to create a dummy env at configuration time just to read off each agent's action and observation space. The original PRs addressed this for symmetric spaces, this would bring the same benefits to environment with different spaces for each agent.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[mgerstgrasser]

@sven1977 I think you probably know the most about this since you wrote the original PRs referenced. I think it's a relatively small change and I'd be happy to submit a PR for it, but I wanted to check what you thought first. Particularly since it slightly changes what env.action_space and env.observation_space (can) mean in a multiagent env. I don't know of anything else that touches either of these so I don't think making these multiagent dicts should break anything, and in practice with our own code it works, but I might have missed something of course.

[gjoliver]

The code base may have this assumption baked in at a lot of places.
We are planning some work for RLlib 2.0 that should make it lot easier to use different policy and spaces for different agents from the env.