In our environmental modeling ticket #1278 I discuss enabling plugin-based models for costmaps as well as height maps. This will help bring a 2D navigation system into the unstructured 3D space. This could also be done using a set of costmap plugins that look at input pointclouds for traversibility and making intraversible as lethal, but always better to have it natively supported in order to make the most use of your vision and robot drive systems.

However, another area has been emerging, the use of urban delivery robots that largely make use of vision systems rather than actually modeling the environment around them. Typically they use something like semantic segmentation for traversible space where your local trajectory planner lets you go and the usual AI detector models to detect / track people. This doesn't involve any direct grid-based modeling or even changing the pixel values into 3D space to trajectory plan in.

We want to be able to support all major uses of robots, so this is another area we should consider moving forward as well. Either as another environment model plugin, or with urban-specific planner/controller servers and algorithms working on them.

If no gridmaps for planning, then can't be using a search / sampling based free-space planner, must be using a sparse navigation graph of intersections / via-points. Does this generalize to all non-urban outdoor use as well? Probably uses semantic maps (via points, gps points, intersection metadata) to do planning / routing -- not costmap or freespace planning. Control probably still uses the local window height / cost map. make a route server separate from the planner server using the semantic info for planning?

Other questions to ask:

• Do they use GPS to the larger route planning / global path planning? We have GPS support with rolling global costmaps already.
• Some place visual SLAM comes into play? Also support coming here in Nav2 very shortly.
• Any other vision systems an urban robot like yours might need and needs to be integrated into the navigation system?
• Any open-source pre-trained semantic segmentation models for urban outdoor robots for users to start with?
• How are the pixel-wise semantic segmentation models used in a local trajectory planner (and the style of planners used). DRL makes sense here, though flaky performance at best unless trained over a ton of data. Is it possible to use other planners on this? Or perhaps you transform plans from 3D space into the camera coordinate space given the marked traversibility space (e.g. for a DWA-like planner, come up with the motion primitives for the robot to traverse, and use the camera extrinsics to project those lines to the camera image frame and score based on the pixel-wise mapping + other metadata)
• For planar environments like indoors, I think the occupancy grid info is fine for 3D and VSLAM to enable path planning. But I think we need to come up with a game plan or understand how we can do this outdoor as well where we dont have that planar assumption, or if the entire method that we do planning in Nav2 isn't in line with how outdoor/urban robots do planning (e.g. no grid-based planning, GPS/VSLAM/3D localization and using pre-defined semantic maps to navigate along in all cases). Maybe that semantics work is more important than I thought to make a seperate indoor and outdoor planner server and algorithms based on the costmap or semantic info?