@XianlinSheng Hey, sorry, I know this PR is months old at this point. Just wondering, what is the purpose of this being an enum? Is there anything else that could really go in here other than a Rect? Just wondering as I'm adding functionality around this and want to know how generic I should keep things

Thanks for asking this question first. XZBBox is the mask of the region of generation. Normally it is a rectangle aligned with grid lines, but after rotation or other transformations this region can no longer be simply described like this. At this moment I already have a complete general polygon implementation in my branch that is necessary for rotation, so you can go to my dev-xzpoly to check out how it is designed. One can choose a form to instantiate as long as the shape can fully describe what you have now. For users of this structure, they only call the 'contains' (check a point is included). 'bounding_bbox' (circumscribed grid-aligned rectangle) method, display/add/sub traits, and possible transformations on it, so adding a new shape needs 1. 'contains' method. 2. 'bounding_bbox' method. 3. display/add/sub traits. 4. supports for your shape in all map transformation operators (it won't compile if the new shape is not covered anywhere that needs to modify the xzbbox).

The purpose of using a XZBBox is to truncate the OSM data and let it generate only inside the region. The fetched OSM data is not just the points inside the LLBBox you selected on the map but includes the complete continuous elements that are partially covered - like selecting objects in modeling software, you will select the whole object as long as the region touches it. An element may extend even much further than the llbbox dimension, and this is the reason why impl #504 found that the generation was still in the original rectangle after the rotation transform on existing coordinates - the existing data is not just the points inside the selected region, and we need to use XZBBox to restrict the generation of the redundant elements outside of it.

This design will build the necessary blocks for features like generation at non origin #29 (using rectangle), rotation #97, clip #97 (using polygon), custom shape, projection-corrected #29 (using other shapes), non-continuous space #459 (general mask). The polygon impl already has a mask field inside for the inference performance of method 'contains', and the rotation code for elevation data gives the idea to rotate a field, so a general mask box is also in fact ready-to-go, simply rearranging some existing methods.

May I know your plan so we can reach the optimal implementation?

Thanks, makes sense!

Basically I'm working on generating the world in batches, where each batch is an n chunk by n chunk square. This is to lower RAM usage, something that users have been complaining about (and even trying to script their way around batching, which kind of works, but has issues)

The way I'm doing things right now is to divide the XZBBoxRect up into these chunks, which are then saved to disk. Doing it this way is quite nice because it means we'll never touch the same chunk twice. Of course, that invariant doesn't hold if XZBBox isn't axis-aligned. Maybe you have some thoughts here?

Maybe there could be an Intersection(XZBBox, XZBBox) variant to XZBBox`. Then the first one could be the user-defined shape, and the second one could be the n chunk by n chunk batch.

Let me understand this situation completely and better before I give you my thoughts. Which process and which data structure is taking up the most RAM? By dividing you mean generating it in multiple steps so it won't store everything at the same time?

The goal is to batch the entire pipeline, i.e. only fetch what we need from overpass for that specific batch, then run it through the entire thing, and save it out to the region file. There will be duplicated work, especially if the batch size is too small, but it would make it feasible to create huge worlds (on the order of entire provinces/states)

OK it's completely alright to have different ideas and find which is the really the optimal and suitable for this case, but that is after both proposals are really understood clearly.

The role of Python is not a second programming language, it's not part of the program, but another user interface, CLI, GUI, Python Shell, and a programmable interface has much greater abilities than these two. It will not be the developers to write Python scripts but users. Still, developers can provide some commonly needed scripts ahead of time as presets. The reason I propose this batch can be written as a Python preset is that I saw some ideas that need a programmable interface will be benefited from such a preset. Like the world generation considering projection corrections (or any approach to generate large world as you wish) in #29 needs a highly customized and programmable way to arrange the pipelines of how to combine the small jobs and form a large world. If such a preset exists, not just the users with resource problems can call the function, world designers can also call it to generate it without knowing dividing details. Projection and city design professionals will write their Python scripts and perform tests and studies to generate optimal combinations of small jobs. They don't care the rust details in generation a job, their scripts are not part of this project and they should have the freedom to choose not to focus too much on computer science but on their professionals, which is the strongest power of Python as a glue.

What you proposed makes sense and is in fact not contradictory to a Python shell. It's just a matter of on which side we realize it of the interface. I do not insist which side it should be, what I care is to make it useful to as more features as possible. So of course we can still write this in rust, but we better consider to extend the ability of the interface too (it can be just the GUI/CLI now) to make full use of this feature.

Right, I'm still concerned with the amount of lift and complexity it will add, plus how much it will slow down other feature development (we'll need to consider all sorts of edge cases as a result of allowing arbitrary user pipelines)

I definitely don't think this is something that'd be needed for #29. It needs a few extra settings at most. Maybe if they had the ability to construct arbitrary pipelines they could use them to solve #29, but it's still a lot of work, just given to the user as a Python script, instead of in the Rust core that can be more beneficial to everyone.

I understand what you mean and definitely we won't start with too big an architecture. These are just some inputs that might be useful when you decide how to expose the batch feature to user, it might be some possibilities if you are thinking how to make it work for as many cases as possible, but don't feel stressed with the information and these are not rules.

Let's back to the original question because either way some problems always need to be solved. The current pipeline includes the map transformation and what's your current idea to make it fit in batches? The map transformation is defined for one pipeline only, if defining it to work on the original map selection, each batches will need to perform different operations, like rotating an original selection does not mean the same rotation on small jobs.

Yeah, might be something we need to figure out for the future. For now I am probably going to only support batching on axis-aligned bounding boxes

OK you can keep working on your code and I'll try to figure out a way to let this support. I'll let you know first once I have some ideas.