@merydian what do you think?

🍎 MacOS Qt6 builds

Download MacOS Qt6 builds of this PR for testing.
This installer is not signed, control+click > open the app to avoid the warning
(Built from commit b731447)

🪟 Windows Qt6 builds

Download Windows Qt6 builds of this PR for testing.
(Built from commit b731447)

Great addition, everything looks good to me. I suppose this should be used in QgsVectorLayer.as_geopandas for better performance?

Do you think implementing this for strings etc. would be worth considering?

@merydian

Great addition, everything looks good to me. I suppose this should be used in QgsVectorLayer.as_geopandas for better performance?

Eventually, yes -- but there's a few more steps first (see below)

Do you think implementing this for strings etc. would be worth considering?

It's not possible to use (variable length) strings in numpy arrays, as all the objects in the array must have equal size. (For this reason I think it'd be possible to support boolean/date/time fields in addition to numeric, but I haven't looked into that).

What I'm thinking for next steps are:

1. Add a similar method to field_as_numpy, for fields_as_numpy (and accompanying QgsVectorLayerUtils.fieldsToDataArray). This would accept a list of (numeric!) fields, and first create an interleaved raw binary blob which is then used as the source of a multi-dimensional numpy array. (The advantage here is iterating ONCE over the data and building up the binary data in one pass)
2. Add a method "as_dataframe". This would scan the layer field types, and for all numeric fields use fields_as_numpy to create a single multidimensional numpy array. Then we'd use the data frame constructor to take columns from that array. For non-numeric fields, we'd have to resort to a similar approach as you used for as_geopandas. We could further optimise this by adding a callback function to QgsVectorLayerUtils.fieldsToDataArray so that we can build the non-numeric column lists during the same layer iteration pass as we use for the numeric ones (otherwise we'd incur two layer iterations, one for numeric and one for non-numeric).
3. Rework as_geopandas to use as_dataframe (or similar code to it) but with the addition of the geometry handling.

Sounds good! Do you have any thoughts on how to split up this work, so that we don't do the same things simultaneously?

Also just noticed that the methods in this PR are called to_numpy, whereas the other implementations are called as_numpy...

Sounds good! Do you have any thoughts on how to split up this work, so that we don't do the same things simultaneously?

How about I add the raw c++ API for fieldsToDataArray, and then you try building the python stuff on top of that?

Also just noticed that the methods in this PR are called to_numpy, whereas the other implementations are called as_numpy...

Thanks, fixed

On second thought, I'd like to try my hand at the whole implementation if that's fine with you? This PR would serve as a nice template I suppose.

@merydian

On second thought, I'd like to try my hand at the whole implementation if that's fine with you? This PR would serve as a nice template I suppose.

Go for it! Just reach out if you get stuck!