Skip to content

Add FlatArray<'_, T>#2207

Merged
workingjubilee merged 53 commits into
pgcentralfoundation:developfrom
workingjubilee:add-flatarray
Nov 30, 2025
Merged

Add FlatArray<'_, T>#2207
workingjubilee merged 53 commits into
pgcentralfoundation:developfrom
workingjubilee:add-flatarray

Conversation

@workingjubilee

@workingjubilee workingjubilee commented Nov 17, 2025

Copy link
Copy Markdown
Member

Introduce a new Rust type for what ArrayTypes "really are": FlatArray. As an unsized type, it will primarily be used as &FlatArray<'_, T> or similar in actual extension code. The long-term plan is for FlatArray to phase out pgrx::datum::Array<'_, T> entirely, as it has several advantages over its predecessor:

  • It uses Element, a subset of BorrowDatum, to power its simplicity in iteration, making it more easily verifiable and maintainable.
  • It is the unsized type itself, allowing composing it with various Rusty pointer types like &, &mut, or PBox.
  • It is no longer endangered by performance cliffs or hidden allocations depending on the type of T.
  • Because of these previous traits, it is possible to soundly and safely allocate a fresh FlatArray, mutate it, and return that one allocation, instead of allocating an intermediate Vec in Rust memory to mutate!

It also has an associated cost in that it no longer supports any T that requires complex unboxing of items like datum::Array<'a, String>. You must instead use array.iter().map(closure) to accomplish that. This itself summarizes the difference: datum::Array is effectively an iteration protocol, but array::FlatArray is an actual data structure.

This addresses the implied feature request in, and thus supercedes, #2086.

@workingjubilee

workingjubilee commented Nov 17, 2025

Copy link
Copy Markdown
Member Author

There's one mystery of why the lifetime isn't something I can use, here, but I'll leave that one for tomorrow. It will require digging into our macro expansion code, which is always "fun".

@workingjubilee workingjubilee force-pushed the add-flatarray branch 2 times, most recently from ad6143c to 667dc32 Compare November 17, 2025 15:58
@workingjubilee

workingjubilee commented Nov 17, 2025

Copy link
Copy Markdown
Member Author

Dismantling this incrementally into other PRs so that it becomes more obvious what is actually a key part of this PR overall, and also to make it easier to pinpoint the macro/lifetime issue.

@workingjubilee

Copy link
Copy Markdown
Member Author

Biggest remaining question for this PR that I've been thinking about:

  • Is BorrowDatum the right constraint, or should it be ArrayElement?

@workingjubilee

Copy link
Copy Markdown
Member Author

The answer is yes, partly because FlatArray itself is BorrowDatum, which means it runs afoul of #1765

@workingjubilee

Copy link
Copy Markdown
Member Author

I think the other Q is "can we get rid of the inner lifetime?"

@workingjubilee workingjubilee changed the title Add FlatArray<'_, T> and PBox<'_, T> Add FlatArray<'_, T> Nov 22, 2025
@workingjubilee

Copy link
Copy Markdown
Member Author

In order to make this work, we need to break BorrowDatum apart and introduce Element, which provides most of its functionality.

Element then receives a blanket implementation of BorrowDatum for it.

@workingjubilee workingjubilee force-pushed the add-flatarray branch 2 times, most recently from b799fc6 to 500148e Compare November 24, 2025 20:42
@workingjubilee

workingjubilee commented Nov 24, 2025

Copy link
Copy Markdown
Member Author

TODO

  • Make less tests in array_borrowed commented-out
  • Write some more functional tests walled on SPI
  • Think about if there's any more compile-fails that should be added
  • Rebase on Add array::Element and callconv::DatumPass #2218 when that is landed
  • Rewrite PR description
  • Investigate multidimensional array support

@workingjubilee

workingjubilee commented Nov 29, 2025

Copy link
Copy Markdown
Member Author

trybuild does not support build-fail tests, despite the name, so mono-time checks aren't viable to test for, annoyingly.

Comment on lines +67 to +70
TooManyBytes,
TooManyElems,
/// One or more dimensions are zero
ZeroLenDim(usize),
ZeroLenDim,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While returning the information we had was cute, it wasn't viable in all cases without compromising the code significantly, and it was always at risk of being incomplete, so I stopped doing that.

Comment thread pgrx/src/array/flat_array.rs Outdated
@workingjubilee

Copy link
Copy Markdown
Member Author

Hm. I think I should/could probably add testing that only relies on direct calls instead of SPI, and that would be enough for some basic functionality, even though it doesn't test the "full cycle".

@workingjubilee workingjubilee merged commit c019e91 into pgcentralfoundation:develop Nov 30, 2025
14 checks passed
@workingjubilee workingjubilee deleted the add-flatarray branch November 30, 2025 18:26
daamien pushed a commit to daamien/pgrx that referenced this pull request Dec 15, 2025
Introduce a new Rust type for what ArrayTypes "really are": FlatArray.
As an unsized type, it will primarily be used as `&FlatArray<'_, T>` or
similar in actual extension code. The long-term plan is for FlatArray to
phase out `pgrx::datum::Array<'_, T>` entirely, as it has several
advantages over its predecessor:

- It uses `Element`, a subset of `BorrowDatum`, to power its simplicity
in iteration, making it more easily verifiable and maintainable.
- It is the unsized type itself, allowing composing it with various
Rusty pointer types like `&`, `&mut`, or `PBox`.
- It is no longer endangered by performance cliffs or hidden allocations
depending on the type of `T`.
- Because of these previous traits, it is possible to soundly and safely
allocate a fresh `FlatArray`, mutate it, and return that one allocation,
instead of allocating an intermediate Vec in Rust memory to mutate!

It also has an associated cost in that it no longer supports any `T`
that requires complex unboxing of items like `datum::Array<'a, String>`.
You must instead use `array.iter().map(closure)` to accomplish that.
This itself summarizes the difference: `datum::Array` is effectively an
iteration protocol, but `array::FlatArray` is an actual data structure.

This addresses the implied feature request in, and thus supercedes,
pgcentralfoundation#2086.
eeeebbbbrrrr added a commit that referenced this pull request Feb 9, 2026
Welcome to pgrx v0.17.0. This is a new minor release that brings a bunch
of internal code refactoring, cleanup, new Postgres headers, and
additional `cargo-pgrx regress` CLI options.

As always, please install the latest `cargo-pgrx` with `cargo install
cargo-pgrx --version 0.17.0 --locked` and also update your extension
crate dependencies with `cargo pgrx upgrade`.

## What's Changed


### New Headers/Symbols

* include `access/tsmapi.h` by @usamoi in
#2155
* Include access/subtrans.h & access/commit_ts.h by @isdaniel in
#2147
* include `utils/guc_tables.h` by @usamoi in
#2243
* Include `scanner.h` in Rust bindings by @piki in
#2163
* Add `commands/publicationcmds.h` to generated `pg_sys` bindings by
@Sinjo in #2221
* Add the ClientAuthentication_hook by @daamien in
#2231
* add storage/dsm_*.h headers by @eeeebbbbrrrr in
#2244
* pg-sys: add catalog/heap.h bindings for PG18 by @charmitro in
#2150
* Add back `peer_dn` field to `Port` on `pg17..` by @workingjubilee in
#2200
* Add `repr(C)` to `struct Port` on `pg13..=16` by @workingjubilee in
#2201

### `cargo-pgrx` improvements

* Add a psql_verbosity parameter for cargo pgrx regress by @daamien in
#2230


### Code Cleanup

* Replace with let-chains in pgrx-bindgen by @workingjubilee in
#2168
* Replace with let-chains in pgrx-sql-entity-graph by @workingjubilee in
#2169
* Restore so-called needless lifetimes by @workingjubilee in
#2167
* Drop stray Postgres 12 support in cargo-pgrx by @workingjubilee in
#2173
* Remove deprecated enum##member constants by @workingjubilee in
#2170
* Remove `variadic!` macro by @workingjubilee in
#2171
* Semi-automated code cleanup by @workingjubilee in
#2189
* Deref instead of borrowing `Option<&String>` by @workingjubilee in
#2185
* Refactor schema.rs using a cargo command builder by @workingjubilee in
#2187
* Replace `impl AsRef<T>` with `&T` by @workingjubilee in
#2182
* Make `PgRelation::heap_relation` unsafe by @workingjubilee in
#2176
* Move `cargo_pgrx::env` to `cargo_pgrx::cargo` by @workingjubilee in
#2186
* Move `CargoProfile` into `cargo_pgrx::cargo` by @workingjubilee in
#2188
* Add Scalar trait for generic handling of primitive arrays by
@workingjubilee in #2153
* Set BGWORKER_SHMEM_ACCESS when building background workers by @cbandy
in #2161
* Stop referencing `pg_config` in schema generation by @workingjubilee
in #2174
* Show line number of failed asserts by @piki in
#2175
* Ghostbust `used_type.rs` a bit by @workingjubilee in
#2190
* Manually bind `static MyProcPort` and `struct Port` by @workingjubilee
in #2162
* Make datetime types import each other via `use super` by
@workingjubilee in #2198
* Migrate datetime types into `pgrx::datetime` by @workingjubilee in
#2199
* Use diagnostic attrs for ABI/SQL-related traits by @workingjubilee in
#2203
* Make `Scalar` provide `pg_sys::Oid` constant by @workingjubilee in
#2209
* Hoist varlena encoding fn by @workingjubilee in
#2208
* Recurse through `T` while anonymizing `&T` by @workingjubilee in
#2212
* `impl BorrowDatum for TimeTz` by @workingjubilee in
#2213
* Change `MemCx::alloc_bytes` to return `NonNull` by @workingjubilee in
#2214
* Add `MemCx<'current, T>` arguments and `PBox<'current, T>` returns by
@workingjubilee in #2210
* Further improve diagnostics for signatures with no SQL form by
@workingjubilee in #2215
* Add `array::Element` and `callconv::DatumPass` by @workingjubilee in
#2218
* Remove unused line in `memcxt_tests.rs` by @workingjubilee in
#2220
* Add `FlatArray<'_, T>` by @workingjubilee in
#2207
* Allow explicit handling of allocation errors by @workingjubilee in
#2226
* Place lbound before updating array len product by @workingjubilee in
#2236
* introduce `#[pg_guard(unsafe_entry_thread)]` by @eeeebbbbrrrr in
#2242


### Project Administrativa 

* Use Rust Edition 2024 by @workingjubilee in
#2165
* Remove testing for deprecated Intel Macs by @workingjubilee in
#2156
* Remove unused Cargo.lock for version-updater by @workingjubilee in
#2180
* Add rust-analyzer.toml by @workingjubilee in
#2179
* Note pgrx requires an unknown minimum Xcode version by @workingjubilee
in #2164

## New Contributors

* @isdaniel made their first contribution in
#2147
* @cbandy made their first contribution in
#2161
* @piki made their first contribution in
#2163
* @Sinjo made their first contribution in
#2221

**Full Changelog**:
v0.16.1...v0.17.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant