Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Aug 11, 2025

Which issue does this PR close?

Rationale for this change

As explained on the ticket

DataFusion's built in ListingTable will soon have metadata caching 🎉

I think we should make sure this feature is more discoverable via the docs

What changes are included in this PR?

Revamp and extend documentation for LIstingTable and the various CacheManager structures

Are these changes tested?

By Docs ci

Are there any user-facing changes?

Better docs. No function changes intended

@alamb alamb added the documentation Improvements or additions to documentation label Aug 11, 2025
@github-actions github-actions bot added core Core DataFusion crate execution Related to the execution crate and removed documentation Improvements or additions to documentation labels Aug 11, 2025
@alamb alamb marked this pull request as ready for review August 11, 2025 21:42
Copy link
Contributor

@jonathanc-n jonathanc-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lgtm, some very minor comments, gets all the points across to users. @nuno-faria can probably add some deeper insight

Copy link
Contributor

@nuno-faria nuno-faria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Co-authored-by: Nuno Faria <nunofpfaria@gmail.com>
Co-authored-by: Jonathan Chen <chenleejonathan@gmail.com>
Copy link
Member

@timsaucer timsaucer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation!

/// Recently Used eviction algorithm.
/// Create a new instance of [`DefaultFilesMetadataCache`].
///
/// # Arguments
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found 2 another places where arguments highlighted like this whereas other functions describe arguments in free style. Just thinking aloud

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree the docs are inconsistent in this regard.

I personally think that a whole section is only really useful when the arguments are non obvious or there is some nuance to them (such as this case). However, that is a subjective judgement call for sure

I think it would not add much clarity if we, for example, added an # Arguments section for all functions, though I do think this is what is done in some other projects

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alamb

@alamb
Copy link
Contributor Author

alamb commented Aug 15, 2025

🚀 I am trying to close out some tickets on Friday so merging this to start the weekend on a high note

Thanks for the reviews everyone

@alamb alamb merged commit 75eb80c into apache:main Aug 15, 2025
27 checks passed
@alamb alamb deleted the alamb/document_listing_table_caching branch August 15, 2025 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate execution Related to the execution crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Parquet Metadata Cache] Document the parquet metadata cache [Epic] Enable parquet metadata cache by default

5 participants