FYI @XiangpengHao @zhuqi-lucas and @JigaoLuo as you may be interested in this content

FYI @nuno-faria @shehabgamin @jonathanc-n @zhuqi-lucas and @etseidl as you are mentioned in the blog post

This PR is now ready for review

Really solid blog post!

Unrelated to this blog post, but I feel the content should have a max width so it becomes easier to read in larger displays. For example, this is how it looks by default:

While this is what it looks with 1000px max width:

Nice blog @alamb. Thanks for having me here! I’ve done my first pass, and I think the topic is great. I’ve left a few comments in the review.

One note on the structure, and it might be worth discussing here as well:

• I found the titles of the subtopics we’re covering to be quite clear, but the number of top-level sections (# in Markdown) seems to exceed the actual number of distinct subtopics.
• That’s also why I think the “Apache Parquet Overview” section felt a bit out of place—it appears suddenly in the middle of the blog.
  • I definitely think including background on Parquet is important. What I meant was that we could consolidate all the Parquet-related background content into a single top-level section, rather than having it scattered throughout.
• It’s possible I misunderstood the intended structure, so feel free to clarify if that’s the case.

So glad to see this blog post coming together, nice work @alamb 🚀

I am starting to work through the comments left on this PR

Unrelated to this blog post, but I feel the content should have a max width so it becomes easier to read in larger displays. For example, this is how it looks by default:

Filed a ticket to address that

• Set max width on blog to make it easier to read in larger displays #100

@JigaoLuo

Nice blog @alamb. Thanks for having me here! I’ve done my first pass, and I think the topic is great. I’ve left a few comments in the review.

❤️ thank you for taking the time to provide feedback

One note on the structure, and it might be worth discussing here as well:

• I found the titles of the subtopics we’re covering to be quite clear, but the number of top-level sections (# in Markdown) seems to exceed the actual number of distinct subtopics.

@nuno-faria mentioned this too and I have demoted all sections one level. Hopefully that is clearer.

• That’s also why I think the “Apache Parquet Overview” section felt a bit out of place—it appears suddenly in the middle of the blog.
  • I definitely think including background on Parquet is important. What I meant was that we could consolidate all the Parquet-related background content into a single top-level section, rather than having it scattered throughout.
• It’s possible I misunderstood the intended structure, so feel free to clarify if that’s the case.

In my mind there is a balance between:

1. Showing / demonstrating how to use external indexes for Parquet using DataFusion
2. Explaining the general concept of external indexes / heirarchal pruning

I believe the post will be more widely read if it is about more than just Parquet and Datafusion, and that by having the background content it will be easier for people to even realize this is a technique that they can use.

So I guess I would say the structure is deliberate, but I can see how it may not be obvious

Let me know if that makes sense

Thank you all for your comments. I think I have addressed all of them and I will plan to address any more comments and publish this blog on Friday, Aug 15

Thanks again -- the feedback on this draft post was really helpful