Skip to content

Conversation

@comphead
Copy link
Contributor

@comphead comphead commented Dec 10, 2024

Which issue does this PR close?

Related to #13726
Closes #.

Rationale for this change

Test Windows compilation time improvements

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the development-process Related to development process of DataFusion label Dec 10, 2024
@comphead
Copy link
Contributor Author

Executed "map.slt". Took 460.5636141s
Executed "window.slt". Took 217.240136s
Executed "struct.slt". Took 1316.049712s
Executed "array.slt". Took 2407.7054095s

Thats really weird, a simple array SLT takes 40 mins

@comphead comphead changed the title CI: Windows compilation time CI: Windows test execution time Dec 10, 2024
@korowa
Copy link
Contributor

korowa commented Dec 15, 2024

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

@comphead
Copy link
Contributor Author

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

Thanks @korowa for the hint I was playing with lots of combination including different Windows images but nothing seems to work.

@comphead
Copy link
Contributor Author

Some projects give up rust testing on Windows like DataDog/orchestrion#415

@alamb wondering should we pause the Windows flow temporarily? The build time is incredibly slow and we holding GH resources for way longer time

@alamb
Copy link
Contributor

alamb commented Dec 15, 2024

Another thing that would benefit the project as a whole is to reduce build time.

One thing I noticed a while ago is that each binary built (e.g. every example) was around 100MB

I was thinking that we could reduce / conslidate the number of examples / binaries built which would speed up all runners (not just Windows)

For example consolidating https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/regexp.rs and other examples of using functions into dataframe_api.rs might be a good win

@comphead
Copy link
Contributor Author

Closing it as there is no any progress after all the experiments

@comphead comphead closed this Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants