Skip to content

Put heavy dataframe dependencies behind feature flag#9971

Merged
rgwood merged 1 commit intonushell:mainfrom
rgwood:dataframe-feature
Aug 10, 2023
Merged

Put heavy dataframe dependencies behind feature flag#9971
rgwood merged 1 commit intonushell:mainfrom
rgwood:dataframe-feature

Conversation

@rgwood
Copy link
Copy Markdown
Contributor

@rgwood rgwood commented Aug 10, 2023

Context from Discord: https://discord.com/channels/601130461678272522/615962413203718156/1138694933545504819

I was working on Nu for the first time in a while and I noticed that sometimes rust-analyzer takes a really long time to run cargo check on the entire workspace. I dug in and it was checking a bunch of dataframe-related dependencies even though the dataframe feature is not built by default.

It looks like this is a regression of sorts, introduced by #9241. Thankfully the fix is pretty easy, we can make it so everything important in nu-cmd-dataframe is only used when the dataframe feature is enabled.

Impact on cargo check --workspace

Before this PR: 635 crates, 33.59s
After this PR: 498 crates, ~20s

(with the mold linker and a cargo clean before each run, the relative difference for incremental checks will likely be much larger)

Copy link
Copy Markdown
Member

@sholderbach sholderbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will make operations like cargo test -p nu-cmd-dataframe a bit surprising, but I think it is overall a good thing for the project.

@rgwood rgwood merged commit d5fa7b8 into nushell:main Aug 10, 2023
@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Aug 10, 2023

I'm having a hard time compiling with dataframes now.

cargo install --path . --features=dataframe,extra,mimalloc 
  Installing nu v0.83.2 (/Users/fdncred/src/nushell)
    Updating git repository `https://github.com/nushell/reedline.git`
    Updating crates.io index
   Compiling nu v0.83.2 (/Users/fdncred/src/nushell)
error[E0425]: cannot find function `add_dataframe_context` in crate `nu_cmd_dataframe`
  --> src/main.rs:45:42
   |
45 |     let engine_state = nu_cmd_dataframe::add_dataframe_context(engine_state);
   |                                          ^^^^^^^^^^^^^^^^^^^^^ not found in `nu_cmd_dataframe`

For more information about this error, try `rustc --explain E0425`.
error: could not compile `nu` due to previous error
error: failed to compile `nu v0.83.2 (/Users/fdncred/src/nushell)`, intermediate artifacts can be found at `/Users/fdncred/src/nushell/target`

sholderbach added a commit to sholderbach/nushell that referenced this pull request Aug 10, 2023
Building broke silently.

`nu` needs to enable the `dataframe` feature of `nu-cmd-dataframe`

Still worth investigating if we can harden the CI against this.
fdncred pushed a commit that referenced this pull request Aug 10, 2023
# Description
Building broke silently.

`nu` needs to enable the `dataframe` feature of `nu-cmd-dataframe`

# User-Facing Changes
Building with `cargo build --features dataframe` works again.

# Tests + Formatting
Still worth investigating if we can harden the CI against this.
@fdncred
Copy link
Copy Markdown
Contributor

fdncred commented Aug 10, 2023

Stefan figured it out #9974

@rgwood
Copy link
Copy Markdown
Contributor Author

rgwood commented Aug 10, 2023

Oops, thanks Stefan!

sholderbach added a commit to sholderbach/nushell that referenced this pull request Aug 31, 2023
Same logic as in nushell#9971

Prevents building the heavy polars and arrow dependencies when just
running `cargo test --workspace` or `rust-analyzer`
sholderbach added a commit to sholderbach/nushell that referenced this pull request Aug 31, 2023
Same logic as in nushell#9971

Prevents building the heavy polars and arrow dependencies when just
running `cargo test --workspace` or `rust-analyzer`
sholderbach added a commit that referenced this pull request Aug 31, 2023
Same logic as in #9971

Prevents building the heavy polars and arrow dependencies when just
running `cargo test --workspace` or `rust-analyzer`

`polars-io` dependency was introduced in #10019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants