https://issues.apache.org/jira/browse/ARROW-11044

@jorisvandenbossche: Would you have you look at this in-progress PR and share your thought ? Replace kernel implemented for most (if not all) types but missing null handling.

Note on ArrayData::GetValues(i) in the context of commit ab2d3d2:

• ArrayData::GetValues(i) use "byte offset" even for buffers[2] which stores variable width values...
  => Should it rather use buffers[1] value offset ?
  => Or force absolute_offset to 0 for buffers[2] ?

On my case, I fixed the bug in my code by forcing absolute_offset to 0.

I wouldn't expect a "replace" operation to do this. Instead, this looks more like a "select" operation. @wesm What do you think?

This might actually rather be the "setitem" kernel as proposed in https://issues.apache.org/jira/browse/ARROW-9430 ? (which is something we want as well)

@jorisvandenbossche: Thank you for bringing up ARROW-9430.

Then @ALL please tell me what would be a good name.

I may have to rework the code a bit. As mentionned in the top description, I started based on fill_null kernel code and the "null handling" logic came as an afterthought, so the code might look a bit convoluted right now...

"setitem" is confusing to me (I would expect something where you pass indices and values to set at those indices). "select" is used in Numpy. Another possible is "cond".

"setitem" is confusing to me (I would expect something where you pass indices and values to set at those indices).

That's https://issues.apache.org/jira/browse/ARROW-9431.
There could be a variant of "setitem" that takes integer indices, and one that takes a boolean mask?

But it's true that for a "setitem" kernel, you would expect the "values to set" (Replacement in the example in the top post) to be of the length of the number of values that get set, not of the same length as the original array.

So what this PR proposes, is actually more a "where" kernel (using numpy's terminology), for which we have https://issues.apache.org/jira/browse/ARROW-10640.
There is some discussion on that PR that this can be generalized to something like numpy's np.choose. And the np.select mentioned by @pitrou is quite related to that as well (the exact semantics are a bit different between both: select uses boolean conditions to choose between the arrays, choose a single array of indices into the choice arrays)

I'd say that personally, I find "select" more intuitively guessable than "where" (which is a weird name for a function). But I could also live with it if it's Numpy's choice. @michalursa

Agreed that "where" is an unintuitive name. For me it's fine to directly use the more general name from numpy, which would actually be both "choose" or "select" (I think the single boolean mask can be seen as a special case for both generalized functions: assuming that the single boolean mask from "where" is interpreted as 0/1 integer indices) for "choose", or is interpreted as a boolean mask + its inverse mask for "select").

Another option mentioned in the JIRA is "if_else" (term used in the R world) or "case_when" (inspired on SQL).

"if_else", "choose", "select" are all fine with me.

Wow, I was not expecting such brainstorming on the name of this kernel. My initial intent was to mimic pandas.Series.replace(to_replace, value), where in C++ to_replace could be an BooleanArray mask (same length as input array) which trigger value replacement => So this kernel could be used after any combination of Compare and Boolean operations to implement specific replacement logic.

This could be extended to another implementation where to_replace could be an IntegerArray (length <= as input array) of indexes to replace.

Here is how I use "replace" in combination with "is_nan" (ARROW-11043) to implement fillna():

template <typename value_type>
std::shared_ptr<DataFrame> DataFrame::fillna(value_type value) {
    auto outdf = std::make_shared<DataFrame>();
    ...
    ...
    for (int i = 0 ; i < this->table_->num_columns() ; ++i) {
        auto field = this->table_->schema()->field(i);
        auto chunked_array = this->table_->column(i);
        auto value_datum = arrow::compute::Cast(arrow::Datum(value), chunked_array->type()).ValueOrDie();
        auto nulls = arrow::compute::IsNull(chunked_array).ValueOrDie();
        auto nans = arrow::compute::IsNan(chunked_array).ValueOrDie().chunked_array();
        auto to_replace = arrow::compute::Or(nulls, nans).ValueOrDie();
        auto clean_chunked_array = arrow::compute::Replace(chunked_array, to_replace, value_datum).ValueOrDie().chunked_array();
        outdf->table_ = outdf->table_->AddColumn(i, field, clean_chunked_array).ValueOrDie();
    }

    return outdf;
}

Wow, I was not expecting such brainstorming on the name of this kernel.

Sorry for the distraction :-) That said, for clarity it seems that "if_else" would be both explicit and non-misleading.

The signature could be if_else(cond, when_true, when_false) with the semantics cond ? when_true : when_false. Nulls would probably just pass through.

Do you want to update this PR?

@bu2 would you have time to update this based on the above comments? (#9024 (comment))

Closing as superceded by #10410.