Documentation preview:

• ✨ Changed pages

@limotova I extracted values from the serverless test and combined them to reproduce the test failure. Using three batches, the final value is 0.23282704603226836, while with a single batch, the value is 0.22797190865484734. Could you check if this discrepancy is acceptable?

    public void testBasic() {
        double[] v1 = {1.97, 2.0, 1.57, 1.48, 1.77};
        double[] v2 = {2.1, 1.74, 1.96, 1.42, 1.59, 2.07, 1.81, 1.59, 1.44, 2.03, 1.81};
        double[] v3 = {2.03, 1.54, 1.55};

        WelfordAlgorithm a1 = new WelfordAlgorithm();
        for (double v : v1) {
            a1.add(v);
        }

        WelfordAlgorithm a2 = new WelfordAlgorithm();
        for (double v : v2) {
            a2.add(v);
        }

        WelfordAlgorithm a3 = new WelfordAlgorithm();
        for (double v : v3) {
            a3.add(v);
        }

        WelfordAlgorithm merged = new WelfordAlgorithm();
        for (WelfordAlgorithm a : List.of(a3, a2, a1)) {
            merged.add(a.mean(), a.m2(), a.count());
        }

        System.err.println("--> merged = " + merged.evaluate());

        WelfordAlgorithm single = new WelfordAlgorithm();
        for (double v : v1) {
            single.add(v);
        }
        for (double v : v2) {
            single.add(v);
        }
        for (double v : v3) {
            single.add(v);
        }
        System.err.println("--> single = " + single.evaluate());
    }

Flyby feedback:

• std_dev(first_name) fails with an unfriendly error message: org.elasticsearch.xpack.esql.EsqlIllegalArgumentException: Cannot find intermediate state for: AggDef[aggClazz=class org.elasticsearch.xpack.esql.expression.function.aggregate.StdDev, type=BytesRef, extra=, grouping=false]. This probably comes from the lack of resolveType() method implementation.
• std_dev(salary_change) I think deserves a test, meaning the aggregation function applied on a multi-value field. You are using salary_change in a test, but you apply mv_max on it, reducing it to a single value field.
• another recent functionality we introduced for stats is the filter specific to an aggregation function. stats std_dev(salary) where languages > 3. It would be good to have a test for this as well. For example, FROM employees | stats std_dev(salary_change + 1) where languages > 3, std_dev(salary_change + 1) where languages <= 3, count(*) by gender
• the row command is a different type of functionality where the source of the data is "static" and it doesn't come from ES. Would be good to have some tests with row as well. For example row a = [1,2,3], b = 5 | stats std_dev(a), std_dev(b) by a

@astefan

another recent functionality we introduced for stats is the filter specific to an aggregation function. stats std_dev(salary) where languages > 3. It would be good to have a test for this as well. For example, FROM employees | stats std_dev(salary_change + 1) where languages > 3, std_dev(salary_change + 1) where languages <= 3, count(*) by gender

I tried to add in tests with salary_change + 1, but since some of the values are empty it looks like it doesn't handle the + 1 very well (it seems like it ends the calculation as soon as it hits an empty value), is this expected? It works as expected with just salary_change (it skips empty values and calculates everything else), and it also handles something like salary * 2 (not multivalue, but no empty values) fine.
(Actually based on the test failure it might not be working properly... I'm unable to reproduce it locally though and am a bit stumped why it's happening)

the row command is a different type of functionality where the source of the data is "static" and it doesn't come from ES. Would be good to have some tests with row as well. For example row a = [1,2,3], b = 5 | stats std_dev(a), std_dev(b) by a

It seems to work with row as far as I can tell, but I'm not sure I understand what I should be looking for when I add by a here? The standard deviation of only one unique value is always 0, but since a is a multi-value here, the result of std_dev(a) by a is the same as std_dev(a), just the former repeats the result 3 times for each value of a

💚 Backport successful

Status	Branch	Result
✅	8.x