To use the changes from this PR as an external component, add the following to your ESPHome configuration YAML file:

external_components:
  - source: github://pr#12605
    components: [sensor]
    refresh: 1h

(Added by the PR bot)

👋 Hi there! This PR modifies 3 file(s) with codeowners.

@esphome/core - As codeowner(s) of the affected files, your review would be appreciated! 🙏

Note: Automatic review request may have failed, but you're still welcome to review.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.60%. Comparing base (6edecd3) to head (a167760).

@@            Coverage Diff             @@
##              dev   #12605      +/-   ##
==========================================
- Coverage   74.62%   74.60%   -0.03%     
==========================================
  Files          53       53              
  Lines       11347    11347              
  Branches     1541     1541              
==========================================
- Hits         8468     8465       -3     
- Misses       2471     2473       +2     
- Partials      408      409       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Memory Impact Analysis

Components: sensor, template
Platform: esp8266-ard

Note: This analysis measures static RAM and Flash usage only (compile-time allocation).
Dynamic memory (heap) cannot be measured automatically.
⚠️ You must test this PR on a real device to measure free heap and ensure no runtime memory issues.

This analysis runs automatically when components change. Memory usage is measured from a merged configuration with 2 components.

I had a PR to do something similar: #8096

The danger with this PR is if there is a legitimate step (larger than the user expects) the filter never recovers and will never report a new value. If there is a step its ok to ignore it for a bit but if the value settles there shouldn't the filter start reporting values again?

Good point, that danger does exist. It would indeed be nice to be able to handle this.

We can introduce a timeout where the reference value is reset whenever it expires when no valid values are passed for the specified period.

Another approach could be to implement a sliding window that records all measurements and have the filter only produce values as long as all measurements are within bounds.

I think I would lean towards the second approach. What do you think?

Good point, that danger does exist. It would indeed be nice to be able to handle this.

We can introduce a timeout where the reference value is reset whenever it expires when no valid values are passed for the specified period.

Another approach could be to implement a sliding window that records all measurements and have the filter only produce values as long as all measurements are within bounds.

I think I would lean towards the second approach. What do you think?

Instead of a sliding window or timeout could just have a counter? Would count the last number of measurements in bounds and would reset to zero if a measurement is out of bounds? Would pass the value forward if the counter is over a threshold.

This is sort of what I did in my attempt I always set the last value to the current value. That is another option instead of a counter.

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

Instead of a sliding window or timeout could just have a counter? Would count the last number of measurements in bounds and would reset to zero if a measurement is out of bounds?

I would suggest just making the delta templatable (if it's not already) then it can be the output of another filter using whatever smoothing technique the user determines, rather than trying to turn this filter into a swiss army knife.

Added - probably need two values: a reference (by default the previous value) and a delta, both templatable.

Could also just add an inverted flag to the regular delta filter and always set last in inverted mode. Simple

Could also just add an inverted flag to the regular delta filter and always set last in inverted mode. Simple

Doesn't solve the problem of what to use as the reference point. Maybe enhance the delta filter so it can take a dict like:

   - delta:
      max_delta: 20%
      min_delta: 10
      baseline: !lambda return id(smoothed_value).state;

Requiring at least one of max_delta and min_delta and where smoothed_value could be a template sensor consuming the raw value of the base sensor with some kind of average. Or enhance the copy sensor to allow choosing the raw value of the source. Or just use two copy sensors, one for the baseline, one that has the delta filter.

The default for baseline would be the previous value of course.

Yeah having it default to the previous and allowing it to be templateable sounds good to me.

Could do:

   - delta:
      max: 20%
      min: 10%
      baseline: !lambda return id(smoothed_value).state;

Instead of a sliding window or timeout could just have a counter? Would count the last number of measurements in bounds and would reset to zero if a measurement is out of bounds? Would pass the value forward if the counter is over a threshold.

It would be dependent on the polling rate, so maybe I would opt for a timeout as a time period but this sounds good to me.

Could also just add an inverted flag to the regular delta filter and always set last in inverted mode. Simple

I am hesitant to do this since it effectively tightly couples the implementations of the delta filter and this filter. It's fine for the implementation I now proposed, but we are discussing timeouts and baselines which would warrant too much complexity to also burden the delta filter with imo.

    - delta:
       max: 20%
       min: 10%
       baseline: !lambda return id(smoothed_value).state;

The baseline sounds neat, but how would we get the smoothed_value? Do we expect users to apply a median before the new delta limit filter? That would force the output of the whole chain to be smoothed too. Or is there a kind of branching of filter chains happening? If so, how would that work?

Something like this will work. While it's tempting to include features in the filter itself to simplify the usage for your immediate objective, the drawback is that it limits what is possible, whereas this approach makes it infinitely configurable for use-cases that we haven't thought of yet.

sensor:
  - platform: adc
    pin: GPIO4
    id: input_id

  - platform: copy
    id: smoothed_value
    source_id: input_id
    filters:
      - sliding_window_moving_average:
          window_size: 20
          send_every: 1

  - platform: copy
    id: output_value
    source_id: input_id
    filters:
      - delta:
          max: 20%
          min: 1%
          baseline: !lambda return id(smoothed_value).state;

There is a lot of information online about outlier detection, but most apply to batch processing of captured data, not real-time streaming. I did find this one and it could be implemented with two copy sensors to calculate the moving average and standard deviation.

Alternatively it could be implemented as a dedicated filter with just a threshold parameter for the Z-score.

But I think the enhancements as discussed for the delta filter would be useful, and easy to implement.

Agreed, this looks like a good solution. I'll see what I can do

I also still have concerns about how this is not stable. If say a wireless sensor goes offline for a bit then comes back online. The delta could exceed max and will never be passed though again until the value comes back down (could be never) as the filter never recovers.

I also still have concerns about how this is not stable. If say a wireless sensor goes offline for a bit then comes back online. The delta could exceed max and will never be passed though again until the value comes back down (could be never) as the filter never recovers.

Any thoughts on how to deal with that? If a custom baseline is provided, it's not a problem. Perhaps if the value becomes unknown (NaN) that should be stored into last_value so the next valid value will re-establish the baseline.

Regarding the stability for the max_value, the baseline is meant to account for this. I have included a full example in the documentation update linked in the OP

Happy new year, all!

Is there anything left that needs to happen before this can be merged?

Is there anything left that needs to happen before this can be merged?

Marking the change requests as resolved is useful as it tells the reviewer that you have addressed the feedback (either made changes or verified they're not required.)

Thanks all :)