Currently, the spec says "The first filter function or filter reference in the list takes the element's BackgroundImage as the input image."
This was a mistake on my part. BackgroundImage is defined as the backdrop up to the isolation parent.
What we really want is a new term, BackdropImage, which is the backdrop of the element, including all parents and siblings that have been painted up to the point of the filter. There is no isolation.