Address Filtering for Address Lookup Services

Why

The Context

PR #3691 surfaced an assumption mismatch between users and iroh's address lookup service. This user expected that relay URLs were being published over mDNS and could not understand why connections weren't being established. We never considered publishing relay URLs on mDNS internally and so our implementation only publishes ip addresses and user defined data.

It turned out there was a valid reason the user needed this: their environment had a firewall that permitted broadcast UDP traffic (which mDNS uses) but blocked other kinds of UDP traffic. So mDNS address lookup was working, but without a relay URL in the advertisement, the two endpoints had no way to reach each other — direct connections were blocked and there was no relay fallback advertised. The fix for their use case was to include the relay URL in mDNS advertisements.

This incident revealed that we had left the choice of what to and not to publish to the address lookup implementations entirely, and didn't leave room for the possibility/flexibility that users may have specific needs while advertising their addresses.

Custom Transports Make This Worse

The upcoming Custom Transports feature will introduce a Custom(CustomAddr) variant to TransportAddr, allowing users to plug in arbitrary transport layers (e.g., Bluetooth). This sharpens the problem:

• Users may want to advertise a custom transport address on one address lookup service but not another.
• Most of our existing address lookup services are built on top of DNS TXT records, which have hard byte-size limits. Custom transport addresses will likely need to be encoded into these records, and there may simply not be room for all of them, and we have no guidelines on what to do if this happens.
• There is no current mechanism for users to express preferences about which addresses get published where.

This is manageable — an address lookup service can document its constraints and make best-effort decisions about what it can fit — but we need to give users a way to participate in that decision. For example, in a byte-constrained mDNS advertisement, a user might want to ensure their preferred custom transport address is published first, rather than leaving the ordering up to the address lookup service itself.

So What Do We Need?

We need a mechanism that:

1. Makes it clear to users that they can influence which addresses each address lookup service publishes.
2. Makes it clear to address lookup implementors that they need to reckon with publishing addresses that are not just IP addrs.
3. Still allows each address lookup service to make its own final decisions about what it can actually publish, given its own constraints.

The fourth and nice-to-have would be some way to force the address lookup implementors into ensuring they have good documentation about the preference of the address lookup service when it comes to illustrating what it can and cannot publish (in terms of kind of addresses as well as number of addresses). Haven't figured this out, so it may just be that we lead by example and have good documentation ourselves.

What

Proposed Design

We introduce a concrete AddrFilter type and add required methods to the AddressLookup and IntoAddressLookup traits that ensures every address lookup implementation accepts a filter. Whether or not they use/respect the filter, is not something we can control in this setup, but it at least gives the expectation that we expect it.

AddrFilter

/// A filter and/or reordering function applied to transport addresses before publishing.
///
/// Takes the full set of transport addresses and returns them as an ordered `Vec`,
/// allowing both filtering (by omitting addresses) and reordering (by controlling
/// the output order). A `BTreeSet` cannot preserve a custom order, so the return
/// type is `Vec` to make reordering possible.
///
/// See the documentation for each address lookup implementation for details on
/// what additional filtering the implementation may perform on top.
// KASEY:
/// Our recommendation would be that the address lookup service will apply this
/// before its own internal filtering, before it publishes.
#[derive(Clone)]
pub struct AddrFilter(
    Arc<dyn Fn(BTreeSet<TransportAddr>) -> Vec<TransportAddr> + Send + Sync + 'static>
);

impl AddrFilter {
    // KASEY: I don't know if this is necessary, but would likely make it nicer to
    // work with
    pub fn new(
        f: impl Fn(BTreeSet<TransportAddr>) -> Vec<TransportAddr> + Send + Sync + 'static
    ) -> Self {
        Self(Arc::new(f))
    }

    // KASEY: I don't know if this is necessary, but would likely make it nicer to
    // work with
    pub fn apply(&self, addrs: BTreeSet<TransportAddr>) -> Vec<TransportAddr> {
        (self.0)(addrs)
    }
}

Changes to IntoAddressLookup

We need to be able to add the AddrFilter when building the discovery service. To do this, we can add a required with_addr_filter method is added to IntoAddressLookup

pub trait IntoAddressLookup: std::fmt::Debug + Send + Sync + 'static {
    /// Sets the address filter for this address lookup service.
    fn with_addr_filter(self, filter: AddrFilter) -> Self where Self: Sized;

    fn into_address_lookup(
        self,
        endpoint: &Endpoint,
    ) -> Result<impl AddressLookup, IntoAddressLookupError>;
}

Usage

What would this look like?

endpoint
    .address_lookup(PkarrPublisher::n0_dns()
        .with_addr_filter(AddrFilter::new(|addrs| {
            // only publish relay and IP addresses to pkarr, not custom transports,
            // returned as a Vec to allow reordering
            addrs.into_iter()
                .filter(|a| !matches!(a, TransportAddr::Custom(_)))
                .collect::<Vec<_>>()
        })))
    .address_lookup(MyBluetoothLookup::new()) // handles Custom addrs internally

Alternatives

We also change the AddressLookup trait to require a set_addr_filter.

It's likely we only need to set this once, when the address lookup is built. BUT if we want to give users the ability to adjust this dynamically, it may be best to also add a set_addr_filter or with_addr_filter method on the AddressLookup trait itself. However, the user would need to hold onto a reference/clone of the address lookup service themselves, since calling this on ep.address_lookup().with_addr_filter would likely mean setting that specific filter to all the different lookup services that exists. But maybe we want that functionality too!

Each address lookup system has its own builder methods, no trait change

Each address lookup service builder could independently add a filter method to its own builder. No changes to the traits would be required.

I rejected this because it meant no consistency and no way to enforce even limited consistency/expectations. Users have no reason to expect filtering to be available or to look for it and address lookup service implementators get no guidance.

This isn't a concern for the address lookup services at all - but for the endpoint

The other option is that this isn't a setting on the address lookup service, but is instead a setting on the endpoint. There are two issues here, I believe.
First, this would likely mean having a single blanket filter/reduce, so the user can't customize it per-address lookup. Second, since publish expects an EndpointAddr with a BTreeSet<TransportAddr> we cannot preserve order once we turn the vec back into a btreeset. The other other option would be then to adjust the field in the EndpointAddr to be a vec, but I don't think we want that?

Do something weird with Ord and TransportAddr

Tbh I'm not sure where you would start with this, since you would want it to be user supplied...

Other options?

I am open to hearing them :)

Potential issues

This gives tools and expectations, not guarantees

The AddrFilter mechanism gives users a way to express intent — "these are the addresses I want published to this service" — and creates the expectation that the address lookup service will use it. But the address lookup service is the final authority over what it actually publishes, so it could decide not to use the user supplied filter and can decide for itself what addresses it wants to publish.

But likely, this is for the best. Address lookup services can have structural constraints the user cannot override. Eg, An address lookup service that encodes addresses as DNS TXT records cannot publish infinite CustomAddr variants it needs to be able to confidently only publish what it can. Other services may have reasons they can't publish them entirely.

But this also means we should have a lot of documentation about our expectations.

The filter application should be consistent across all implementation:

1. The user-provided AddrFilter is applied first.
2. The address lookup service's own internal filtering is applied second.

Also, every address lookup implementation should document:

• Whether and how the user-provided AddrFilter is applied.
• What additional internal filtering the implementation applies (e.g., "only relay URLs are published; direct addresses are dropped regardless of the filter").
• Any capacity limits that may cause addresses to be dropped even after filtering (e.g., "only the first N addresses are published due to DNS TXT record size constraints").
• When capacity limits exist, whether addresses are dropped silently or with a warning log.

The goal is that a user reading the documentation for any address lookup service can predict exactly what will be published, without reading the source.

Feedback

I'd love a way to have more guarantees here, but I haven't figured out the structure that would allow it. In fact, potentially the fact that this isn't a 100% rigid structure may actually be beneficial for some address lookup services.

Behavior Changes

By default, no address lookup service applies any filtering — all addresses are passed through to publishing as-is. Filtering only occurs if an AddrFilter is explicitly set, either via a preset or a user-supplied closure. This is a breaking change for Pkarr publishing, which checked to see if a relay url existed, and if so, cleared all ip addresses, and Dht publishing, which had an explicit include_direct_addresses option.

For users using our defaults, this will not effect them. For users that have relays disabled, this will effect them if they use the default n0 preset, BUT, likely this will not effect most, because a user that has disabled relays they probably have not used the default n0 preset, and instead used the empty_builder and built whatever address lookup services they needed on their own.

Presets

Since we are leaving the choice for what to publish to the users, I've updated the Presets to include that the default explicitly filters address to only include the relay url. This is the same behaviour as before, but needed to be updated because the option to include_direct_addresses for, eg DhtAddressLookup, was removed.

New preset: N0DisabledRelay
I added a new preset that filters relays from published addresses and sets the relay mode to disabled. Not married to keeping it, but could be a good shorthand for folks. Did not add this as an option on the Endpoint::builder (eg Endpoint::disabled_relay_builder), however.

Breaking Changes

• iroh
  • removed
    • DhtAddressLookupBuilder::include_direct_addresses removed. You can effect what addresses are published by using the new DhtAddressLookupBuilder::set_addr_filter method.
  • changed
    • trait IntoAddressLookup renamed to AddressLookupBuilder

Change checklist

• Self-review.
• Documentation updates following the style guide, if relevant.
• Tests if relevant.
• All breaking changes documented.