Technical design pair: @ansd.

Proposed Changes

This PR makes the most expensive part of federation link shutdown — closing AMQP 0-9-1 connections to the upstream — parallel, by notifying links in the prep_stop shutdown callback.

This yields very significant efficiency gains with hundreds or thousands of links, all without changing the supervision tree structure.

Why Not Use simple_one_for_one?

Indeed the simple_one_for_one OTP supervisor restart strategy would shut down all child processes concurrently for us. But this would require changing the child identity (key)
to an Erlang PID, which would require intrusive and painful to test changes (such as
an ETS table that maps their PIDS to the current identities and the other way around).

Throttling to Avoid Overwhelming the Upstream

To avoid overwhelming the upstream schema data store (which could be a 7-9 node cluster on 3.x with Mnesia), we limit the degree of parallelism and add batching with configurable throttling delays into the process.

The entire link shutdown process is now capped at 180 seconds (by default), and should not meaningfully exceed that time period even on nodes with many thousands of links.

By default we close up to 128 links per batch, with a 50 ms delay, and a 180 second hard cap (timeout) for the entire link termination operation.

Data Safety Considerations

Federation uses publisher confirms by default, and most users never change it, therefore
aggressive connection closures are safe and acceptable.

In addition, the user can set resource-cleanup-mode to never to make sure that the
upstream resources (e.g. internal queues used by exchange federation) are never deleted
by the links running in the downstream cluster.

Show Me The Benchmark Data

Microbenchmarks (Supervisor Child Process Termination)

Below are some microbenchmarks that measure everything beyond the actual
AMQP 0-9-1 connection termination part on an 8 core aarch64 CPU from 2022:

┌───────┬──────────┬────────────┬─────────┐
│ Links │ Parallel │ Sequential │ Speedup │
├───────┼──────────┼────────────┼─────────┤
│ 1,000 │ 16ms     │ 6,401ms    │ ~400x   │
├───────┼──────────┼────────────┼─────────┤
│ 5,000 │ 30ms     │ 32,689ms   │ ~1,000x │
└───────┴──────────┴────────────┴─────────┘

Worst Case Scenario Calculations

If we consider the worst case scenario where every link connection hits its timeout,
1K links would take about 83 minutes to start for the sequential (status quo) version and 5.6 seconds (see below) with these changes.

Real World Federation Links with Outgoing Connections

With the throttling delay of 0, the time it takes to shut down N links to a remote upstream
cluster look like this:

┌───────┬─────────────────┐
│ Links │      Time       │
├───────┼─────────────────┤
│ 10    │ 100ms           │
├───────┼─────────────────┤
│ 50    │ 580ms           │
├───────┼─────────────────┤
│ 100   │ 1,067ms         │
├───────┼─────────────────┤
│ 1,000 │ 5,579ms         │
└───────┴─────────────────┘

Maintenance Mode Integration

Maintenance mode integration of these changes needs to be done with care: since maintenance mode stops all client connection listeners, we run the risk of stopping the listeners
before this part of the federation shutdown has a chance to do its job as designed.

For that reason, we have to special case the federation plugins in the core and first trigger their termination, then stop the listeners.

When the node is revived (the maintenance mode is rolled back), all links are restarted.