What happened?

After some time, only one event emitted by a composition function can be seen.

This occurs because we are exceeding the event refill rate. By default each object gets an allowance of 25 events, which has a refill rate of 1 event every 5 minutes. Crossplane itself will emit a normal system event for a successful composition run once per reconcile (1 minute by default). This means that by itself Crossplane will fill up the event limit, and exceed the refill rate.

The following behavior is observed as a result:

• For successful reconciles with no events from composition functions, the event limit will be reached and a new event will only be seen every 5 minutes. When the event appears, it will increase the event count by 5.
• User events emitted by functions will prevent the system events from being seen after the limit is reached for as long as the user event is emitted.
• Only the first user event can be seen.
• Fatal results have highest precedence.
• If the user event stops being emitted, the system event will be seen the next time the event refill occurs. The system event will be shown with the full count of missed events (e.g., if crossplane attempted to emit the system event 100 times, the event will go from 0 to 100 as soon as it appears).

How can we reproduce it?

Create a claim or composite resource, wait for the event limit to be reached (~30 minutes). Optionally, configure the composite resource to be processed by a composition function that emits one or more events.

What environment did it happen in?

Crossplane version: 1.16
Cluster: kind
Image: kindest/node:v1.29.2