Description

Currently Windows does not support edge based events. As a result we rely on level based events which causes various issues.

For example in testlist //test/common/network:connection_impl_test FlushWriteAndDelayCloseTimerTriggerTest the scenario that depends on setting a setDelayedCloseTimeout on the server.

This event on Windows is added to the queue in Windows but it gets buried by the Level write events. Some logs that indicate this behavior are here:

[debug] event_del: 000001FB4DB1FC78 (fd -1), callback 00007FF76C608820
[debug] event_process_active: event: 000001FB4D6E48F8,  EV_WRITE  call 00007FF76C4C6670
[2020-06-30 15:05:41.026][6448][trace][connection] [source/common/network/connection_impl.cc:506] [C0] socket event: 2
[2020-06-30 15:05:41.026][6448][trace][connection] [source/common/network/connection_impl.cc:607] [C0] write ready[debug] event_del: 000001FB4DB1FC78 (fd -1), callback 00007FF76C608820
[debug] event_process_active: event: 000001FB4D6E48F8,  EV_WRITE  call 00007FF76C4C6670
[2020-06-30 15:05:41.026][6448][trace][connection] [source/common/network/connection_impl.cc:506] [C0] socket event: 2
[2020-06-30 15:05:41.026][6448][trace][connection] [source/common/network/connection_impl.cc:607] [C0] write ready

This causes the test to hang and exposes a limitation for envoy on windows, to set Scheduled Callbacks and Timer events.

Proposed Solution: Set event priorities

We could utilize libevent event_base_priority_init to create more granular priorities on jobs.

Create an enumeration with priorities and use it when we create new SchedulableCallbackImpl

enum eventPriorities:
{
   Standard,
   High
}

SchedulableCallbackImpl::SchedulableCallbackImpl(Libevent::BasePtr& libevent,
                                                 std::function<void()> cb, eventPriorities job_priority)

This behavior could be limited only to Windows (or to both if we would think it has wider value)

Questions

1. How important is the scenario of Timers and Scheduled Callbacks?
2. Are there any scenarios that this could cause a regression?

@nigriMSFT @sunjayBhatia @wrowe