Update for FreeBSD

tl;dr; I have a branch that compiles and passes all the tests on FreeBSD.

Take this as a very experimental first step.

Non comprehensive TODO list

Provide a more elegant shim layer instead of inlining #ifdefs which is nasty
Discuss / decide what to do with hot restart; is it ok to use files instead of abstract sockets on FreeBSD?
Is it ok to disable TCMalloc?
Make OSX pass all tests
consistent mutex is not available on Darwin.

See: https://github.com/lyft/envoy/compare/master...Reflejo:freebsd?expand=1

What was done

TCMalloc disabled (commit)

envoy compiled w/TCMalloc seg faults on OSX and creates deadlocks on FreeBSD on very specific situations while removing events with event_del. I got deep into this deadlock for hours before I realized that TCMalloc was contributing to this (probably indirectly but still).

Make currentThreadId multiplatform (commit)

I decided to use pthread_getthreadid_np on FreeBSD for convenience which I believe it was added around fbsd 9. Alternatively pthread_self could be used which is more portable.

Add kqueue implementation for monitoring symlinks (commit)

inotify is not available on BSD hence why I implemented the watcher using kqueue. There is a caveat though, envoy is mainly using the watcher for symlinks but freebsd doesn't support O_SYMLINK (i.e. get a file descriptor from a symlink). I included a kernel patch for freebsd11 adding this functionality. FWIW OSX supports O_SYMLINK.

Handle `EAGAIN` when querying `SO_ERROR` on bsd (commit)

When a connect() is performed on a non-blocking socket, it'll return -1/EINPROGRESS immediately and when select()'ed for writability it'll succeed or fail but when it fails the reason is hidden. getsockopt(,,SO_ERROR,,) solves this issue on some systems but on some cases such as FreeBSD getsockopt returns 0 even on EAGAIN.

This was a big problem on freebsd that'd hang connections so instead of trusting on SO_ERROR I'm also calling getpeername() which will return -1/ENOTCONN if the socket is not connected and read(,,1) will produce the right errno.

UUID implementation on FreeBSD (commit)

Envoy was getting uuids from /proc/sys/kernel/random/uuid on linux (which afaik uses urandom). Instead I changed it to use uuid_generate_random on Linux and hand craft a uuid using arc4random on bsd.

Hot restart on BSD doesn't use abstract sockets (commit)

envoy uses abstract sockets which are only supported on Linux. As a temporary hack; I'm just using concrete files on FreeBSD, but a decision needs to be taken for the whole hot restart system to make it more portable.

Fix tests on FreeBSD (commit)

Some tests were making assumptions that only worked on Linux (for example at what point on the event loop a connection would be open/ready). I fixed some tests that were using NonBlock.

Additionally I had to disable TcpClientConnectionImplTest.BadConnectNotConnRefused since FreeBSD will happily connect to almost any ip (even checking the kernel code I couldn't find a case that'd return ENETUNREACH on a common gw configuration).

OSX

The delta from here to make it compile in OSX is actually very small. With this change it'll compile but I still haven't try to make the tests pass.

cc @dch Since I'm on and off here (more off than on), feel free to take over this and let me know how I can help.

OSX support #128

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions