Awesome! I saw you mentioned you could use help with manual testing, but can you discuss a bit what automated testing could look like? We have access to many platforms via Github runners so that's where my head is at.

CI will at least make sure that my library builds on Windows.

Transcripts that import DLLs could also be written, of course. But that requires a DLL. I guess we could also build one of those in CI, but I'd have to do some reading on how to get that done.

It's just going to bomb if you try to send one of these to another machine.

I don't really see how you could support that with any kind of assurance. DLLs are just files, so even if you sent the information used to load from the DLL on one machine, there's no guarantee that the file with the same name is actually the same library, or the same version of it with identical calling conventions, etc. And if you send something from Windows to Linux, the file names for 'the same' library are probably different.

I would say that if you want to send code around, you send code that uses some API that it gets passed in (possibly via abilities), and that API is implemented locally by foreign functions. The code you send around is portable, but the local implementation of the API isn't. Sending around DLL references that somehow adapt to the local system doesn't seem like a realistic expectation.

Just brainstorming: we could have a Github job that installs the current build of ucm, installs whatever binary libs, whether it's zlib, libcurl, libmysqlclient, whatever we feel is going to exercise the FFI well, and then run Unison programs against them.

Or could these function references serialize just fine and work on the other end as long as the other end has the "same" symbol loaded? If achievable, the sandbox-checking should definitely treat all foreign functions as tracked - you do not want to be able to load a library locally, then hop to the cloud cluster and call the same C functions if the same C libraries happen to be installed in the same place.

Could a library bundle some kind of library-specific test to determine if the compatible function exists on the other side?

I would say that if you want to send code around, you send code that uses some API that it gets passed in (possibly via abilities), and that API is implemented locally by foreign functions. The code you send around is portable, but the local implementation of the API isn't. Sending around DLL references that somehow adapt to the local system doesn't seem like a realistic expectation.

I'm pretty sold by this argument. Just pass around functions that use an ability, much more portable.

So I think bombing is fine during serialization, though I'd like it be with a meaningful error that tells the user what they did wrong.

Okay, decision here: we're not going to do anything fancy for transferring FFI pointers over the network, will just bomb with an error. In userland, you could build something which is more portable (using abilities, say).

@dolio is going to get CI fixed up and at least doing smoke tests on Linux / Mac / Windows. @aryairani can advise on this.

Other types like additional primitives and arrays will likely be a follow-up PR.

I wonder if there's any reasonable way we can use the type-system (or some other compiler mechanism) to tag a block as being unable to be serialized.

There's already a good amount of frustration with things like Debug.trace hiding inside pure expressions which prevents them from being evaluated in Share docs;
I could see similar frustrations where you're writing normal unison code, but one of your calls into a library uses FFI and all of a sudden your whole app bricks because now you can't send it over Remote or serialize that continuation like you were before, and you don't find out until runtime.

We have the workaround that you can just express your ffi stuff as an ability and run that ffi outside of where that serialization happens, but I think it'd be a lot more user friendly if we could enforce or hint at that pattern so newer unison users don't get surprised or bitten by this.

I can imagine a system ability like IO, e.g. {Unserializable} which can't be handled except in main;

Not sure exactly how that'd work, and may not need to be in the first iteration, but I think if we're breaking Unison's promise that you can serialize anything that it's good to think of the UX.

I wonder if there's any reasonable way we can use the type-system (or some other compiler mechanism) to tag a block as being unable to be serialized.

It's definitely possible but I do not want to do that in this PR. :) I really want this PR and the core FFI functionality to be low level and unsafe and as out of the way as possible.

Adding fancy types could instead be done in various ways as separate userland layers. Like if you're planning to do distributed stuff, I think it's a reasonable thing to pass around functions that use an ability rather than direct pointers to the foreign functions. But again, I don't want to get fancy with this PR or the basic FFI primitives.

Also, it's not clear to me that abilities are able to track this. The problem with serialization is that any reference to the value in the closure will fail, not just if you apply the function.

I suppose you could arrange to fail remotely, by serializing something and only failing if you try to apply the function remotely. Then perhaps you could argue that an annotation on the imported function is showing you that its effects won't be supported remotely. But I'm not sure how good that is, either.

BTW, it looks like Runar added pinned arrays as a separate type, so they should be relatively easy to add as FFI arguments without people having to worry about whether it's safe to pass. It requires a little tweaking of the implementation (not the interface), though. So I'll leave it to a future PR.

I think this is ready to go, unless anyone has objections.

Yeah looks good to me.