reboot wit-bindgen-go

[dicej]

This re-introduces Go support, with a new implementation written from scratch. While the previous incarnation targeted TinyGo and reused the C generator for lifting and lowering, this one targets "big" Go and handles lifting and lowering itself. In addition, it supports idiomatic, goroutine-based concurrency on top of the component model async ABI, including streams and futures.

This implementation is also distinct from the
go-modules project, which has a number of limitations (e.g. no async support, not safe for use with "big" Go) and is no longer being actively maintained.

Note that the async support currently requires a small patch to the Go runtime library. I plan to work with the upstream project to make that patch unecessary in the future. Components that don't use async features should work with stock, unpatched Go.

One of the tricky parts about lowering values and passing pointers to e.g. {stream,future}.{read,write} is that we must tell the Go garbage collector to pin the pointers (i.e. mark the pointee as both immovable and uncollectable) for as long as the host has access to them, but then unpin them promptly afterward to avoid leaks. We do this using runtime.Pinner instances, most of which are locally scoped and easy to reason about. However, we must use a couple of globally-scoped pinners as well: one for cabi_realloc allocations by the host when it lowers parameters (which we unpin after lifting those parameters) and another for returning values from sync-lifted exports (which we unpin in post-return functions).

There are a couple of known missing features which I'll open GitHub issues for:

1. Resource handles are not being restored for unwritten items in the case of an incomplete stream or future write.
2. Importing and/or exporting multiple versions of the same package will cause name clashes.

In addition, I plan to expand the test coverage beyond the basics covered in this commit.

[dicej]

I just realized I need to make CI run the go tests. I'll push an update.

[alexcrichton]

Agreed we'll want to ge this running on CI, but overall looking great! I only skimmed codegen though and would mostly lean on tests to exercise bits and bobs

[dicej]

I was so focused on the runtime tests that I never ran the codegen ones. Looks like I have more work to do. I'll switch this to draft mode while I do that.

[ydnar]

Thanks to @dgryski for pointing me to this!

@dicej this looks great. We solved the multiple versions issue by optionally generating a versioned Go package, e.g. wasi/http/v0.2.1.

Does this require the WASI 0.1 adapter to function in big Go?

[dicej]

@dicej this looks great. We solved the multiple versions issue by optionally generating a versioned Go package, e.g. wasi/http/v0.2.1.

Good idea. Currently we generate package names like wasi_http_0_2_1_types because I'm a Go novice, but versioning the Go package makes more sense.

Does this require the WASI 0.1 adapter to function in big Go?

Yes, currently. I've opened golang/go#76775 as a baby step towards adding GOOS=wasip3 upstream. Meanwhile, you can use any p2 or p3 features you like by generating bindings from the WASI WIT files and using them directly.

BTW, here's an example that shows off concurrent outbound requests, etc. Eventually we'll want to do something like you did here to make that use net/http interfaces, etc.

[ydnar]

I haven’t dug too deeply into the code—how does it handle the lifting/lowering of pointer types?

Big Go uses 64-bit pointers, which means that record types containing pointers (list, string) aren’t ABI compatible with WASI.

Our plan with the go-modules repo and wit-bindgen-go was to rely on big Go merging support for GOARCH=wasm32. TinyGo’s pointer size on wasm is 32-bit.

[dicej]

I haven’t dug too deeply into the code—how does it handle the lifting/lowering of pointer types?

It doesn't assume the canonical ABI layout of a given record or variant type matches the layout of the Go representation. Instead, the record fields and variant payloads are lifted and lowered one-at-a-time, with pointers converted between uintptr and uint32 as necessary. We know those conversions will be lossless because there's no way to create a valid pointer with any of the high 32 bits set in a wasm32 module.

[ydnar]

Got it. So when a pointer to a list containing pointers is passed, does that require copying on every function call?

We were able to guarantee the Go binary representation matched the canonical ABI, to avoid copying.

[dicej]

Got it. So when a pointer to a list containing pointers is passed, does that require copying on every function call?

Yes, and not only that. We must also pin those pointers for GC safety when converting them to the canonical ABI, and also unpin them when converting from the ABI. The pin/unpin requirement will remain even if/when GOARCH=wasm32 becomes a thing, so lifting and lowering lists containing pointers will remain O(n) operations regardless.

We were able to guarantee the Go binary representation matched the canonical ABI, to avoid copying.

Indeed; Big Go required a different approach, though.

[dgryski]

What needs to be done to this implementation to allow this to work with TinyGo?

[dicej]

What needs to be done to this implementation to allow this to work with TinyGo?

I don't yet have a great mental model of how TinyGo differs from Big Go, so I don't know. Does TinyGo support e.g. runtime.Pinner?

Also, IIRC, TinyGo relies on Asyncify to support concurrency. We'd need to hook into that somehow to bridge goroutines with the component model async ABI and thereby ensure that I/O operations only block the calling goroutine, allowing others to continue running. For Big Go, I had to patch the runtime library to get special access to the scheduler; not sure what the equivalent would be for TinyGo.

One way to start would be to make an example app based on bindings generated by this tool and try building it with TinyGo to see what breaks.

[dicej]

BTW, how does go-modules handle GC safety with TinyGo? It looks like TinyGo uses the BDW conservative GC library. Is there some mechanism to ensure allocations made by the host via cabi_realloc are left untouched during GC until the host has finished using them?

[dgryski]

We have a partial fix here tinygo-org/tinygo#4897 but will need a better one. We don't support pinning (yet) but we also don't have a moving collector. (There three garbage collectors available in TinyGo.). Supporting "don't free this yet" flags would fall under something similar to the above PR.

Yes, moving to wasip3 would mean moving away from asyncify, which is something we're looking forward to getting rid of.

[dicej]

We don't support pinning (yet) but we also don't have a moving collector.

You might already know this, but runtime.Pinner doesn't just prevent an allocation from moving but also ensures that it isn't collected. So perhaps TinyGo's version of runtime.Pinner could just store the pointers it is given and make sure they're treated as roots by the GC.

[dgryski]

Yes, that's what I meant by the "don't free this yet" flag. The PR I linked to attempts to avoid that by storing pointers in a map, but having wit-bindgen-go explicity pin/unpin allocations would solve some of the issues that PR had.