FFI support

[headius]

This PR will build out support for loading the RBS library via FFI, avoiding the C extension on implementations that don't support the CRuby extension API.

[headius]

I believe these two classes are the only two defined by the C extension, correct? I am taking the approach of trying to mimic the C ext's API but implement what it does via FFI (not yet in this PR).

[headius]

I've made an attempt to start binding the FFI backend. I had to come up with my own Makefile and figured out generally how to use the API, but I'm still getting a crash when running anything through rbs_parse_signature, erroring while updating the constant pool. Seems like I'm doing something wrong, but I don't see it.

Help wanted!

-- C level backtrace information -------------------------------------------
/Users/headius/.rubies/ruby-4.0.0/bin/ruby(rb_vm_bugreport+0xbc8) [0x10431c140]
/Users/headius/.rubies/ruby-4.0.0/bin/ruby(rb_bug_for_fatal_signal+0x10c) [0x10413f380]
/Users/headius/.rubies/ruby-4.0.0/bin/ruby(sigsegv+0x90) [0x1042713a8]
/usr/lib/system/libsystem_platform.dylib(_sigtramp+0x38) [0x19d9296a4]
/Users/headius/work/rbs/build/librbs.dylib(rbs_constant_pool_insert+0x128) [0x10a67ee14]
/Users/headius/work/rbs/build/librbs.dylib(parse_simple+0xce8) [0x10a6795a8]
/Users/headius/work/rbs/build/librbs.dylib(parse_optional+0x34) [0x10a678814]
/Users/headius/work/rbs/build/librbs.dylib(parse_intersection+0x44) [0x10a67584c]
/Users/headius/work/rbs/build/librbs.dylib(rbs_parse_type+0x58) [0x10a67571c]
/Users/headius/work/rbs/build/librbs.dylib(parse_type_decl+0x138) [0x10a67aeb8]
/Users/headius/work/rbs/build/librbs.dylib(parse_module_members+0x4c4) [0x10a67d780]
/Users/headius/work/rbs/build/librbs.dylib(parse_class_decl+0x38c) [0x10a67bef0]
/Users/headius/work/rbs/build/librbs.dylib(parse_module_members+0xa6c) [0x10a67dd28]
/Users/headius/work/rbs/build/librbs.dylib(parse_module_decl+0x464) [0x10a67b920]
/Users/headius/work/rbs/build/librbs.dylib(rbs_parse_signature+0x998) [0x10a6774b0]
/Users/headius/.gem/ruby/4.0.0/gems/ffi-1.17.3-arm64-darwin/lib/4.0/ffi_c.bundle(ffi_call_SYSV+0x50) [0x10a458050]

[ParadoxV5]

erroring while updating the constant pool.

The thing in #2826?

[headius]

@ParadoxV5 I looked through that and I'm not sure if it fixes anything for me. I suspect it's something I'm not setting up or initializing properly. It fails with either CRuby or JRuby.

It can be tested pretty easily by running make shared to get a dynamic library build, commenting the code shown below, and running the code below that...

diff --git a/lib/rbs.rb b/lib/rbs.rb
index d2b04d36..4c78316f 100644
--- a/lib/rbs.rb
+++ b/lib/rbs.rb
@@ -70,8 +70,8 @@ require "rbs/type_alias_regularity"
 require "rbs/collection"
 
 begin
-  require "rbs_extension"
-rescue LoadError
+#   require "rbs_extension"
+# rescue LoadError
   require "rbs/ffi/parser"
   require "rbs/ffi/location"
 end

$ LD_LIBRARY_PATH="$LD_LIBRARY_PATH:build" ruby -I lib -r rbs -e 'result = RBS::Parser.parse_signature(File.read("sig/method_types.rbs"))'

I was trying to follow the implementation of the C extension as implement my FFI-based _parse_signature the same way, but I must be missing something.

[soutaro]

@headius I want to confirm if the code initializes the global constant pool.

rbs/ext/rbs_extension/main.c

Line 486 in 088d66a

rbs_constant_pool_init(RBS_GLOBAL_CONSTANT_POOL, num_uniquely_interned_strings);

(And as @ParadoxV5 says, I'm working to delete the global constant pool to make the parser thread safe.)

[headius]

@soutaro My reading of the code showed that rbs_parser_new calls that function for me on the constant pool it carries. Is there another constant pool I need to initialize?

Thread safety is an excellent goal! Happy to help if I can, perhaps by testing it on JRuby once I get past this issue?

[headius]

@CufeHaco I appreciate your interest but it seems that perhaps you or Claude misunderstood what I am trying to do here. I don't think voltage monitoring has anything to do with memory seg faults which is what I'm running into in the current code. I gave permission for @soutaro to hide your comments so they don't confuse anyone else finding this PR for the first time.

[headius]

@soutaro I think that might have been it! The following patch appears to fix the crash. I'm unclear why there's a constant pool on both the parser and in a C global, but I'm glad you're working to eliminate it!

diff --git a/include/rbs/util/rbs_constant_pool.h b/include/rbs/util/rbs_constant_pool.h
index 30d569f6..81ff085d 100644
--- a/include/rbs/util/rbs_constant_pool.h
+++ b/include/rbs/util/rbs_constant_pool.h
@@ -82,6 +82,7 @@ typedef struct {
 } rbs_constant_pool_t;
 
 // A global constant pool for storing permenant keywords, such as the names of location children in `parser.c`.
+RBS_EXPORTED_FUNCTION
 extern rbs_constant_pool_t *RBS_GLOBAL_CONSTANT_POOL;
 
 /**
@@ -91,6 +92,7 @@ extern rbs_constant_pool_t *RBS_GLOBAL_CONSTANT_POOL;
  * @param capacity The initial capacity of the pool.
  * @return Whether the initialization succeeded.
  */
+RBS_EXPORTED_FUNCTION
 bool rbs_constant_pool_init(rbs_constant_pool_t *pool, uint32_t capacity);
 
 /**
diff --git a/lib/rbs/ffi/parser.rb b/lib/rbs/ffi/parser.rb
index c8095472..7188af42 100644
--- a/lib/rbs/ffi/parser.rb
+++ b/lib/rbs/ffi/parser.rb
@@ -180,8 +180,12 @@ module RBS
     attach_function :rbs_parser_free, [:pointer], :void
     attach_function :rbs_parse_type, [:pointer, :pointer, :bool, :bool, :bool], :bool
     attach_function :rbs_parse_signature, [:pointer, :pointer], :bool
+    attach_function :rbs_constant_pool_init, [:pointer, :uint32], :bool
+    attach_variable :RBS_GLOBAL_CONSTANT_POOL, :pointer
   end
 
+  Native.rbs_constant_pool_init(Native.RBS_GLOBAL_CONSTANT_POOL, 7)
+
   class Parser
 
     def self.new_parser(str, start_pos, end_pos)

[headius]

Things are moving along and I've added struct definitions for most of the core types.

This may or may not be the best way to use this library from non-C-extension languages but it's a quick way to get it functional. Regardless of how the library's native types and functions are accessed, there's still quite a bit of C extension code to be reproduced in Ruby. The current logic works to parse signatures and get a list of nodes from the RBS parser.

This is running on JRuby using just FFI and the dynamic library version of RBS.

$ LD_LIBRARY_PATH="$LD_LIBRARY_PATH:build" ruby -I lib -r rbs -e 'result = RBS::Parser.parse_signature(File.read("sig/method_types.rbs")); pp result[2]'
[#<RBS::Native::NodeListNode node=#<RBS::Native::Node type=10 location=#<RBS::Native::Location rg=#<RBS::Native::Range start=#<RBS::Native::Position byte_pos=0 char_pos=0 line=1 column=0> end=#<RBS::Native::Position byte_pos=1692 char_pos=1692 line=58 column=3>> children=#<RBS::Native::LocChildren len=6 cap=6 required_p=7 entries=#<FFI::StructLayout::ArrayProxy:0x51b01960>>>> next=nil>]