C extensions not yet supported by TruffleRuby

[eregon]

This issue summarizes C extensions not yet supported by TruffleRuby.
Issues for failing C extensions should be filed separately as a new issue.

Note that we intend to support most C extensions, but we need to prioritize.

Not yet supported

Extension	Description	Possible workaround until supported
mini_racer #1827	Evaluate JS code

Works with workarounds

Extension	Description	Workaround
nokogiri #62	HTML/XML parser	Installs, but need to use system libraries for now. Use bundle config build.nokogiri --use-system-libraries / gem install nokogiri -- --use-system-libraries. See their documentation for details.

Works out of the box

Extension	Description	Comment
bootsnap	Boot Rails apps faster	Works as of bootsnap 1.4.0. For previous versions, it can be commented it out in the Gemfile.
msgpack	binary serialization	Works and all specs pass, as of 1.0.0-rc12.
mysql2 #1388	MySQL client	Installs and works.
pg #1421	PostgreSQL client	Installs and works as of 1.0.0-rc12.
hiredis #1546 #1974	Redis client	The pure-Ruby redis client is an alternative, hiredis seems to also have a pure-Ruby backend as fallback (not used if loading the C ext works)
rugged #1088 #1642	bindings to libgit2
rbtree #1343	Red-Black Tree, also used for SortedSet optionally
oj #1566	Fast JSON parser	The json stdlib. oj >= 2.7.1 installs, but does not run yet.

And many more: openssl, psych, zlib, syslog, etc, puma, sqlite3, unf, ...

[skunkworker]

Although bootsnap would be nice it's not critical in transitioning production apps from MRI to TruffleRuby. For my case having database gems like pg and mysql2 would be the most beneficial by far.

[byroot]

bootsnap can easily be made compatible by disabling a couple optimization like it's already done for JRuby. The real blocker is that it depends on msgpack. But I'm tracking this issue and whenever msgpack is supported I can fix bootsnap.

[eregon]

@skunkworker Indeed, the table is in no particular order but pg/mysql2/nokogiri are the most important right now out of this list.
@byroot That's great to hear. I did a bit of work to get basics of msgpack running some time ago, I'll try to progress on that.

[paneq]

@eregon could Truffle use JDBC adapters as a workaround for pg/mysql ? perhaps related: pgjdbc/pgjdbc#1189

[nirvdrum]

@paneq TruffleRuby on the GraalVM should be able to support JDBC, as we have a rich Java interop API. Having said that, I don't think anyone on our team has spent any considerable amount of time looking into that.

Assuming it does work, it'd be a fine option for making JDBC calls from Ruby. Unfortunately, the use case we're really after is Rails. Unlike many other languages, Ruby does not have a DB abstraction layer as part of its standard library. Consequently, ActiveRecord ties very closely to the native mysql & pg drivers. This is something that has been a very big problem for the JRuby team. They provide an implementation of ActiveRecord based on JDBC, but since ActiveRecord's internals are subject to change, this breaks with every minor Rails release. And since ActiveRecord doesn't have a real plugin API, any ActiveRecord plugin you see really is a monkeypatch to the internals and these often don't work with the JRuby implementation.

A secondary concern is the binary available via all the Ruby version installers does not include Java interop support at the moment. So you could not use JDBC in that case either.

This is all a roundabout way of saying that working with JDBC for the use cases most Rubyists would want a DB for would entail quite a bit of effort. And we'd rather allocate that effort to supporting the native extensions, which would lead to a much smoother usage path.

[zw963]

@nirvdrum , maybe support sequel is a good start, it use plugin system (like roda, same author).

[eregon]

@zw963 I think we run Sequel itself, but Sequel depends on pg, mysql, etc for actually talking to databases, so we need to support these C extensions anyway.
There are some pure-Ruby database drivers like postgres-pr, but those have a different API, won't work with ActiveRecord, and so not really a good workaround.

[ylluminate]

Sequel for Roda would be important via the pg & mysql support, yes. I think these are pretty good starts. With the recent deplorable announcement by GitHub to move away from Ruby & Rails since the MS acquisition, I think some heat is on. I've heard of some serious debates at workplaces coming up again against Ruby and Truffle is becoming more important than ever if it can reach a culminating point of running Rails out of the box and with minimal extra effort (such as special conditions for nokogiri). Humans are very social and shifty creatures that rely on mass acceptance and good feelings often rather than fact and marketing, hate to say it, is key.

[paneq]

deplorable announcement by GitHub to move away from Ruby & Rails since the MS acquisition, I think some heat is on

@ylluminate Could you provide a link?

[ylluminate]

@paneq https://www.theregister.co.uk/2018/08/16/github_rails_microsoft/

[zw963]

#1388, pg and mysql is worked now since 1.0.0-rc6

[ntl]

@zw963 it appears as though pg 1.0.0 can be compiled successfully, but versions 1.1.0, 1.1.1, and 1.1.2 cannot. Is this an error you all have seen? If not, I can paste the compile error here or file a new issue.

[sbellware]

Update: This comment is replaced by issue #1421

[eregon]

@sbellware Could you file this as a separate issue?
This issue is for keeping track of general progress, not individual compilation/running issues.

[sbellware]

@eregon Done. The new issue is #1421.

[eregon]

@byroot

bootsnap can easily be made compatible by disabling a couple optimization like it's already done for JRuby. The real blocker is that it depends on msgpack. But I'm tracking this issue and whenever msgpack is supported I can fix bootsnap.

In TruffleRuby 1.0.0-rc6, gem install msgpack succeeds, but require "msgpack" fails.
Is msgpack necessary to run bootsnap or is it more like an optional dependency?
We'll probably get msgpack working in the future, but it would be useful to know if bootsnap needs it or it could be made optional easily for TruffleRuby.

[byroot]

Is msgpack necessary to run bootsnap or is it more like an optional dependency?

Kind of both. In short bootsnap need to be able to serialize a big hash efficiently, and messagepack shines here.

In theory we could fallback to use JSON or whatever, but it would complexify the codebase, and slow boot down significantly.

Alternatively we can probably turn bootsnap off entirely for truffleruby.

[eregon]

@byroot Thanks for the reply. Just an idea, would using Marshal (which is part of the core library) instead work well? Or is it slower than msgpack to serialize that big hash?

[byroot]

Or is it slower than msgpack to serialize that big hash?

It's between 4 and 5 times slower. It's also much less compact.

[eregon]

@byroot Thanks for the info, we'll look at getting msgpack to work then.

[JongleurX]

I've been watching this project eagerly for a while as it matures, hoping that it will let users of Redmine (a popular project-management tool) get better performance across the board. I found some references to Redmine in the TruffleRuby GitHub repo, so I'm hoping the talented folks creating maintaining this project will have a vested interest in supporting Redmine.

Please let me know if any information is missing and I'll be glad to post more detail. I posted separate bug reports per @eregon's request, see below.

• childprocess 0.9.0 gem opt fail: UnboundMethod#bind, rb_data_typed_object_wrap #1430
• jquery-rails 3.1.5 gem opt fail: @polyglot_invoke #1431
• railties 4.2.10 gem opt fail: UnboundMethod#bind (builtin), rb_data_typed_object_wrap, @polyglot_invoke, INVOKERootNode #1432
• "RegexpError: invalid multibyte escape" error on "rake generate_secret_token" and "rake db:migrate" Rails commands #1433

[eregon]

@JongleurX Could you file that report as a separate issue? Otherwise, it is very hard to track.
I tried to clarify at the top of this issue that this issue is only a summary, and not where bugs should be reported.

[JongleurX]

Sure, I've reported separate issues for each gem that generated errors, and another separate one for the two rake failures. See the links above (I edited my original comment).

[eregon]

@JongleurX A single new issue is fine actually, since the last two errors seem related, and the opt fail should be fixed in the upcoming release (see #1422).

[dbackeus]

Perhaps ethon should be added to this list?

Related issue: #1524

Typhoeus, one of the best performing http adapters, depends on this.

[eregon]

@dbackeus ethon uses FFI, it's not a C extension. I commented a bit more about it on its issue #1524.

[eregon]

@byroot msgpack now passes all specs on TruffleRuby, and it will be in the next release.

[byroot]

Awesome. Then I'll try to make bootsnap compatible. I suppose I can use truffle-heador similar ?

[eregon]

@byroot Currently, you can either compile from source or wait for the next release (should be beginning of February, probably easier, I'll ping here when it's there). We also plan to have nightly builds in the future (#1483).

[byroot]

👌

I still can start now by adding minimal support for jruby & truffle, i.e. make bootsnap a noop on non MRI.

Then we can attempt to add load path cache support later on, but even if we don't at least people won't have to comment out bootsnap to try truffle.

[eregon]

@byroot That seems a good starting point indeed. Please ping me on the bootsnap PR.

[byroot]

PR opened: Shopify/bootsnap#221

[pojntfx]

I'd like to add the popular grpc gem to this list.

[eregon]

@pojntfx Could you file a new issue with the error you see?

[eregon]

I'll close this issue because all C extensions listed in the description work now 🎉, except mini_racer which is tracked in #1827.