Don't preinstall jruby-launcher

[headius]

At some point jruby-launcher started to be installed by default in all ruby-build builds. These days, we prefer our shell script launcher and recommend users only install jruby-launcher if needed (e.g. on Darwin or BSD where scripts can't be put into shebangs).

I'd like to have a discussion about removing the install of jruby-launcher from ruby-build.

See related discussion here on the setup-ruby action: ruby/setup-ruby#774

[eregon]

FWIW it was added by #50, following rbenv/rbenv#94 (comment).

[headius]

rbenv/rbenv#94 was 14 years ago and based on JRuby 1.6. I think it's time for a fresh look.

JRuby currently ships a jruby launcher script based on Bourne shell. It is clean and fast and should be enough for most users.

On Linux, the native launcher is not ever necessary, since Linux allows scripts to be in the shebang line.

On non-Linux Unixes like Darwin or BSD, the native launcher might be necessary if the jruby script is to be used directly in shebangs. Ruby libraries installed by RubyGems should honor the JRuby default behavior to use env in shebangs. As long as the scripts are installed by RubyGems or use env, there's no need for the native launcher.

The problem with pre-installing is that in recent years RubyGems will keep track of which platforms a given native extension has been built for, and on JRuby it's using an arch value that includes the current Java release number. If you build on that release, and then switch to a different one, it warns: rubygems/rubygems#3520

I have been trying to work with @deivid-rodriguez to do something about that but we don't have a clear path forward. So to avoid users seeing this warning, I'd like to consider not pre-installing the launcher in ruby-build.

[eregon]

Since you know best about jruby-launcher, could you make a PR?

[headius]

@eregon As I said in my description, I want to have a discussion about removing it before I push any PR.

[eregon]

So you'd rather debate things than fix things? (and ask more of other people's time for little reason)
The discussion can happen on the PR too, if any is necessary.
If someone has to figure out if e.g. old JRuby versions still benefit/need jruby-launcher, or on which platforms to install, I think you would be best placed to do that, so I see no point to discuss this in an issue first.

[headius]

I see no point to discuss this in an issue first

I'm not sure what to put in a PR given the complexities of different platforms, and I'd like to get input from ruby-build maintainers. If you have no input, that's fine.

[mislav]

I don't feel like I can meaningfully weigh in on this topic, since up until now I did not know what jruby-launcher did exactly or why it was necessary. (I have no experience with JRuby or the JVM in general.) As ruby-build maintainer, however, I think it's in the best interest of the project that we occasionally review old practices, such as is proposed here by @headius in a way that sounds to me entirely reasonable, if a little devoid of context. I would personally like to see jruby-launcher out as it caused problems in the past for ruby-build users, so as a result I see it as a maintenance burden and a liability. In general, the fewer steps are involved in an installation of Ruby, the better.

I do sense slight hostility in @eregon's written tone that I do not understand where it comes from. I skimmed the other thread referenced in this issue which provides a lot more context about jruby-launcher, and I would have to more deeply study it to be able to weigh in myself, but so far I fail to see why this would be a topic so sensitive that it would warrant getting worked up about. We are all problem-solvers, and this is just another technical problem that I think should be right up our alley.

@eregon, you've expressed frustration in the past about dealing with Issues noise in this repo. If you feel like this particular thread is not worth your time since it isn't a PR yet, you can tap out (i.e. mute the thread) and I can take it from here. I do appreciate your expertise, since of all the ruby-build maintainers you seem the most knowledgeable on this particular topic as evident by the thread in the other repo, but expertise shared should ideally not come at such a cost to your well-being.

Now, for my own learning, I'll be the first one to propose a thought experiment: let's say we simply strike down the gem install jruby-launcher step from ruby-build and naively push out a new release: would our users notice, or would JRuby apps Just Work™️? What could break, and in which platforms/environments? And finally, how could we minimize that breakage—for example, would we only consider doing this starting with new JRuby versions in ruby-build, but keeping old ones behaving the same? Keep in mind that I'm asking all this with general ruby-build users in mind, and ideally separately from discussions pertaining to ruby/setup-ruby in GitHub Actions and about ruby-builder. Thank you.

[headius]

@mislav Thank you for the feedback and thought experiment! I'll lay out some facts here to try to give you a bit better context.

Launchers in the JRuby dist archives

The normal JRuby dist archives (.tar.gz and .zip) ships a few different ways to launch JRuby:

• bin/jruby.sh which is copied to bin/jruby

  This is a pure Bourne shell launcher that contains all the latest and greatest launcher features and fixes. If you are just launching "jruby" it should handle all use cases. It gets iffy if you want to put it in a $! shebang line (more on that in a moment).

• bin/jruby.exe

  This is actually the same codebase as jruby-launcher, but built for Windows and included in the archive for ease of use (since Windows can't run our shell script). It lags way behind on features but we don't like maintaining it and have not put much effort into updating it. We also only ship a 32-bit version, which causes other minor issues on 64-bit environments.

• bin/ruby symlink to bin/jruby

  This is in our repository, but apparently it has not been shipping in our dist archives. This is why installers may have to add a manual symlink. This should be fixed when we finalize Pregenerate bin/ruby and bin/ruby.bat jruby/jruby#8875, which also adds a trivial ruby.bat to launch the jruby.exe executable (also mentioned in Provide ruby.exe so it's a drop-in replacement for MRI jruby/jruby-launcher#29).

The shebang issue

The reason we originally created jruby-launcher was for the shebang issue, which affects most (all?) non-Linux unixes. As described in rbenv/rbenv#94, if the shell script is put into a shebang on e.g. Darwin or BSD, it will fail to launch the shebanged script as Ruby and instead try to run it as Shell. By installing jruby-launcher, you get a real binary that can be used in shebang lines.

We also force RubyGems to generate all binscripts with "env shebangs" like $!/usr/bin/env jruby which gets around the script-shebang issue, but sometimes gems might ship additional scripts that don't get "envified", and individual users might not know to use that trick.

The new jruby-launcher for unix

Because we know the launcher is sometimes needed on non-Linux unixes, and we know it was always way out of date, we recently modified the build for unix to simply exec /bin/sh with our shell script and pass arguments through. This has worked out great, and we now can be confident that unix users installing jruby-launcher won't lose features from the script.

The RubyGems ext build issue

So why don't we want packagers like ruby-build to pre-install the launcher? Because of an issue in RubyGems that causes it to warn if you install jruby-launcher under one Java version and then try to run JRuby under a different Java version: rubygems/rubygems#3520. There's no "critical" issue here, except that it will warn until the user either uninstalls or reinstalls jruby-launcher while running that other Java version. In the PR I'm working on for Warbler, it actually caused failures due to extraneous subprocess output.

Where do we go from here?

My first instinct is to just get packagers to stop pre-installing jruby-launcher, so we avoid the RubyGems issue and we know everyone is starting out with our preferred shell-based launcher. As long as libraries are installed from gems and scripts use env shebangs, it should all be just fine.

Where things get fuzzy is if it's not "all fine". If users run into issues like rbenv/rbenv#94, they would need to know to install jruby-launcher to fix it. If they don't, they'll either give up or file issues with packagers like you.

If all unixes allowed scripts in shebangs, this would be a non-issue.

If we could get some traction to fix the RubyGems issue, it would be fine for everyone to pre-install jruby-launcher in platform-specific packages.

I'm just sort of stuck in the middle here.

Back to the thought experiment

The most likely thing to break if we just go ahead and stop installing the launcher would be any scripts people have that don't use env shebangs. This will only affect non-Linux, and should not affect binscripts installed by RubyGems, but there it is. They'll get the weird output from rbenv/rbenv#94. I've tasked my "shell guy" with figuring out a way to inform them what needs to be done, but it might not be possible to present a nice error.

The workaround for them would be pretty trivial: install jruby-launcher or use env shebangs.

Again I appreciate your thinking through this with me. I don't want to make a drastic change without some discussion, since I know there's details I'd miss.

[headius]

It's also worth pointing out #2517, wherein I had you remove your deletion of the jruby.sh script. This allowed us to use the new "skinny" unix launcher. It's not directly related to this issue or the RubyGems bug, but I thought I'd point out that this has been an evolving area.

[headius]

Oh, and one last detail...

The reason we ship any launchers at all is because the java command line is horrendous.

Here's what actually runs when you do jruby -e 1 on Java 8:

<java_home>/bin/java -Xss2048k -Djffi.boot.library.path=<jruby_home>/lib/jni -Djava.security.egd=file:/dev/urandom -Xbootclasspath/a:<jruby_home>/lib/jruby.jar -classpath <jruby_home>/lib/jruby-prism.jar: -Djruby.home=<jruby_home> -Djruby.lib=<jruby_home>/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main -e 1

And with newer features on Java 21:

<java_home>/bin/java @/Users/headius/work/jruby94/bin/.jruby.java_opts @<jruby_home>/bin/.jruby.module_opts -Xss2048k -Djffi.boot.library.path=<jruby_home>/lib/jni -Djava.security.egd=file:/dev/urandom -XX:+AutoCreateSharedArchive -XX:SharedArchiveFile=<jruby_home>/lib/jruby-java21.0.5.jsa -Xlog:cds=off -Xlog:cds+dynamic=off --module-path <jruby_home>/lib/jruby.jar -classpath <jruby_home>/lib/jruby-prism.jar: -Djruby.home=/Users/headius/work/jruby94 -Djruby.lib=<jruby_home>/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main -e 1

[eregon]

I'll start with something positive and say #2545 (comment) is a great summary.
If either issue started with that it would likely have gone better.

The reasons I got frustrated with the setup-ruby issue and this one are many:

• Preinstalled jruby-launcher causing warnings on Java 8 ruby/setup-ruby#774 is a poorly-written issue, the link is wrong, the description is incomplete and make maintainers guess lots of details, the check of "I tried to reproduce the issue locally ... and it did not reproduce locally" is wrong, and by the reporter own admission it's most likely not an issue of setup-ruby or ruby-builder. Asking how jruby-launcher gets installed as a discussion instead of issue would have been fine, if it stayed just about that. I think it would also take like ~10min to figure out https://github.com/ruby/ruby-builder/tree/master/.github/workflows uses ruby-build.
• I should not have replied on Preinstalled jruby-launcher causing warnings on Java 8 ruby/setup-ruby#774, or commented it needs a better report or is the wrong place to report, or just close it as invalid.
• At the core of the problem here it's an issue between jruby, jruby-launcher and rubygems. Looking at the Extension gems built for JRuby warn when switching Java version rubygems/rubygems#3520 though it seems very likely it affects every gem with s.extensions = in its gemspec (like psych or any gem which compiles a native library), not just jruby-launcher. So then it's an issue between jruby and rubygems. As such, there should be no need to report this issue anywhere else than jruby/rubygems or to workaround here, the proper fix needs to happen in jruby and/or rubygems.
• Either JRuby supports installing gems on some JDK version and using them on some other JDK version or it doesn't. If it doesn't, then Extension gems built for JRuby warn when switching Java version rubygems/rubygems#3520 and linked are a non-issue. If it does, then it seems clear JRuby cannot have the Java version in any RbConfig::CONFIG value, otherwise it's bound to cause such issues. It's also a recurrent problem that JRuby pretends its "arch" or "platform" is "java" but it's simply not true for gems building a native library (e.g. psych) or executable (e.g. jruby-launcher). Maybe RubyGems needs or should be changed too, I recall an issue where RubyGems/Bundler was using the darwin version (probably because it's part of RUBY_PLATFORM on CRuby) and clearly that gets more in the way than it helps, and that got fixed.
• I get that finding the right fix in JRuby/RubyGems for this might be non-trivial and the desire to workaround for jruby-launcher until then. But then, why not make jruby-launcher an empty gem with no s.extensions =? That should solve the problem nicely in one place, and if there is an issue one could always revert that change. If it works well then of course it'd make sense to remove the unnecessary install in Ruby installers.
• The well-known robust solution here is to have a native launcher. This is what CRuby & TruffleRuby do. FWIW TruffleRuby also tried various things for the launcher including a tiny executable on macOS just execve'ing to the shell script, but it became clear only a true native launcher works well, and it's also necessary to get access to the main() argc/argv to implement Process.setproctitle, and for a user script with a #!/.../bin/jruby shebang to work, etc. There have been many JRuby launchers and I recall the pain first hand to keep them in sync and it's good to see these days it's only 2/3 of them: jruby.sh, jruby.exe and the native executable execve'ing to jruby.sh. Since on Windows the only option is jruby.exe (= jruby-launcher) another proper fix seems to make jruby-launcher the only one launcher, and remove jruby.sh. In terms of shipping JRuby probably it would mean that the initial bin/jruby would need to compile or install the native launcher somehow, or JRuby should ship separate archives per platform (seems cleaner). JRuby already ships libffi/jnr native libraries precompiled for many os/arch, so maybe the launcher could use a similar technique. It could also be integrated as a generic post-install hook (like TruffleRuby has).
• This is the first comment that pissed me off: Preinstalled jruby-launcher causing warnings on Java 8 ruby/setup-ruby#774 (comment) Neither of these things are true. I understood that as you're saying I'm saying BS. Well, I recall well enough to know I'm not, and in fact just further in that comment it's somewhat clear it used to be valid concerns (any inability to handle command line arguments has been fixed). I even included 2 IIRC to clarify I'm not criticizing but just remembering those were the concerns with the jruby.sh launcher and the reasons to use jruby-launcher, but maybe that didn't come across as intended.

[headius]

@mislav There are some recent developments...

• The RubyGems issue appears to have some traction and we are considering options like removing the Java version from the arch value that RubyGems uses to identify built extensions.
• I'm working through other tweaks in JRuby (bin/jruby and bin/ruby to invoke bin/jruby.sh) and jruby-launcher (install over bin/jruby and bin/ruby both) that will also weigh into this issue.

I'll be moving forward with this stuff at the beginning of the week.

[headius]

We have moved forward to normalize our RbConfig::CONFIG["arch"] value to be similar to CRuby's, with a hyphenated version at the end that RubyGems and Bundler can ignore for extension purposes. I believe if that gets fixed then the warnings reported here will go away. I do not know when the version-ignoring patch will ship, though.

We'll keep this issue on hold for the moment until it's clear which direction to go moving forward. Old releases can't be changed anyway and hopefully we'll have more clarity before we push the next versions of JRuby 9.4 and 10.