What’s the Status, Kenneth?

I’m pretty sure it started like this.

“Hello, Evan dear.” — Evan’s Grandma Betty

“Grandma Betty!” — Evan

“You know, I was thinking about your little Rubinius thingy while I was gardening today. How’s it doing?” — Evan’s Grandma Betty

“Oh, Grandma Betty, that’s so sweet of you to think of me and it. It’s coming along still, of course.” — Evan

“That’s good to hear. I was specifically wondering how the 1.9 support is looking.” — Evan’s Grandma Betty

“Grandma Betty! You do always love the Latest and Greatest™, don’t you? Well, let’s see. I’d say that 1.9 support comes down to 7 main areas: Encoding, String, Regex, Symbol, IO, Argument Processing, Windows Support. Each has varying degrees of progress.” — Evan

“OK. I wish that there was some way for me to keep more up to date on these happenings without needing to calling you. When I call you, I want to hear about that wife and baby of yours.” — Evan’s Grandma Betty

“You got it, Grandma Betty. I’ll get my Top Nerds™ on the case.” — Evan

[sometime later]

“Alright, Grandma Betty. We made the Rubinius Status Board™ especially just for you. It’s status.rubini.us” — Evan

That conversation may or may not be completely fictionalized. There is one thing that is most emphatically true: Rubinius has a Status Board™ now. If you are ever curious how progress is going on things, go check out status.rubini.us. Tell your friends.

XOXO RBX.

Rubinius All Around the World

As a result of the Rubinius Rewards, people from all over the world have requested Rubinius stickers and t-shirts. That just warms our little hearts and humbles us everyday. Thank you so much. It means the world to us. Keep using Rubinius with passion and excitement. Keep spreading the good word to your friends and anyone who’ll listen and care. And keep telling us how Rubinius is working for you. However good or bad, we want to know.

Here’s a map of all the places where we’ve sent stickers and/or t-shirts. You can also view it on a larger map. Wow.

Again, thank you from the bottom of our hearts.

Stay safe, kids.

XOXO RBX

Rubinius Summit 062011 in Pictures

Once a month Evan Phoenix, Brian Ford and I (Shane Beccker) get together for a few days in-person face-time. Usually we meet up in San Francisco at the Engine Yard HQ and spend time with our dear and lovely boss, Dr Nic. We like to call these little get togethers Rubinius Summits. For the June 2011 Rubinius Summit, Brian came down to LA where both Evan and I live. We spent the week working together at The Farmhouse on moving the ball forward toward the 2.0 release. Here’s a few pictures of some of the things that went down.

I Used Rubinius at RailsConf 2011 stickers

"I Used Rubinius at RailsConf 2011" stickers at The Farmhouse in Hollywood, CA

These stickers arrived and are ready to be shipped out. If you DMed your address to us on Twitter, we’ve got you. If you never contacted us, but used Rubinius while at RailsConf 2011 in Baltimore, email us at community@rubini.us to get your sticker.

T-Shirts!

Nearly 600 more @Rubinius shirts just arrived at the @farmhouse in Hollywood, CA

This is what nearly 600 shirts looks like. We’ve got a huge back log of people all over the world who responded to our offer of free Rubinius t-shirts. Those will all get shipped out over the next week or so.

Progress on 1.9 Support

Serious @rubinius big brain nerdery happening between @brixen and @evanphx at the @farmhouse in Hollywood, CA

Brian and Evan, like a couple of mad scientists, sketched out how to handle 1.9 block arguments for Rubinus 2.0.

A sketch of Ruby 1.9 block args in Rubinus 2.0 by @evanphx and @brixen at the @farmhouse in Hollywood, CA A sketch of Ruby 1.9 block args in Rubinus 2.0 by @evanphx and @brixen at the @farmhouse in Hollywood, CA

Here are the sketches of 1.9 block arguments for Rubinus 2.0.

One More Thing

We came up with an idea this week for how to better communicate outwardly and be more transparent. We built it up real quick and are launching it …really soon. Here’s a little teaser.

Rubinius teaser 1 Rubinius teaser 2

XOXO RBX

Inside the Rubinius 2.0 Preview Release

Houston, we have a problem. Complex projects are difficult to transition forward and Rubinius is a complex project. Rubinius is at the point of a major transition and we need your help to move forward.

Just over a year ago, Rubinius released version 1.0. Since then, dozens of contributors have added over 2,200 commits to the master branch and we have released seven new versions. At the same time, we have added nearly 450 commits on a development branch to remove the global interpreter lock (GIL) and begin adding Ruby 1.9 and Windows support.

Rubinius 2.0 Developer Preview

Today we are releasing Rubinius 2.0 Developer Preview. The primary focus of this release is true Ruby multi-threaded concurrency. Additionally, it contains the beginnings of Ruby 1.9 and Microsoft Windows support.

Rubinius 2.0 Developer Preview

We are seeking developers interested in running their Ruby and Rails projects on Rubinius to help us iron out issues as we work toward the final 2.0 release. Let’s look at the details of the 2.0 developer preview.

One of the central features of Rubinius 2.0 is a fundamental change in the threading architecture. In Rubinius 2.0, Ruby threads will run with true concurrency. This means that if multi-core or multi-CPU hardware is available, Ruby code on different threads will actually run in parallel.

Another central feature of Rubinius 2.0 is support for both Ruby 1.8.7 and Ruby 1.9.2 syntax. The Rubinius 2.0 support for Ruby 1.8.7 should be 100% backward compatible with the support in the Rubinius master branch. The full Ruby concurrency is enabled regardless of what language syntax mode is active.

Repository Migration

We are migrating the main Rubinius repository to the Rubinius organization.

The RVM project has already been updated to work with the new repository. So rvm install rbx will use the new repository. If you have an existing clone of Rubinius, update and run rake github to point your existing repository to the new one.

You can clone directly from the new repository.

Installation

For testing Rubinius with your application, installing with RVM is probably the easiest. To install the 2.0 developer preview, ensure that RVM is updated:

rvm get head
rvm reload

Then, install the Rubinius 2.0.0pre branch:

rvm install rbx-2.0.0pre

If you are interested in helping develop Rubinius, we suggest you clone the repository directly and build. Rubinius runs fine from the source directory so you don’t need to install it. For more details about building from a clone, see Getting Started.

Configuration

The goal of Rubinius 2.0 is to fully support both Ruby 1.8.7 and 1.9.2 in a single executable. When running Rubinius, you can select which version to use. There are configuration options to change which versions are supported and which version is the default.

The first configure option is --enable-version=X[,Y]. This option sets which language versions will be supported. An example is:

./configure --enable-version=1.8,1.9

The language versions can be specified as 18 or 1.8. Multiple versions are separated by a comma and no space.

The second configure option is --default-version=X. This option sets which language version is the default when running Rubinius. An example is:

./configure --default-version=1.9

These options are intended to be used together. You must enable a version to select it as the default, so the options must come in the following order:

./configure --enable-version=X,Y --default-version=Y

Once you have enabled support for a language version, to select that mode when running Rubinius, use the -X18 or -X19 command line switches. For example:

rbx -X19 -v

This command will run Rubinius with the 1.9 language mode enabled.

You may also use the RBXOPT environment variable to select the language mode. Assuming that 1.8 is the default mode, the following command would run Rubinius with the 1.9 language mode enabled:

RBXOPT=-X19 rbx -v

Right now, only the 1.8 language mode is enabled in the 2.0.0pre branch. However, we will be enabling the 1.9 mode in the very near future. The default mode will continue to be 1.8 unless you configure Rubinius to run in 1.9 mode.

Building on Windows

On Windows, Rubinius uses the MinGW-w64 compiler to build a native application. There remains a lot of work to do for Windows support but the Rubinius VM is currently compiling on Windows 7.

If you are a bleeding-edge Windows developer interested in diving into Rubinius, here’s how to get started:

  1. Install MRI 1.9 using RubyInstaller.
  2. Install MSysGit.
  3. Install the 32bit or 64bit MinGW-w64 toolchain.
  4. Clone the Rubinius repository.
  5. In CMD, run ruby configure.
  6. Run rake.

Moving Forward

In the near future, we will release another version of Rubinius from the current master branch. We hope to merge 2.0.0pre into master as soon as possible and continue working toward the 2.0 final release. We’ll make that decision based on your reports about how the 2.0.0pre branch is working with existing 1.8.7 applications.

We greatly appreciate your help polishing up the Rubinius 2.0 release! Follow us on Twitter for updates and stop by #rubinius on freenode.net to chat about anything on your mind.

Rubinius Support on EngineYard AppCloud

Rubinius 1.2.3 is currently in beta on Engine Yard AppCloud, and we encourage users to give it a try. Engine Yard will support Rubinius 2.0.0 after it’s officially released.

To learn more about Rubinius, join Engine Yard for a Platform Options Webinar in July.

Announcing Rubinius Rewards

tl;dr

Email community@rubini.us to get stickers and t-shirts. Include your mailing address.

We Made T-Shirts and Stickers

We heard you like stickers and t-shirts, so we made some. They debuted at RailsConf 2011 in Baltimore, MD. Then we gave almost all of them away in just a few hours. They went like hotcakes. It turns out that there was a lot of pent up demand for Rubinius swag.

Not everyone could be at RailsConf to get the goods, of course. Even some people who were there didn’t get their shirts/stickers. If you didn’t get one and want one, we’re very sorry, but don’t you worry. We’ve got you covered.

General Availability Stickers

A box of Rubinius stickers

We’ve got a box of stickers in three designs: square, bumper and die-cut.

Email us and tell us which one you want.

General Availability T-Shirt

We’re printing 500 more grey Rubinius t-shirts in the two different designs and in a a handful of sizes (women’s small and medium, unisex small – xx-large).

Email us and tell us which design / size you want: square or horizontal in women’s small, women’s medium, unisex small, unisex medium, unisex large, unisex x-large or unisex xx-large.

Rubinius "Use Ruby™" T-Shirts at the Farmhouse in Hollywood, CA

Photo by Tj Nelson Jr @tjnelsonjr

First Commit Sticker

Going forward, we want to reward everyone who makes a contribution to Rubinius. As a very small token of our gratitude, we’re mailing a Rubinius sticker (and a handwritten thank you note from one of us) to everyone after their first commit. So, if you’ve ever thought about dipping your toe into Rubinius (or diving headlong into the deep end), now’s the best time ever. Help us make Rubinius better (in big and small ways) and we’ll send you stuff.

Tenth Commit Shirt

We want you to stick around and keep helping Rubinius to get better and better. If you make 10 commits to Rubinius, we’ll send you a special shirt only available to committers. That design is still a secret for now, but it’s just for 10+ committers.

Please, don’t try to game the system by intentionally breaking stuff up into smaller commits just to bump up your count. Let’s keep it honest.

Quarterly Committer Merit Badge Stickers

In addition to getting a generally available sticker after your first commit, at the end of each calendar quarter (every three months) we’re sending a sticker to everyone who committed to Rubinius during that quarter.

E.g. after July 1, 2011, we’ll print and ship a sticker to everyone who committed between April 1 and June 30. Each quarter’s sticker has the year / quarter in the corner. Keep committing every quarter and you’ll keep collecting the merit badge stickers.

One More Thing — I’m Committed* Sticker

Rubinius is obviously older than the new Rubinius Rewards program. To backfill for all the contributions people have made over the years up until, we have a super duper limited edition never to be made again sticker… the asterisk.

Rubinius stickers on my laptop

Get in Touch

If you’re a past committer, email us your mailing address get your special merit sticker. If you’re a new committer, we’ll try to take note and reach out to you. If you don’t hear from us, don’t be afraid to contact us with your mailing address.

Up Next…

Rubinius International Outposts.

Adam Prescott on Scopes in Rubinius

Adam Prescott wrote a lengthy article about variables, closures and scope in Ruby, mentioning Rubinius at the end:

One of the really cool things about the Rubinius implementation of Ruby is that it exposes, by requirement, a level of internals which you can’t find in MRI, including some internals with scopes. Because these internals are exposed in Ruby itself, you can play around with scopes as objects, using VariableScope, including getting access to the available local variables within that scope, with VariableScope.current.locals.

How are you using Rubinius? What are you doing with it that you couldn’t do before? What is it missing for you to really dive in? Let us know. We’re listening.

— Use Ruby™

Rubinius T-Shirts and Stickers

Allow Myself to Introduce Myself

Hi. I’m Shane (@veganstraightedge / iamshane.com). I’ll be helping out around here now. I was recently hired by Engine Yard to be their Open Source Cheerleader. (Yes, there will be costumes at some point.) My primary focus is Rubinius! Rubinius! Rubinius!

I have lots of big plans for helping get Rubinius used by more people in more places and how to have a better conversation with people who are using it. For now though, I tell you about just the first thing that I’ve done.

T-Shirts

The First Ever Rubinius T-Shirt from RailsConf 2007

The First Ever Rubinius T-Shirt from RailsConf 2007

A long time ago (2007) in a city far, far away (Seattle, WA), Evan and I made the first batch of Rubinius t-shirts. There were about two dozen made and given to the handful of contributors at the time. This was at RailsConf in Portland, OR. Since then there hasn’t been anything made with the Rubinius logo on it.

Now is the time to rectify that. We’ve made two different t-shirts. Same messaging, slightly different design orientation. Both are the same colors: white on asphalt. They are available in an assortment of sizes from women’s small to unisex xx-large.

Use Ruby™ Rubinius T-Shirts from RailsConf 2011

Use Ruby™ Square Ruby T-Shirt from RailsConf 2011

Use Ruby™ Horizontal Ruby T-Shirt from RailsConf 2011

If you’re going to RailsConf 2011 in Baltimore, MD, stop by the Engine Yard booth to pick up a free Rubinius t-shirt. After RailsConf we’ll have ways to get one from us directly.

Stickers

To go along with the t-shirts, we’re making a whole grip of Rubinius stickers. Most of which will also be available for free from the Engine Yard booth at RailsConf in Baltimore. Again, after RailsConf you’ll be able to get them directly from us. More on that later. Here’s what we have in store.

Wide Screen (7in x 3.75in)

Rubinius : Use Ruby bumper sticker

Rubinius : Use Ruby square sticker

Rubinius : r logo diecut sticker

I’m Committed Merit Sticker

This one you have to earn. At the end of each quarter, we’ll send out a little one inch square sticker with a year/quarter combination in its corner to everyone who committed something to Rubinius during that calendar quarter. For all of the incredible folks who’ve contributed something (however big or small) already, you’ll get a special sticker. Instead of a date, there’s an asterisk. Only those of you that’ve committed so far will get that one.

Rubinius : I'm committed past committers sticker

Come find us to get free Rubinius shirts and stickers. And stay tuned to the Rubinius blog and to the Twitter account : @rubinius. We have lots more in store.

— Use Ruby™

Running Ruby With No Ruby

Humans have come a long way since our cave-dwelling days. No, that’s not a metaphor for primitive software. I mean literally since we lived in caves. One of the big inventions is the lock. There are birds that bury food and will move it later if they notice they were watched burying it. But they have no access control. Any bird may come along and dig up the food.

Humans, though, are smarter than the average bird.

We have numerous systems that implement the analog of a lock, namely, some sort of access control. For every one of these systems, we have other systems that attempt to circumvent or defeat the access control. Two sides of the ubiquitous coin of life.

In software, attempts to implement access control typically involve distinguishing between source code and some form of executable code. Direct access to the source code is not permitted. Further, the format of the executable code resists attempts to derive the source code. There are several reasons for this:

  1. Licenses: If a vendor licenses individual copies of the software, they want to prevent unrestricted copying of the software without paying for a license.
  2. Intellectual Property: A vendor may have trade secrets or other proprietary information that is disclosed by the source code.
  3. Security: Full access to the source code may reveal vulnerabilities in the software or related systems. This is sometimes called Security Through Obscurity and is generally disparaged, but things are not so black and white and there may be valid security concerns in the source code.

The process that separates the source code from the executable program is typically a compilation step. However, Ruby code is not typically associated with any sort of compilation. That’s one of the great things about Ruby, right? There is no edit-compile-link-load cycle to wait on. Just edit and run. But if there is no compilation step, how do we separate the source code from the executable code?

You may recall from my last post that Rubinius does compile Ruby code to a bytecode format that the virtual machine executes. I also promised to explain how you could run the bytecode directly.

But first, let me very clearly state that there are a number of caveats. In fact, I’ve included a whole section on them below. Please read them. We will assume that you have and that you understand them. If you have any questions, please ask.

Application Distribution Scenario

Let’s review what we would like to accomplish. We’ll assume affable Abe is a developer writing an application for customer Cain.

  1. Abe writes some Ruby code.
  2. Abe compiles the code to a set of files.
  3. Abe packages the files up into an application with no Ruby source.
  4. Abe sends the application to Cain.
  5. Cain installs the application.
  6. Cain runs the application.

In this scenario, I’m assuming a very vague definition of application. In other words, the process below will fit in with a broad spectrum of bundling and distribution schemes.

Application Layout

Let’s assume that you have the following application layout. This mirrors what you would expect to see in a gem. You could also consider this as a subtree in your larger project.

widget
|- lib
|- widget.rb
\- widget
   |- red.rb
   |- blue.rb
   \- green.rb
1 # widget.rb
2 require 'widget/red'
3 require 'widget/blue'
4 require 'widget/green'
1 # widget/red.rb
2 puts "I am red"
1 # widget/blue.rb
2 puts "I am blue"
1 # widget/green.rb
2 puts "I am green"

Compiling Ruby Files

The Rubinius bytecode compiler is accessible through a command-line script. See rbx compile -h for all options. We will only need one simple option in our case to easily create a separate tree containing one compiled file for every Ruby source file in our source tree.

rbx compile -s '^widget:widget-compiled' widget/

Let’s dissect this command. The -s option defines a transformation to apply to every filename. The transformation has the form <source>:<destination> where <source> can be a Regexp. In our case, we would like to change any path starting with widget to start with widget-compiled. This way, we create a separate tree of our compiled files. The final option is the widget/ directory. The rbx compile command will happily compile a single file or a directory of files. Note that if we did not pass the -s option, rbx compile would have created the compiled files alongside the source files.

If we now look at widget-compiled, we should see the following:

widget-compiled
|- lib
|- widget.rbc
\- widget
   |- red.rbc
   |- blue.rbc
   \- green.rbc

Loading Pre-compiled Files

Now that we have a separate tree of only compiled files, how do we load them? Well, first, let’s load our source files so we know what to expect. Note that the technique used in this post should not substitute for a robust test suite.

$ rbx -Iwidget/lib -e "require 'widget/lib/widget'"
I am red
I am blue
I am green

Ok, that is what I would expect. Now, to load the compiled files:

$ rbx -Iwidget-compiled/lib -e "Rubinius::CodeLoader.require_compiled 'widget/lib/widget'"
I am red
I am blue
I am green

The crowed erupts with applause and hooting.

Golly gee, you guys… Blush

Let’s review. Our goal is to take a tree of Ruby source files and create a tree of compiled files that can be sent to a customer and loaded to perform exactly as the Ruby source would if loaded directly. The most direct and simple way to accomplish this is to use the Rubinius compiler command-line script to compile the tree of Ruby source files to a separate tree. Then, load the root of that tree with Rubinius::CodeLoader.require_compiled "root".

Caveats

I will admit, I have resisted fiercely against encouraging or even permitting Rubinius users from using what I showed above in their code. Not because I am an ogre who is trying to steal your fun, but because there are serious issues with allowing this. So, please read the following carefully.

  1. We, Rubinius, absolutely reserve the right to change any part of the underlying compiled file mechanism. Since we are publishing the Rubinius::CodeLoader.require_compiled(name) method, we will respect that contract. What it says is, given a name, we will load a representation of that name. DO NOT assume that "some_file" is actually referencing "some_file.rbc". We may change the way compiled files are stored and may change the format of the compiled output.
  2. We have created this facility to meet a need we had in Rubinius. Since our compiler is written in Ruby, we have to run Ruby to run the compiler. But since we need to compile Ruby to run it, we need to compile the compiler. But since… To handle this, we build the compiler using a bootstrapping version of Ruby. Then we load the pre-compiled compiler files as shown above. The approach is quite general, as demonstrated. However, a better approach may be appropriate for a particular application. In that case, talk to us about how you think it should work and we can point you in a direction to try implementing what you need.
  3. We assume no responsibility for any security breaches resulting from your incorrect assumption that .rbc files provides any meaningful security mechanism. I cannot stress this enough. The compiled file mechanism is a cache. It is a way to avoid recompiling Ruby source code that has not changed. The compiled format is simple. We reserve the right to provide disassemblers for our compiled code. We are happy to assist you with direction for implementing a more secure system for your needs.
  4. There is no mechanism that is completely safe from cracking when it comes to software access control. Witness how often Microsoft’s products have their security features defeated. Also witness how often attempts at DRM are circumvented. The most secure system I have seen uses a special compiler and a hardware dongle. The compiler takes critical parts of the application logic and breaks them up so that part of the computation is performed on the dongle. This is significantly harder to defeat than binary editing an executable to turn a license key check into a no-op. The folks you most want to keep from accessing your information are the ones most capable of doing so. Security and access control are very hard problems.

Conclusion

Rubinius compiles Ruby code to bytecode before running it. It is possible to save the bytecode representation and reload it later. Using this mechanism, it is possible to avoid providing the Ruby source code and run an application directly from the compiled bytecode. The mechanism we use to do this was created to solve our problem of bootstrapping the Rubinius bytecode compiler, which is written in Ruby. The mechanism is not intended to be used for security.

It is possible to extend the Rubinius code loading mechanism to support custom formats for on-disk compiled bytecode and to load those formats. This can be done entirely in Ruby code. If this interests you, please talk with us about it.

Making Rubinius .rbc Files Disappear

Rubinius is rather unusual as a Ruby implementation. It both compiles Ruby source code to bytecode and saves the compiled code to a cache so it does not need to recompile unless the source code changes. This can be great for utilities that are run often from the command line (including IRB). Rubinius merely reloads the cached file and runs the bytecode directly rather than needing to parse and compile the file. Sounds like a real win!

Unfortunately, it is not that simple. We need some place to store that cache and this is where the thorns on that pretty rose start poking us in the thumbs. The solution we have been using since forever is to store the cached file alongside the source file in the same directory, like so:

$ echo 'puts "hello!"' > hello.rb
$ ls hello.*
hello.rb
$ rbx hello.rb
hello!
$ ls hello.*
hello.rb	hello.rbc

That doesn’t look too crazy, but it can get more complicated:

$ mv hello.rb hello
$ rbx hello
$ ls hello.*
hello.compiled.rbc	hello.rbc

Whoa, what is hello.compiled.rbc? Since hello did not have an extension, we add that longer compiled.rbc to make it clear which file the cache is for. Also, note that we have that hello.rbc hanging about even though the original hello.rb is gone.

To summarize the issues with our caching scheme:

  1. It requires an additional file for every Ruby source file.
  2. It requires some potentially complicated naming scheme to associate the cache file with the source and not clash with other names.
  3. Removing or renaming the Ruby source file leaves the cache file behind.

Again, the advantage of the cache file is that you do not have to wait for Rubinius to recompile the file if you have not changed the source. Let’s see if we can get all the advantages with none of the disadvantages. That old saying comes to mind, Having your cake and eating it, too, so we may not be successful, but it is worth a shot.

First, let’s take a step back. This issue is not unique to Rubinius. Python has .pyc and .pyo files. Java has .class files. C/C++ has .o files. Lots of things need a place to store a compiled or cached representation of some data. Every SCM worth mention has some mechanism to ignore the files you don’t want to track. The same is generally true of editors. So in some sense, this is a solved problem. However, we have always received complaints about the .rbc files, so we thought we would try to make other, hopefully better, solutions available.

Solution 1: No Cache

One simple solution is just to never ever ever create the compiled cache files in any form anywhere. We have an option for that:

$ ls hello.*
hello.rb
$ rbx -Xcompiler.no_rbc hello.rb
hello!
$ ls hello.*
hello.rb

Win! Not one lousy .rbc file in sight. Although, that’s quite the option to type. Never fear, we have a solution to that below.

Here is our scorecard for solution 1:

Use Case: Use when you never want any compiler cache files created. For example, on a server where startup time is not really a concern.

Pros: No .rbc files at all.

Cons: Startup will be slightly slower depending on what Ruby code you are running. It will be more noticeable in a Rails application, for example. However, the Rubinius bytecode compiler is several times faster than it was a couple years ago so it may not be an issue for you.

Solution 2: Cache Database

What if we could put all the compilation data in a single cache location, something like a database? We have an option for that.

This option is a little more complex, so let’s take it in two steps.

$ ls hello.*
hello.rb
$ rbx -Xrbc.db hello.rb
hello!
$ ls hello.*
hello.rb
$ ls -R .rbx
60

.rbx/60:
60c091c3ed34c1b93ffbb33d82d810772902d3f9

Success! No .rbc files here. But what’s with all the numbers in the .rbx directory and how did that directory get there?

The -Xrbc.db option without any argument will store the compilation cache in the .rbx directory in the current working directory. The cache files themselves are split into subdirectories to avoid creating too many entries for the file system to handle in one directory.

What if you have a special location where you would prefer all compilation cache files be saved? No problem, just give -Xrbc.db a path as follows:

$ ls hello.*
hello.rb
$ rbx -Xrbc.db=$HOME/.my_special_place hello.rb
hello!
$ ls hello.*
hello.rb
$ ls -R $HOME/.my_special_place
60

/Users/brian/.my_special_place/60:
60c091c3ed34c1b93ffbb33d82d810772902d3f9

If you primarily work with projects, putting the .rbx directory in the current working directory may be the best solution because it keeps the compilation cache with the project. It is easy to add an SCM ignore for the directory and easy to remove the directory to clear the cache (e.g. in a clean task).

However, if you are frequently running scripts in many directories, you may not want to litter .rbx directories everywhere. In this case, putting the directory in your $HOME dir or /tmp may be preferable. Additionally, /tmp may be cleared on every reboot so you will not accumulate many stale cache files.

Note that, right now, Rubinius does not clear the cache directory. It will happily continue adding to it indefinitely. However, this may not be an issue unless you are cycling through a bunch of Ruby files, for example, working on a number of Ruby projects in series. In that case, using a per-project (per current working directory) cache is probably the best option.

Here is how solution 2 shakes out:

Use Case: You want to combine all compilation cache files in one location.

Pros: No .rbc files mixed in with the rest of your files.

Cons: You may still need a per-project or per-working-directory cache directory. However, you can easily specify where to put that directory.

Using RBXOPT for Options

As mentioned above, the -X options can get a little long and you certainly don’t want to retype them constantly. We have added support for the RBXOPT environment variable, which is an analog of the RUBYOPT environment variable that we already support.

Use RBXOPT to specify -X options that Rubinius should use. For example:

export RBXOPT=-Xrbc.db=/path/to/dir

You can check out all the -X options with rbx -Xconfig.print or rbx -Xconfig.print=2 for more verbose output. If you want to use multiple -X options in RBXOPT, use quotes and separate the options with a space:

export RBXOPT='-Xrbc.db -Xagent.start'

Conclusion

Rubinius saves a compilation cache for compiled Ruby code to avoid wasting time and resources recompiling source that has not changed. However, we need some place to store the cache. Rubinius provides options for omitting the cache altogether or for storing it in a directory of your choosing. Note that the format of the compilation cache is an implementation detail and we reserve the right to change it at any time, so please don’t rely on it being in any particular format.

We have not turned on -Xrbc.db by default yet because we don’t know what a good default is. So give us feedback on your use cases and what you would find most useful.

Finally, whenever we discuss the compilation cache we are inevitably asked if you can run directly from the cache and not use the Ruby source at all after it has been compiled. The short answer is “Yes”, the long answer is “It depends”. I will be writing a post exploring this question in detail shortly. For now, get out there and write more Ruby code!

Why Use Rubinius

Why should I use Rubinius? We have been asked that question many, many times over the past four years. It is a great question. It is an important question. It’s a hard question. I’m not holding out on you. I want to give you an answer that sates your curiosity, helps you make informed decisions, and empowers you to speak eloquently when you are inevitably asked, “Why do you use Rubinius?”

The trouble is, there are many different situations in which people use Ruby and there is simply no answer, however comprehensive, that really speaks to everyone’s concerns. So rather that boring you at length, I thought a Choose your own adventure style would be a better approach.

From the list below, select the persona that best describes you. Don’t worry, if the one you select doesn’t sound right, you can easily backtrack here. Read as many as interest you. After all, none of us fit easily into any one box. When you are done exploring all the fascinating reasons to use Rubinius, let’s meet up at the Conclusion for some parting words.

Enjoy!

Choose Your Persona

Rails or Ruby Newby

You are pretty new to programming and after hearing about Ruby on Rails you watched a screencast and made a website. You are curious and enthusiastic.

You are the empty teacup of the Zen proverb. You are a fresh-faced flower glistening with the morning dew. The sun smiles on you and you smile back. You seem to like this Ruby language that makes programmers happy and you’ve come to lend your cheery spirit…

Welcome!

So, you have heard of this thing called Rubinius or rbx or whatever and some folks you respect or admire seem to like it and naturally you want to know what the big deal is and you’re like, “Yo, why would I use Rubinius?”.

Cool.

Well, you should use Rubinius because I said so. Try your code on it. Tell us what worked for you. Tell us if something didn’t work by opening an issue. Set your imagination loose and tell us what tool you would use if you could.

Spend some time reading the Rubinius source code. Start at the kernel/ directory. It’s full of Ruby code! As you read through how Ruby is implemented, how it actually works, it will give you a level of understanding of your code that many programmers don’t have in any language.

Most of all, hang on to your curiosity and enthusiasm. Those were vital to the creation of the Rubinius project in the beginning and have sustained us through many challenges. We can make our Ruby experience better, freeing us from the shackles of other languages and foreign libraries. We can have fast and reliable web servers, games, editors, websites and applications written in Ruby. We can have first class tools written for and with Ruby. The world can be rosy red without our glasses.

Back to personas

The Creative

Ruby is groovy. No, not that Groovy, eww, no. I mean:

groovy |ˈgroōvē| adj.

  • fashionable and exciting : sporting a groovy new haircut
  • enjoyable and excellent : he played all the remarkably groovy guitar parts himself

(Apple's dashboard dictionary widget.)

Ruby respects creativity. It has an aesthetic. You don’t just write Ruby code, you write beautiful Ruby code. It would be unthinkable to do otherwise. Sure, there is more than one way to do many things. This is not some sterile laboratory. We are not automatons; we are people. Of course, being utilitarian is not bad. But other languages have that angle pretty well covered. There is probably only one right way to implement Python.

Rubinius has an aesthetic, too: excellence, utility, simplicity, beauty, joy. Mostly in that order. Useful code that isn’t of very good quality is a drag. It slows you down. It gives you a headache. It drives you away. We strive to keep it out of Rubinius. On the other hand, we are not just writing sonnets here. This is Serious Business™. We have some hard-core problems to solve. So we strive for excellent, useful, beautiful code that is a joy to work with.

Of course, this is an ongoing process. It is a journey, not a destination. There are areas of Rubinius that could use a thorough cleaning or a new perspective on making the implementation of this beautiful object-oriented language more beautiful and object-oriented.

We welcome your artistic perspective. Help us improve the dialog between Rubinius and the person using it. The command line doesn’t have to be a desolate place of obscure, condescending error messages. Web interfaces to the diagnostic tools deserve a good dose of user-experience and interaction design. You know that feeling you get when looking at an Enterprise web application? That weird plastic-masquerading-as-quality-material feeling? The too much 1996-Enterprise-faux-rounded-corner-wanabe-2006-hip gloss? Gives me the willies whenever I have to use an app like that. Yeah, we don’t want that.

We want to create tools that are powerful, graceful, easy to use, and beautiful to look at. Beautiful tools are easier to use. (Yehuda Katz provided a couple links related to this: The Impact of Design and Aesthetics on Usability, Credibility, and Learning in an Online Environment and In Defense of Eye Candy. If you know of other research, leave us a comment.) So if you have a creative bent but enjoy writing code also, try out Rubinius and let us know where it could use some polish.

Back to personas

Experienced programmer

That saying, Time is Money, you live by it. You have applications to deliver and you choose the best tool for the job. You are professional, conscientious, duly cautious, and not inclined to episodes of emotional exuberance about the latest fad. You accept compromises. There are always trade-offs. The correct approach is cost-benefit analysis. The numbers tell the story and level-headed decision making follows the numbers.

You have heard about Rubinius and you are curious whether it may be appropriate for your current project. As usual, rather than speculating or paying too much heed to the buzz, you look into it yourself. After some investigation, you discover that:

  1. Much of Rubinius is implemented in Ruby itself. This may be a big help when tracking down troublesome bugs.
  2. Rubinius has a very fast bytecode virtual machine, as well as a modern generational garbage collector so memory profiles should be more predictable and consistent in deployed applications.
  3. It has a profile-driven JIT compiler that uses type-feedback to aggressively inline methods resulting in significant performance improvements.
  4. It has a built-in debugger and precise method profiler, both of which are fast due to being well integrated.
  5. It has a built-in API for monitoring a VM out-of-process, even on a remote machine. We are building a variety of diagnostic tools atop this API.

Of course, even if the technology in Rubinius sounds terrific in theory, how suitable is Rubinius for your application? How does it perform under your specific constraints? Again, you do some investigating. You have a solid test suite for your application, so you start by running that. If you hit any problems, please open an issue to let us know.

If everything goes well with the tests, you start running some of the benchmarks that you have accumulated while doing performance tuning. Of course, no sensible person asks for benchmark results from other people’s code. That defies logic. It’s like asking if your program will run because your Aunt Mabeline likes decaf coffee. It’s contrary to the very point of benchmarking, where you are trying to correlate two values that are connected.

Again, if you note an significant issues, please let us know. Sometimes Rubinius exposes issues in existing code. Performance characteristics of real applications are vital to making Rubinius faster. Also, if you have suggestions for tools you would like to use, tell us. If you just want to chat about the technology, that’s fine, too. We’re hanging out in the #rubinius channel on freenode.net.

Back to personas

Seasoned programmer

Well, I am being kind by saying seasoned. You know when you look in the mirror that jaded and cynical are much more apt. You’ve seen it all and it has worn you down. You’ve been fighting the good fight, carefully guarding that last flicker of optimism that burns in the secret place deep in your heart. You’ve programmed Java/.NET/C++ professionally. You’ve even sucked it up and written some PHP and Python when asked; you are a professional, they ask and you deliver. You’ve seen attacked servers on fire off the shoulder of Rackspace…

Rubinius has a lot to offer you. Remember that little flicker of optimism? It is only the idealists that get ground down by the complete indifference to pursuit of an ideal in so much of the world. Deep down, you are an idealist and you will find plenty to refresh you here.

Rubinius aims to be the best possible implementation of Ruby by putting Ruby itself front and center. We are using modern technology and always improving. We change when there is a better way to do things. We judiciously rewrite and are not too attached to any code or algorithm. The legacy Enterprise isn’t on the steering committee. Our work will be done when you can use Ruby, just Ruby, to solve your thorny problems.

Sure, that sounds idealistic. But never mind the pessimists that tell you that you have to compromise. If you are not idealistic, you will not be unsatisfied with things that are not as good as they could be; you will not try to change the world. So give Rubinius a try, you may be surprised. And if you are, put all that hard-earned wisdom you have gained to use for the betterment of Ruby.

Back to personas

Academic Researcher

Forgive me for staring, I know it is impolite. I’m just… intrigued. Of course, you know Ruby is a late bound language, every message sent could conceivably fail to find a target, potentially resulting in an uncaught exception and program termination. There’s shared state, wild orgies of mutation that disallow any reasonable attempt at automated parallelization. Program proof is as oxymoronic a concept as military intelligence. It’s a very messy affair of programming and meta-programming and meta-meta-programming, which, for the love of Lisp, could be done so simply with macros. There’s all this eager evaluation and complete disregard for purity. Despite vast odds, somehow programs are written that actually run. You have noted all this with great objectivity but you are nonetheless interested.

Excellent, we are pleased. We have much to learn and welcome the opportunity for lively discussions about bringing formal methods to bear on the problems of making Ruby as fast as possible.

Java benefited tremendously from the amount of attention it received by academic researchers. Ruby can benefit from some of this research as well, not to mention the research into Smalltalk and Self that preceded it. But Ruby has its own set of problems to solve and deserves specific attention. The problems are hard but not insurmountable. Rubinius is already demonstrating that. The suggestion that we need to add more keywords, restrict Ruby dynamism, or write public static final int all over are simply nonsense.

Rubinius already leverages research for fast virtual machines, garbage collection (e.g. the generational approach and the Immix mark-region algorithm), and JIT compilers (based on pioneering research done in Self and used in the JVM Hotspot VM). Rubinius uses the exceptional LLVM project for optimization and code generation in the JIT compiler. We are also working on better infrastructure for the JIT to address Ruby complexities head-on.

Rubinius would be excellent to use in teaching. A compiler construction class could study the architecture of the bytecode compiler written in Ruby and experiment with exploratory changes to the compiler using IRB without having to recompile anything! A 30-minute introduction to Rubinius could proceed immediately to simple AST generation and have students experimenting with their own syntax immediately. While it is easy to get started, there is plenty of depth for exploring complex topics in virtual-machine construction and garbage collection.

Whether you are interested in language research or language pedagogy, Rubinius is an great project to consider. We look forward to hearing from you.

Back to personas

Über programmer

You learned the untyped lambda calculus sitting on your mother’s knee while she worked on her doctorate in computer science. You were substituting terms before you even uttered the word, “dada”. You wrote three different Lisp implementations in Commodore Basic before you were seven. You can write multi-threaded web servers in one pass with no tests and never hit a deadlock or critical data race. You write parsers and compilers for odd languages on a Friday night for the heck of it while waiting for the pizza to arrive before a night out at the karaoke bar where you give an inspiring performance of Laga Gaga’s Poker Face.

(Loooong pause. You’re not reading this. You’ve already written one or a few languages on Rubinius and posted them to our Projects page. But anyway, I’ll continue…)

You are the Luke Skywalker of Ruby; Yoda has nothing more to teach you. Only your fate confronts you now. Use the Source Luke and save the Federation of Ruby loyalists from the Evil Oracle and its Java the Hurt.

There are a number of domains in which Ruby could benefit tremendously from excellent libraries:

  1. Servers and web servers: the web is here to stay but the argument that all applications are going to be in Javascript on the client is not valid. A variety of hybrid client-server architectures will continue to be the norm. We need software that enables application authors to build a suitable solution to their particular problems rather than trying to stuff their apps into someone else’s solution with layers of wrapping.
  2. Concurrency: multi-core is here to stay but it is not only functional programming that is suitable for high-concurrency applications.
  3. Graphical user interface: the web browser is also here to stay but it is not the last word in applications. There are many cases where GUI apps are the best option and Ruby needs a mature library or set of libraries to build these apps on any major platform. I know some of these libraries exist, but they seem to be collecting dust lately.
  4. Big data and data analysis libraries: our industry repeatedly witnesses the same pattern: domain X starts with huge applications running on huge horsepower servers for huge businesses and then it starts appearing in small applications on small computers for small businesses. Accounting and geographic information systems (GIS) are two examples. Data analysis is coming to a laptop near you.

These are general areas in which Ruby can be an excellent solution. So how does Rubinius fit in? Rubinius is dedicatedly pushing more and more into Ruby itself. Each of these domain is typically handled in Ruby right now by going to find a library in a foreign language to wrap in a fuzzy Ruby embrace. Rubinius is calling on the über-programmers of the world to implement solutions in Ruby to help us identify performance challenges and address them.

Rubinius is also being used in some fascinating language experiments. Two of these are Atomo (http://atomo-lang.org which is implemented in Haskell, with a Rubinius implementation code-named quanto) and Fancy (http://fancy-lang.org). So, if language design is your cup of tea, Rubinius offers an excellent platform for experimentation.

Back to personas

Philosophy Student Seeking the Meaning of Ruby

Like your persona description, you tend to be long winded. You find most descriptions too brief, almost dismissive. There are words and words should be used to delve into the minutiae of minutiae. You, more than anyone, want to know “Why?” with every fiber of your being. You will continue asking long after the supply of hallucinogens has been exhausted and everyone else is drooling in their sleep.

For you, Rubinius is an existential dilemma crying out for justification. If we already have MRI, why build Rubinius?

It would be accurate to say that Rubinius has a philosophy. That philosophy is simply this: Ruby should be a first class language. What does that mean? Simply that it should be possible to solve problems writing Ruby code.

Let’s consider libraries: Being first class means not having to wrap a Java library or build a C extension. If wrapping the library were the end of the story, it wouldn’t be so bad. But that is never the case. Libraries have bugs, weird APIs, incompatibility with other libraries, threading issues, and disappearing maintainers. They may even be incompatible with newer versions of the language in which they are written.

This list goes on. To address any one of these issues requires delving into a different language with weird and incompatible semantics. If the library is your core competency, that’s not such a big deal. But I will wager that it is not, which is why you are using the library in the first place. Also, the language in which you are wrapping the library (Ruby here) is not likely the core competency of the library author, or you probably wouldn’t need to be wrapping it. So Ruby wrapping one of these libraries will always be a second-class citizen. Decisions will be made about the library’s API that do not give one thought to the Ruby programs using it. Furthermore, the code written in that foreign language does nothing to support the ecosystem of Ruby. The knowledge gained in writing the library and the improved skills of the library author do not benefit Ruby. Ruby deserves better.

Ruby has gotten a big boost recently with the production release of MRI 1.9.2. There are significant speed improvements and welcomed additions to the core libraries, like powerful handling of String encodings. At the same time, the Complex and Rational libraries were added to the core library and rewritten from Ruby to C code. This is disappointing. We should be able to solve these problems more effectively in Ruby itself.

The philosophy of Rubinius is to make Ruby a first-class citizen. Ruby plays second fiddle to no one. There is no other language whose history, semantics, or vested interests compete with Ruby’s. It is true that there are difficult problems to solve in making Ruby fast. But much of the technology already exists and we will build what does not. Evan often quips that if we can get Rubinius caught up to the dynamic language technology of ten years ago, Ruby will be light-years ahead. That may be overstating how far behind Ruby is, but it illustrates the focus of Rubinius.

There’s the saying, In theory, there is no difference between theory and practice. In practice, there is. In Rubinius, theory and practice are merging. We are motivated by the desire for Ruby to be a first-class language. But we are also showing real progress in making that a reality. The Rubinius VM executes Ruby code blazingly fast. The JIT compiler, while still being quite young, is showing great promise. Compatibility with MRI is quite good and speed is constantly improving.

Is the Rubinius philosophy valid? We think the proof is in the pudding.

Back to personas

Manager

No, it did not cross my mind to describe this persona as Pointy-haired Boss. Not only would that be unfair to Dilbert, but that persona would be reading an article on Web Scale. No, you are someone who has fought hard battles in the trenches and learned valuable lessons: it’s about execution and execution depends on good technology.

Rubinius is building solid technology. We started the RubySpec project and have contributed tens of thousands of lines of code to it. With the support of Rubyspec, in just over four years as a public project, we have basically caught up with MRI 1.8.7 in compatibility and performance. For some code, our performance is much better, for other code, it is not as good. However, Rubinius is built on solid, modern technology and the project’s trajectory and velocity are outstanding.

Rubinius is a completely new implementation of core Ruby. Rubinius did not start as a port of existing code. Furthermore, Rubinius implements its own virtual machine and garbage collector in C++. The bytecode compiler that targets the virtual machine is pure Ruby. The core Ruby library is mostly Ruby with some primitive operations in C++. The JIT compiler uses the LLVM project. Given the amount of work being done in the project, Rubinius is pacing extremely well relative to other implementations.

Currently, we are working on support for Ruby 1.9 features, Windows support, and full concurrency with no global interpreter lock (GIL).

If you are looking at Ruby to implement your next project, rest assured that Ruby will have the support of excellent technology. If you are already using Ruby, consider investigating how your application runs on Rubinius. We welcome the feedback and look forward to solving challenging engineering problems.

Back to personas

Knowledge Seeker

You thirst for Knowledge. You follow it wherever it leads you. You’ll happily walk Haskell’s hallowed halls of pure laziness or sit at the feet of the meta-program gazing raptly at class transmorgrification. You don’t judge. You have more than enough knowledge to be dangerous, enough to know that the universe is amoral and knowledge is the only Truth there is. Nor does any mere mortal language bind you. All languages are finite. You’ll be here today and gone tomorrow; there is no permanence for the knowledge seeker.

Rubinius is merely a step along the path you journey. Take what you want, it is all free. As a Ruby implementation, it has much to offer your quest for knowledge. The Ruby code in the core library is accessible and easy to follow. The interface between Ruby and the C++ primitives is consistent. The C++ code itself is restrained. You won’t need a PhD in Turing-complete template languages to understand it.

Rubinius offers extensive opportunities to learn about programming languages in general and Ruby in particular. When I first started working with Rubinius, I knew a little bit about garbage collection and virtual machines. I would call what I knew, toy knowledge. As I struggled to learn more, it seemed helpful to consider layers of understanding:

  1. General programming language semantics: the procedure abstraction, looping and iteration, recursion, references and values, etc.
  2. Ruby semantics: modules and classes, access restrictions, blocks and lambdas, etc. Even with fundamental programming knowledge, a particular language can be confusing. When I was learning C, a friend was also studying it. One day he walked over and threw The C Programming Language book down on my desk and said, “This for loop makes no sense!” He was quite upset. “Look,” he said, “in this example for (i=0; i < n; i++) how can i < n get executed after the code in the body?!” It’s easy to laugh at that confusion, but coming from BASIC, that really threw him. Deepening our understanding to this second level requires confronting some “counter-intuitive” notions.
  3. Hypothetical implementation: knowing how Ruby works, how might one implement it. I think this is an important layer of understanding and it is easy to miss or gloss over it. By pausing at this layer and thinking how you might implement something, you test whether or not you are really understanding it.
  4. The MRI implementation: Reading the MRI source code is an excellent way to investigate Ruby. For one thing, it will inform you how Ruby actually works, and you may be surprised.
  5. The Rubinius implementation: here you are exposed to the philosophy of Rubinius and the challenges to implementing Ruby. We are attempting to bring the beauty of Ruby as an object-oriented language deep into the core of Ruby itself.

While the Rubinius code itself offers many opportunities for learning, don’t hesitate to drop by the #rubinius channel on freenode.net and ask us questions. Perhaps you already know a lot about another language and are interested in how Rubinius implements some feature. Or you may be relatively new to programming languages and have some basic questions. We enjoy talking about these concepts. If you are quite new to Rubinius, you may find these posts informative:

Finally, consider helping other knowledge seekers by writing blog posts on what you learn about Rubinius. Or, help us write documentation!

Back to personas

Language Enthusiast

You like languages for their intrinsic value. Of course the world comes in many shapes and sizes. You wouldn’t have it any other way. That’s the fun and spice, joie de vivre, raison d’etre, supermarché… Sometimes you get carried away writing a program in another language just because you like how the letters arrange down the screen. Ruby is definitely one of the impressive languages and sometimes you almost notice a tiny bit of favoritism in your normally egalitarian attitude.

As with any enthusiast, you like to experiment. Your interest is not mere curiosity or sterile investigation. You want to get your feet wet and your hands dirty. Rubinius is an excellent opportunity to delve into a number of fascinating subjects. We can merely suggest a path; your experiences along the way will tell you whether or not Rubinius has value to you.

If you are most interested in languages themselves, the syntax and arrangement of features, Rubinius offers you immediate gratification. Look for Evan’s upcoming post on his Language Toolkit or check out the code to prattle, a Smalltalk dialect used to illustrate the ease of building a language on Rubinius. Also look at some of the existing languages projects targeting Rubinius.

If it is the machinery under the covers that is more interesting, start reading some code. The bytecode compiler lives in lib/compiler/. The virtual machine is in vm/, and the garbage collector is in vm/gc. As you are reading through, consider helping us write better documentation. There are already sections for the virtual machine, garbage-collector, JIT compiler and bytecode compiler in the documentation, so adding content is easy.

You may also be interested in these previous posts about Rubinius:

Most of all, experiment. Rubinius is easy to hack on. Are you curious about a particular feature needed in your language? Try adding it to Rubinius. Think Lua is all the rage because it uses a register VM? You could probably write a register-based bytecode interpreter for Rubinius in an afternoon. That’s just an example, of course. The point is to play around with your ideas and have fun doing it. I think you’ll find Rubinius to be an adventuresome companion.

Be sure to let us know what you’re working on. We like to be inspired, too! Consider writing a blog post about things that you find interesing, like this recent post by Yehuda Katz.

Back to personas

Conclusion

So there you have it. Just like there are many different viewpoints, there are many different reasons to use Rubinius. Not all those reasons make sense to everyone. We believe, however, that Rubinius has something to offer to just about everyone interested in Ruby. Most importantly, try it!

If we didn’t answer your question here, leave us a comment. If you have a reason for using Rubinius that we didn’t mention, let us know. As always, we appreciate your feedback. Chat with us in the #rubinius channel on freenode.net, watch our Github project, and follow us on Twitter.

P.S. Thanks to David Waite for suggesting the Academic Researcher and Language Enthusiast personas, I always forget those!

Introduction to Fancy

Fancy is a new general-purpose programming language targetting the Rubinius VM.

This blog post will give a short introduction to the language, what kind of problems it’s trying to solve and why I chose Rubinius as the VM to run Fancy on.

What is Fancy?

Fancy is a new general-purpose, dynamic, pure object-oriented programming language heavily inspired by Ruby, Smalltalk and Erlang that runs on the Rubinius VM. It’s the first fully bootstrapped language, aside from Ruby, running on Rubinius. This means that the compiler that generates bytecode for Rubinius is written in Fancy itself.

You can think of Fancy as a mix of features from the mentioned languages above, taking each of their strengths and improving upon their weaknesses. Fancy has a very small core and is largely based on the concept of message passing, just like Smalltalk. It tries to have as many language concepts being first-class values in the language.

Just like Ruby, Fancy is a dynamic object-oriented language that allows changing code at runtime, everything being an expression and generally embracing more then one way to do things. Fancy also has all the literal support that Ruby has, plus literal syntax for Tuples and Patterns (more on that below).

In contrast to Ruby and just like Smalltalk, Fancy has a very small amount of built-in keywords and all of the control structures are implemented in terms of message sends to objects using closures.

The third language that served as an inspiration is Erlang, from which Fancy takes the idea that concurrent programming should be easy by having the Actor Model built into the language. This part is still a work in progress, but should come together soon. The fact that Rubinius has a built-in Channel type, inter-VM communication capabilities and even an actor library makes implementing this easier than in traditional systems.

Why Fancy?

I believe there is real value in having a language that supports certain things out of the box. Especially when it comes to things like asynchronous and concurrent programming, having proper semantics built into the language can often help developers more than a library can. Very often it’s not just about the functionality itself but also about the semantics you want that functionality to have. This can cause problems particularly if the language’s semantics differ from what your library is trying to solve. A good example is the callback-based approach to asynchronous progamming which leads to code that differs both in semantics as well as how code is structured, compared to synchronous code. Ideally you’d still want to write code in a synchronous fashion, where exceptions pop up naturally while still being highly asynchronous.

In that sense Fancy is more flexible than Ruby as there’s not many special case semantics built in to the core language. Everything’s done via message passing, which fits nicely the actor model approach to concurrency. Fancy’s syntax is a lot simpler, too.

Since all the core control structures are just implemented in Fancy itself and adhere to the message passing protocol, you can easily override them for your personal needs. This is especially interesting when implementing domain specific languages. Say, you’d want to add some logging to conditional or looping constructs - it’s as easy as overriding a method in your DSL’s classes. Fancy also has class-based mixins, so it makes it easy to share functionality across class hierarchy boundaries.

Finally, I created Fancy because I wanted a language implementation that was well documented, easy to understand and very flexible to extend. Ruby is a nice language, but it has some inconsistencies and there’s only so much you can do when you’re bound by backwards compatibility. By starting fresh, Fancy has a clean, simple and easy to extend core which allows further exploration of features and abstractions.

Why target Rubinius?

The initial implementation of Fancy was a simple interpreter written in C++, similar to how Ruby 1.8 (MRI) works. It was a simple AST walker. After moving to Rubinius and writing an initial bootstrap compiler in Ruby, the codebase shrank to about 20% of the original implementation while actually being more performant. This of course is mostly due to Rubinius’ architecture and JIT compiler but it was a great experience nontheless.

The nice part about having a common virtual machine and runtime is that you’re not forced to a completely different platform to get the job done. Fancy and Ruby can coexist in the same application nicely and calling code from one another is dead simple. In fact, as of now, Rubinius doesn’t know anything about Fancy. And it shouldn’t. As long as all languages running on top of it adhere to the same interface (in this case the bytecode), it should just work fine.

Choosing Rubinius as a successor platform for Fancy was easy. It’s built for Ruby, a language that’s closely related to Fancy. Rubinius, while having been developed as a VM for running Ruby code, is very flexible and there are many features that abstract over Ruby’s external semantics. It was just a natural choice given the fact that Rubinius’ architecture and design was heavily influenced by Smalltalk VMs. Also, it’s a very nice dynamic bytecode virtual machine. The community is very responsive and helpful. Bugs get fixed instantly, there’s always someone to help out and overall it’s been a great experience.

Let’s look at some code!

OK, enough talking. Let’s have a look on how to get some Fancy code up and running. Our little sample application will be a simple IRC bot that connects to Fancy’s irc channel on Freenode and says hello to everyone that greets it. To make life easier, there’s already a Fancy package out there that helps with exactly this task: FancyIRC.

FancyIRC is a simple IRC client library inspired by Ruby’s IRC bot framework Cinch. It’s much simpler and the code is fairly easy to read, but it gives you a similar interface for writing IRC clients or bots.

So let’s get going by installing Fancy. You can either use the Fancy Rubygem and install it with Rubinius or get the code from GitHub and run rake in the directory. You’ll also then have to add the bin directory to your $PATH. If you want the latest and greatest version of Fancy I recommend building directly from source, as the Gem might not be up to date all the time. For demonstration purposes, let’s install the Rubygem.

$ rbx -S gem install fancy

To get the FancyIRC package we use Fancy’s built-in package manager, which knows how to find the code on GitHub and install it locally:

$ fancy install bakkdoor/fancy_irc

Writing the code

 1 require: "fancy_irc"
 2 
 3 greeter_bot = FancyIRC Client new: {
 4   configuration: {
 5     nickname: "greeter_bot"
 6     server: "irc.freenode.net"
 7     port: 6667
 8     channels: ["#fancy"]
 9   }
10 
11   # greet person back
12   on: 'channel pattern: /^[hH]ello greeter_bot/ do: |msg| {
13     msg reply: "Hello to you too, #{msg author}!"
14   }
15 
16   # "echo" command
17   # invoke with: !echo <text>
18   on: 'channel pattern: /^!echo (.*)$/ do: |msg, text| {
19     msg reply: "#{msg author} said: #{text}"
20   }
21 
22   # tell bot to shutdown via !shutdown command
23   on: 'channel pattern: /^!shutdown/ do: |msg| {
24     msg reply: "OK, shutting down"
25     System exit
26   }
27 }
28 
29 greeter_bot connect
30 greeter_bot run

I think the code is pretty straight forward. This should give you a feeling for what Fancy looks and feels like. There is of course lots more to Fancy than what was shown here. It would not fit into a single blog post.

A quick list of what’s currently being worked on:

Interested?

If you got interested in Fancy and want to know where to go next, here’s a short list of things to check out:

Running Multiple Rubinius Branches Simultaneously with RVM.

This article is written with the assumption that you have RVM installed already. If you do not, follow the Installation Instructions followed by the Basics closely first.

Named Ruby Installs

Everyone familiar with RVM knows that it allows you to quickly and easily install a particular Ruby interpreter by simply running, for example,

rvm install rbx

What is not widely known (yet) is that there is a “Named Rubies” feature that allows you to install altered versions of the same Ruby installation along side the original.

In the case of Rubinius there is this facinating branch called ‘hydra’. So let us see how we can have the Rubinius master branch installed as the main rbx with the hydra branch installed along side as well.

As above you first install rbx which is currently defaulted to -head version so

rvm install rbx

is currently equivalent to

rvm install rbx-head

After we have the mainline head Rubinus branch installed, we now want to use the named rubies feature. This is done using the -n specifier in the Ruby identifier string. So for example to install our hydra branch as an RVM ruby with the name ‘hydra’ in it we do the following:

rvm install --branch hydra rbx-nhydra

Now we can see that they can be used together! Using the Rubinius master environment,

$ rvm rbx ; ruby -v
rubinius 1.2.1 (1.8.7 6feb585f 2011-02-15 JI) [x86_64-apple-darwin10.6.0]

Whereas using the Rubinius hydra environment,

$ rvm rbx-nhydra ; ruby -v
rubinius 1.3.0dev (1.8.7 6feb585f xxxx-xx-xx JI) [x86_64-apple-darwin10.6.0]

We see that the next release of Rubinius (hydra branch) is indeed version 1.3.0 whereas the master branch is version 1.2.1.

Also please note that RVM creates wrapper scripts, so you do not need to switch out the entire environment just to run the differen versions either:

For Rubinius master,

$ rbx-head -v
rubinius 1.2.1 (1.8.7 6feb585f 2011-02-15 JI) [x86_64-apple-darwin10.6.0]

For Rubinius hydra,

$ rbx-head-nhydra -v
rubinius 1.3.0dev (1.8.7 6feb585f xxxx-xx-xx JI) [x86_64-apple-darwin10.6.0]

There is a lot more available to you than this, for more information on RVM capabilities please visit the RVM Website and also come talk to us in #rvm on irc.freenode.net during the daytime EDT.

I hope that this is helpful and informative to you!

~Wayne

Rubinius, What's Next?

On Tuesday, we released version 1.2.1 (see the Release notes). This release weighs in at 256 commits and 21 tickets closed in the 56 calendar days since the release of 1.2.0. Many thanks to those who contributed patches and to everyone who helped us test it.

While we were working on 1.2.1, we were also working on a Top Secret project that we’ve craftily hidden in plain sight. I’d like to introduce the work we are doing on the hydra branch and the features you can expect to see in Rubinius soon.

Daedalus - A new build system

Rubinius is a fairly complex project. It combines multiple components into a single system. We have worked hard to contain this complexity and from the beginning we insisted that building Rubinius be as simple as possible. For example, Rubinius can be run from the source directory, there is no need to install it first. Typically, building requires:

./configure
rake

The Rubinius system combines:

  1. External libraries written in C/C++, sometimes built with just Makefiles and sometimes using autotools.
  2. The virtual machine, garbage collector, and JIT compiler written in C++.
  3. The virtual machine interpreter instructions, including support code for the JIT, and instruction documentation all generated at build time from an instruction template.
  4. The core library and bytecode compiler written in Ruby.
  5. Various C extensions like the Melbourne parser, BigDecimal, Digest, and OpenSSL libraries. In the case of the parser, we have to build two versions, one for the bootstrapping system and one for the Rubinius system being built.

It has not been easy to make this work and over the years we have compiled a list of exactly what we need in a build system. Evan, in typical form, started hacking out a first pass and created daedalus, our new build system. It features such exotic (and extremely useful) features as SHA-based change detection, parallel builds, single-process execution, and use-aware configuration options. Allow me to elaborate.

Full-on Concurrency

Nobody likes waiting in line. In fact, the more desirable a thing is, the less we want to stand idly waiting in a line for it, tapping our foot, twiddling our thumbs. The same could be said about our programs.

Threads give us the ability to add concurrency to our programs. However, unless the hardware either has multiple CPUs or multiple cores per CPU (or both), the apparent concurrency will still be executing serially. Since there are so many multi-core CPUs around these days, our programs should be getting stuff done in parallel.

Unfortunately, there’s a twist. Even with native threads on a multi-core CPU, the amount of parallelism you get depends on how well you manage locks around shared data and resources. Sometimes managing these locks is complex and you opt for one big lock, essentially only allowing one thread at a time to run. That big lock is usually called a global interpreter lock (GIL) or global VM lock (GVL).

The Rubinius VM originally had green (user-space) threads, but it has had native threads with a GIL for a while now. In the hydra branch, Evan and contributors like Dirkjan Bussink have been working on replacing the GIL with fine-grained locks so that threads truly execute in parallel. This work has been going very well, owing in part to the fact that so much code in Rubinius is actually written in Ruby. Contributors like Chuck Remes have been running hydra under heavy concurrency loads and Rubinius is performing well.

Rubinius also has experimental support for Fibers and a built-in Actor library. There is more work to be done but Rubinius is quickly becoming an excellent platform for concurrency, with a variety of approaches available to the programmer. Evan has also suggested rewriting the Rubinius IO subsystem to enable even better APIs for concurrency, all from Ruby.

Performance

Forget everything anyone has ever told you about Ruby being slow. There are two things that make Ruby, as implemented, slow: 1) inexperience; 2) inadequate tools. These two result in one big thing: doing too much. Or, as they say: No code runs faster than no code. We have been working for 4+ years to build adequate tools in Rubinius, and there is plenty of experience in Smalltalk, Self, and other languages for making dynamic languages fast.

Presently, Rubinius typically runs pure Ruby code almost 2 times faster than MRI 1.9. However, there are also cases where Rubinius is slower. These mostly involve core libraries that are implemented in C in MRI. There are three main fronts on which we are attacking performance issues: 1) improving the algorithms in the Ruby code that implements the core library; 2) continuing to tune the VM and garbage collector; and 3) improving the JIT compiler. Which leads me to one of the most exciting things we are working on…

JIT Intermediate Representation (IR)

The just-in-time (JIT) compiler is the key to making Ruby fast. One of the biggest challenges with a dynamic language like Ruby is knowing what method is actually being invoked when a message is sent to an object. Consider the following code:

 1 class A
 2   def m(x)
 3     ...
 4   end
 5 end
 6 
 7 class B
 8   def m(x)
 9     ...
10   end
11 end
12 
13 class C
14   def work(obj)
15     obj.m(y)
16   end
17 end

What method is being invoked by obj.m(y)? There is no way to definitively know this by looking at the source code. However, when the program is actually running, we can know precisely what obj is and precisely which method m was invoked. This is called type profiling and that is exactly what the Rubinius VM does. Then the JIT uses the type information to make decisions like whether to inline a method into another method. When methods are inlined, it gives the optimizer more data and more possibilities to remove redundant code. The less code we can run, the faster Ruby will be.

Presently, the JIT compiler converts Rubinius bytecode into LLVM IR and LLVM handles the thorny task of generating machine code. However, Rubinius bytecode is designed for fast execution by the virtual machine rather than as a rich intermediate representation. So Evan has started work on a new JIT IR.

This new IR will help us to express Ruby semantics in a way that enables many powerful optimizations and will ultimately allow LLVM to generate even better machine code. Put another way, Rubinius loves Ruby code! Right down to the metal. There’s no fighting a foreign type system or the semantics of a language at odds with Ruby’s rosy view of the world.

Ruby 1.9

MRI 1.9 introduced two completely different changes to Ruby. The first was a new implementation based on a bytecode virtual machine. While the virtual machine replaced the AST-walking interpreter, little else changed architecturally. Mostly the same core library and garbage collector code exists in MRI 1.9 as was in MRI 1.8. The second change introduced some new syntax (minor) and encodings (major). Many of the other changes, for example, returning Enumerator objects from methods that take blocks, have been back-ported to Ruby 1.8.7 and are already available in Rubinius.

So, the key to supporting Ruby 1.9 in Rubinius essentially involves supporting the 1.9 syntax changes and encodings. We have begun implementing the parser changes and introduced the foundation for Encoding-aware Strings. A good amount of work remains to be done, but over the next month we expect that we will be starting to run Ruby 1.9-specific code in Rubinius.

Tools of Information

It has been said that printf is the mother of all debuggers. That illustrates two points: 1) data is often buried in our program code; and 2) we should have tools (e.g. a debugger) that enables us to access the data without manually instrumenting our code.

Presently, Rubinius has a built-in debugger, precise method profiler, memory analysis tool, and Agent interface that permits querying a running Rubinius VM–even one running on a remote machine–for a variety of information.

We will be adding the ability to track the location where objects are allocated to assist finding object leaks or code that is creating unusually large numbers of objects. We are also working on a tool to graphically display information like number of running threads, amount of CPU usage, and amount of memory used while actively monitoring a VM.

I am also curious about correlating this VM information with external data to enable play-back review. For example, I would like to monitor RubySpec runs and correlate which spec is running with the VM data. I imagine a simple monotonic reference ID provided by the VM would be useful in correlating these two otherwise unrelated pieces of data. The RubySpec runner would request the ID before running each spec and the Agent monitor would request the ID when gathering VM data. Later the two data sets could easily be merged.

When you find yourself manually instrumenting some code, consider what data you are trying to get your hands on and let us know the scenario. We’ll likely be able to build a tool that will open up new vistas into the behavior of your Ruby programs.

Windows®

However one may feel about Windows as an operating system, it is undeniable that the vast majority of people in the world use Windows. We believe those people have an inalienable right to use Rubinius, too.

Thanks to the wonderful, hard-working MinGW-w64 folks, we are able to compile the Rubinius VM into a native Windows executable. Presently, the VM will compile, link, and attempt to load the Ruby core library. More platform-specific work is needed to load the library. The next step after that will be getting the RubySpecs to run and start fixing issues.

Since the Windows work is being done on the hydra branch, the other features discussed above will be available on Windows as soon as we complete them.

Multi-language-ualization

The Rubinius VM began as an effort to create a modern, first-class environment for running programs written in Ruby. However, it turns out that Ruby is a terrific language for writing subsystems for other programming languages. Actually, this should come as no surprise; Ruby is a fabulous general purpose programming language.

To support experimenting with writing other languages that run on the Rubinius VM, Evan has started to put together a Language Toolkit. This includes things like a built-in PEG parser, convenient ways to create methods from Rubinius bytecode, and decoupling method dispatch from Ruby semantics.

Hopefully, Evan will introduce us to all this in a future blog post, but here is a taste of what you can do:

 1 class Hello
 2   dynamic_method :world do |g|
 3     g.push :self
 4     g.push_literal "Hello, world"
 5     g.send :puts, 1, true
 6     g.ret
 7   end
 8 end
 9 
10 Hello.new.world

Of course, that is much more concisely written in Ruby, but combine this ability with a built-in PEG parser and you can be experimenting with your own fascinating syntax in a matter of minutes.

Check out the Rubinius Projects page for some of these language experiments. One language in particular is Fancy, which is fully bootstrapped (i.e. the Fancy compiler is now written in Fancy) on Rubinius.

Documentation

One the one hand, Rubinius just runs Ruby code, and you shouldn’t need any special knowledge to run your application on Rubinius. On the other hand, as I’ve discussed above, there are some specific Rubinius features that may be very helpful to you. However, they can only be as helpful as the documentation we have for them.

Before we released 1.2.0 in December last year, I spent quite a bit of time getting a new documentation system in place. Since then, we’ve had contributors help with translations to Russian, Polish, Spanish, and German. Adam Gardiner started documenting the garbage collector algorithms. Yehuda Katz (you may have heard the name) has contributed documentation for the bytecode compiler complete with diagrams!. Chuck Remes wrote up a great piece on the memory analysis tool.

We really appreciate these contributions. We understand the need for great documentation and we have been creating better support for it. In many cases, all that is needed is to just open a file and start writing. Of course, one cannot expect to understand much about Rubinius without digging into the code. If there is a particular part of Rubinius that you are curious about, jump in the #rubinius channel on freenode.net and ask us questions. We can point you in the right direction and help clarify things. If nothing else, let us know which part of the missing documentation is most important to you and we can start filling that in.

How you can help

There you have it, some super exciting things coming very soon for Rubinius and for Ruby! We would love to have your help making Rubinius even better. The most important thing you can do is try running your Ruby code. Give us feedback. Let us know what features or tools would make your life easier. Help us to build them.

Rubinius adopts Ruby’s rosy view of the world. We want to empower you to solve your hardest problems with Ruby, and have fun doing it.

Rubinius Has a Blog!

Many thought the day would never come, but Rubinius finally has a blog. That’s not all, though: We have integrated the website, blog, and documentation using Jekyll. The source code for it all is in the main Rubinius repository.

People have often requested that we write more about the awesome features in Rubinius. We hear you and we’d love to do this. However, there is always a trade-off between working on those awesome features and writing about them. Until now, it’s been rather painful to write docs or blog posts because we did not have good infrastructure in place. Now, I think we do. I’m sure there are still a lot of improvements we can make, but we have a good place to start. I’d like to give a brief tour of our new system.

The primary goal was to improve collaboration and reduce friction for writing new documentation and blog posts. That’s right, improve collaboration. There are many people who have experience developing Rubinius and running their applications on it. We love how people have collaborated with source code commits. Now anyone has the ability to write a blog post as well. I’ve written a basic How-To - Write a Blog Post document. If you have an idea for a blog post, just let us know. We will exercise a bit of editorial control just to ensure the topics are appropriate for Rubinius, but generally, we are thrilled to have your contributions.

Recently, we added the rbx docs command. This will run a web server on your machine and open a browser window to display the Rubinius documentation. Now the documentation will also be available at the rubini.us website. I have added a basic outline and a bunch of files to further simplify the task of writing docs. In many cases, merely open a file and start writing docs in Markdown format.

We have also begun translating our documentation to other languages. I am excited about this, being a huge language geek. I wish that I were proficient in 10 languages so I could polish our documentation for the many people who are not native English speakers. Alas, I only have a fair ability to write in Spanish, so we are again depending on your help. I started the translation effort by passing the existing English docs through Google translate. We have a beginning guide for How-To - Translate Documentation. I’ve been told by kronos_vano in our #rubinius IRC channel that he’s already working on a Russian translation. I personally would love to see Japanese and Chinese translations.

So that’s a brief introduction to our new infrastructure for documenting and explaining Rubinius. It’s been such a joy to see so many people contribute to the Rubinius source code over the years. We hope that the blog, documentation, and translations will further empower people to contribute and benefit from the value that Rubinius has to offer the Ruby community.

¡Adelante!