I attended RubyConf in San Francisco last month, which is an annual conference about the Ruby programming language. Yukihiro Matsumoto (”Matz”), the creator of the Ruby language, gave the keynote about the 0.8 true language: a language can’t be good for everyone and every purpose, but we can strive to make it good for 80% of what is needed in a programming language. He talked about domain-specific languages (DSLs) and, while he was talking about programming DSLs, it struck me that we, as humans, commonly invent domain-specific languages that transcend our social cultures, instead encoding a culture that crosses national and linguistic boundaries.

Yuki Sonoda organized a series of talks called East meets West with presentations by Japanese Rubyists. In her talk description, she makes the case for our bridging the Japanese-English language gap between Ruby programmers:

Ruby needs your help. There are many issues. But there are too few developers. 92% of Ruby’s development in this 3 years were done by only 10 developers. 73% were done by only 5 developers. Ruby seems to be a cathedral project rather than a bazaar project.

There must be many reasons for this situation. I think a large reason is the language barrier between English-speaking Ruby world and Japanese-speaking Ruby world. So I will talk about how to solve this problem.

All of the top 10 committers speak Japanese and live in Japan. So they discuss in Japanese. Some of the most important decisions are done in these discussions. But this means that most of Rubyists, who do not speak Japanese, can not understand the discussions. For non-Japanese speakers, there has been no way to understand the most important issues in the development of Ruby.

I want to share the current issues of Ruby. I also want to request help from Rubyists who don’t speak Japanese.

There were two “lightning” tech talks given by Japanese Rubyist who each said that it was their first English language presentation. I started to think about what kind of vocabulary I would need to give a tech talk in Japanese or even just to understand one.

I approached Matz after his keynote to ask if he would record some phrases about the Ruby language in Japanese. He agreed, and I set out to capture a dozen or so phrases that would never appear in a phrasebook and might be interesting to say to a Japanese Rubyist at a conference.

I approached random people in the hallways and during lunch and asked questions like: if you would to use the word “closure” in a sentence, what would it be? Jim Weirich came up with my favorite: “Closures may be used to implement objects, and object may be used to implement closures.” Sarah Mei wondered how to read code aloud in Japanese — when would you use a Japanese word and when is the code pronounced phonetically. She guessed correctly that you would say “object.method” phonetically as obujekuto dotto mesoddo. I was intrigued that what Ruby calls the “shovel” operator (<<) is phonetically derived from the bitshift operator which has the same symbolic representation in C and Java, and is thus translated as bitto shifuto.

You can see all of the phrases that Matz recorded on the home page. One of these days, I will make it so there is a direct link. If you program in Ruby and speak English or Japanese, I’d be interested in knowing if there domain-specific phrases you would like to be able to say. I wonder if I learned enough “code” words along with some basic Japanese whether I could actually understand a Ruby Kaigi talk even before learning how to converse in Japanese.

I wish I remembered everyone’s names who suggested phrases. If you read this, please comment so that I can say thanks! and many thanks to the Japanese engineer who translated the phrases for me and her colleagues who helped! In my zealous pursuit of my goal, I neglected to keep track of everyone who helped me along the way. Thank you all.

“The english of today is not what it used to be, but then again, it never was,” writes Guy Deutscher in The Unfolding of Language. In Chapter 3, he includes a fascination chronology on a few hundred years of so-called decline:

• In comparing the English language to that of two generations ago, a reviewer in the Times Literary Suppliment reminisced that then “a mistake was a mistake not a sign of free expression.”
• 1946 George Orwell wrote “the English language is in a bad way” compared to the language of previous generations
• 1848 linguist August Schleicher dismissed the English of is day as “ground-down,” noting “how rapidly the language of a nation…can sink” and that it was likely to further “sink into mono-syllabicity”
• 1780 Thomas Sheridan reported a recent decline of the English language “during the reign of Queen Anne [1702-14]… it is probably that English was… spoken in its highest state of perfection”
• 1712 Jonathan Swift wrote “our Language is extremely imperfect… its daily Improvements are by no means in proportion to its daily Corruptions”

English speakers are not unusual in feeling this way about their language.

Take modern German, for instance, which by common consent is a mere shadow of its former glory two centuries ago, in the Golden Age of Goethe and Schiller. That may well be, but during Goethe’s lifetime those in the know were of a rather different opinion. In 1819, the fairy-tale compiler and linguist Jacob Grimm compared the langauge of his day to that of previous centuries, and lamented that “six hundred years ago, every common peasant knew –that is to say practised daily — perfections and niceties of the German language of which the best language-teachers nowadays can no longer even dream.

The thesis of the book is that this it is precisely this destruction of language where the mystery of language creation lies — “all languages change, all the time — the only static languages are dead ones.” (p.55)

We don’t have to wait generations to hear the how languages change. It can be witnessed by traveling to different parts of a country or, more dramatically, by listening for differences in dialects of a single language across multiple countries.

I highly recommend the book which is a little academic at times, but full of fascinating historical and modern references that illustrate how languages evolve.

Wolf and I fixed a bug today where we needed to truncate a string of text that we use internally to annotate the database.  Now, the annotation is just for our reference, so we limit it to 50 bytes — that’s bytes, mind you, not characters, even though the PostgreSQL database will tell you it is “character varying(50)”

We use unicode internally, specifically UTF8, which is a fabulous and widely used standard.  However, it does have a challenging property where a character may be 1-4 bytes long. We were frustrated with what we thought ought to be a simple problem of truncating a string so that it would be no more than 50 bytes.  The tricky part, of course, was that the 49th byte might actually fall in the middle of a character.

To solve the problem, we added a method to the Rails Multibyte::Chars class, which is part of ActiveSupport.  For those who speak Ruby, Rails and RSpec, below is the solution we came up with (first the spec, then the implementation).

The solution we came up with was borrowed from the private translate_offset method. The key interesting part is that you can discover whether you’ve chopped up a string in the middle of a character by calling chunk.unpack(’U*’) — the unpack method on String in Ruby will throw an exception when you ask it to interpret the UTF-8 characters as unsigned integers with the “U” directive.

describe "Chars#limit_bytes" do
  it 'should return "" on ""' do
    "".mb_chars.limit_bytes(0).should == ""
    "".mb_chars.limit_bytes(1).should == ""
  end

  it 'should truncate single byte character strings as expected' do
    a = "abcd"
    a.mb_chars.limit_bytes(0).should == ''
    a.mb_chars.limit_bytes(1).should == 'a'
    a.mb_chars.limit_bytes(50).should == 'abcd'
  end

  it 'should truncate multi-byte character strings at character boundaries' do
    k = "こんいちわ"
    k.mb_chars.limit_bytes(0).should == ''
    k.mb_chars.limit_bytes(1).should == ''
    k.mb_chars.limit_bytes(3).should == 'こ'
    k.mb_chars.limit_bytes(4).should == 'こ'
    k.mb_chars.limit_bytes(5).should == 'こ'
    k.mb_chars.limit_bytes(6).should == 'こん'
    k.mb_chars.limit_bytes(7).should == 'こん'
    k.mb_chars.limit_bytes(50).should == 'こんいちわ'
  end
end

module ActiveSupport #:nodoc:
  module Multibyte #:nodoc:
    class Chars
      def limit_bytes(limit)
        limit -= 1 while !valid_boundary?(limit)
        s = @wrapped_string.slice(0,limit)
        s.mb_chars
      end

      def valid_boundary?(length)
        chunk = @wrapped_string.slice(0,length)
        begin
          chunk.unpack('U*')
          true
        rescue
          false
        end
      end
    end
  end
end

We’ve written this up as a Lighthouse ticket in case the Rails folk want to add it to the platform or if other people developing multi-lingual database apps run into the same challenge and look there.