Kopongo

That site's at cotswold.co.za.

We wanted to create a guest house site with the text in various languages.

Since the people who were going to translate the site from english into [French|Italian|German] were going to be 'geographically distributed', and not likely available at the same time, I thought it would be too complex to coordinate all those translations and the updating of the site. Creating the site in rails as a CMS was the easiest way to deal with it. When someone is ready to add or translate text, they can login and do so.

Basic Ingredients: Rails and postgres, along with the globalize, file_column and exception_notification plugin.

Installing software on the mac again. This one was not fun, but it did work out fine.

One of my projects uses the very cool RMagick gem, which requires ImageMagick or GraphMagick to be installed. This has always been a step I dread, ever since I unsuccessfully tried to install that whose thing on cygwin a year ago despite much effort. I have since installed all this on various linux boxes, and it’s usually quite simple.

Not on the Mac.

I first followed the very clear instructions I found at Hivelogic http://hivelogic.com/articles/2006/06/10/rmagick_os_x.

Unfortunately, that did not work. The RMagick gem did not install properly, and I got the now infamous bus error. Something about misc.rb, line 317.

I found some hints here http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/216875, and tried to install RMagick from source. No good. Same issue.

Then I tried looking around the web, and found many people asking the same questions. Going down further on the google list of issues for this, I started finding articles telling me to use GraphicsMagick. http://poocs.net/2005/7/6/making-rmagick-suck-less-with-tiger

Often the solution was something like ‘use graphmagick instead’. Fine, I am sure that would work, but my Linux production servers are setup with ImageMagick. Do I want to use different software in development than I use in production? No way. This has to compile.

I tried to figure out exactly what was going on in this line 317 which was such a problem. Who was not cooperating?

I wondered why hivelogic’s instruction did not quite match up with the compilation instructions at the RMagick site. Finally, I found this: http://blog.labratz.net/articles/2006/10/10/really-truly-getting-imagemagick-rmagick-working-on-osx-from-source-without-using-macports-darwinports-or-fink I t’s basically the hivelogic instruction plus the wmf and lcms libraries, and ImageMagick compiled with the instructions at the RMagick site. I added the ghostscript and the fonts as well.

And it works great!

I added the globalize plugin to an application following some instructions on the web. There are quite a few sets of instructions out there, and nothing seems to be 'official'. Depending on what you use, you can run into difficulties.

My problem was with the functional tests. Running the tests, I get something like this:

 88) Error:
test_index(ServicesControllerTest):
ActionController::RoutingError: No url can be generated for the hash {:controller=>"services", :action=>"index"}
...
...   /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_controller/test_process.rb:336:in `get'
    ./test/functional/services_controller_test.rb:18:in `test_index'

The problem is that there is no default route that the tests seem to find. All of that code that the instructions told me to put into environment.rb does not seem to set any default locale. I knew that adding locales to the tests in the controllers would work, but I was not ready to start changing every

    get :index

to

    get :index, :locale => 'en'

in every controller test. Way too much work.

What seemed to work for me was to set up a default route. I changed this:

map.connect ':locale/:controller/:action/:id'

to

map.connect ':locale/:controller/:action/:id', :defaults => { :locale => Globalize::Locale.language ? Globalize::Locale.language.code : 'en'}

and all was fine again.

Moving to edge broke a bunch of stuff, especially a bunch of that code that I had managed to completely forget about.

It's like this. You've been rolling along fine on 1.2.3, and, for some reason, you go to the edge release.

       % rake rails:freeze:edge

Some stuff will break. So, here's what's going out, and how to replace it:

1. Un-initialized active_resource

   This one might happen early. The symptom:

   % ruby script/server
      Exiting
      /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/mongrel.rb:15: warning: already initialized constant OPTIONS
      /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/mongrel.rb:18: undefined method `options' for []:Array (NoMethodError)
   

   Weird. Something with mongrel? Let's try webrick:

   % ruby script/server webrick
     => Booting WEBrick...
      /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/webrick.rb:11: warning: already initialized constant OPTIONS
      /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- active_resource (MissingSourceFile)
   

   This seems to be because the original rake rails:freeze:edge done forgot to bring the activeresource library into the mix. The fix is quick:

     % svn export http://dev.rubyonrails.org/svn/rails/trunk/activeresource vendor/rails/activeresource
   

2. Cookies needed some new password.

   What is this? Turns out all you need to do for this is to add a little config item to the environment.rb Like this:

   config.action_controller.session = {:session_key => "_myapp_session", :secret => 'this_can_be_long'}
   

3. pagination

   Remember all that code the scaffolding used to spit out. Some of it, including the pagination, is still sitting around in a lot of places. It seems that pagination got a bad rap in the meantime, and it has now been taken out of edge. So, how do you replace them? The rails deprecations page is not exactly very helpful.

   Well, quite a few choices are around. Quite a few people seem to have stepped in to provide this lost functionality, and it's a matter of choosing the plugin. One is [will_paginate][willpaginate], another is paginatingfind, and yet another is the paginator gem.

   I did not have the time to test them all out (trying to move on, you know?), so I looked through a couple articles and discussions, and picked the will_paginate. Hardly any changes needed, and seems to work fine. Nice work.

4. Webservices

   Yep, webservices are still around, but edge is not all that interested in loading them. The initialization failed. Looking in the code, the webservices were not even in the rake:freeze list .

   Ok, so do it manually:

   svn export http://dev.rubyonrails.org/svn/rails/trunk/actionwebservice/ vendor/rails/actionwebservice -r7144
   

   Try again: nothing. Fine, google the thing. Looks like someone ran into this problem. He's got the fix all documented there.

5. push_with_attributes changes

   The push_with_attributes call is out. If you have a join table with any extra non-joining columns in it (and therefore not a 'pure' join), you cannot use the push_with_attributes call to save those values anymore.

   saving objects:

   Something that used to work in 1.2.3 was this: let's say you have you have some golf website, where you are keeping track of how far a player hits a ball with a given club.

   Your database would be setup something like this: table player (id, name), table clubs (id, name), and finally table player_club_performance (id, playerid, clubid, distance) to tie them together.

   player = Player.new(:name => 'Tiger')
   player.club_performance.create(:club_id => 7, :distance => 190)
   

   This seems to save the record (the player_club_performance row) immediately. Since player has not yet been saved, it does not have an id, assigned to it. Therefore, if you have any db constraints on that table, something like club_id not null (you should, it's safer that way), this will crash.

   If you use method build instead, it just creates the object. When you call p.save (the player), it will save the other associated objects as well.

   player = Player.new(:name => 'Me')
   player.club_performace.build(:club_id => 7, :distance => 300)
   player.save # saves player and club_performance
   

6. Sessions

   Don't know what happened to this one. I have always used the default file session, but this broke. First, I never noticed any files getting generated in the tmp/sessions directory. Second, on some actions where a bunch of stuff is saved in the session, I would get this:

   [2007-07-06 16:50:13] FATAL  DISPATCHER FAILSAFE RESPONSE (has cgi) Fri Jul 06 16:50:13 +0000 2007
   Status: 500 Internal Server Error
        CGI::Session::CookieStore::CookieOverflow             
          .../vendor/rails/actionpack/lib/action_controller/session/cookie_store.rb:91:in 'close'
   

   This one might be a bug in the edge. I tried setting it to CGI::Session::PStore (as per AWDWR 2, pg 440), but it seemed not to know what that was. It seems defined in the vendor/rails/actionpack/lib/actioncontroller/cgiext/session.rb, but did not pick it up. I looked around at the loading/initialization of this for a bit, the just switched to the database session, which worked fine the first time.

7. Other random stuff

   • assert_invalid_column_on_record Let's say that your Person model validates age, and says it's got to be a number or less than 1000 or something. It used to be okay to do something like this in a test

       post :create, :person => { :name => "elvis", age: => "10000"}
       assert_invalid_column_on_record "person", "age"
     

     That assert is out. The new way to do it is:

      post :create, :person => { :name => "elvis", age: => "asdfg"}
      assert(assigns(:person).errors.invalid?('age'))
     

Some plugins are plain good. I tend to forget where they are, and then have to go around hunting for them again. Here's so that I don't have to hunt around for these next time.

    ruby script/plugin install exception_notification
    ruby script/plugin install tztime
    ruby script/plugin install tzinfo_timezone
    ruby script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/
    ruby script/plugin install http://svn.pragprog.com/Public/plugins/annotate_models
    ruby script/plugin install svn://errtheblog.com/svn/plugins/will_paginate
    ruby script/plugin install svn://caboo.se/plugins/court3nay/spider_test
    ruby script/plugin install http://terralien.com/svn/projects/plugins/query_trace/

Also

    svn propset svn:ignore "*.log" log/
    svn propset svn:ignore "ruby_sess*" tmp/sessions/   
    svn propset svn:ignore "*" tmp/pids/
    svn propset svn:ignore "*" tmp/cache/
    svn propset svn:ignore "*" tmp/sockets/

My functional tests broke when I went to Rails 2.0. It took a while to grok the stack trace:

ArgumentError: wrong number of arguments (2 for 1)
..../vendor/rails/activerecord/lib/active_record/base.rb:1965:in `attributes_with_quotes'
    ..../vendor/rails/activerecord/lib/active_record/base.rb:1965:in `update_without_lock'

Turns out the problem was that in Rails 2.0, ActiveRecord's 'attributeswithquotes' method now takes 2 arguments instead of a single one. The problem is that globalize overrides this method in the db_translate.rb, so things break.

It's easy to fix, though. Adding the extra argument to globalize's db_translate.rb seems to do fix what hurts.

I knew that the rails pluralization did some cool fancy stuff, but never really had to mess with it until today. I was trying to duplicate some bug I ran into, so I created a brand new rails app to recreate the bug in there. I had to create some model, so created "Dive", and did the whole scaffold thing that goes along with it.

I ran the tests, and they then started to complain about "Dife" not being found.

Curious, I went to the console:

    % "dives".singularize   # => dife

Huh? What's a 'dife'? Even dictionary.com does not even know what a 'dife' is! No matter, I knew that you could fix this up in the environment.rb file. I went in there and uncommented that default block in there, and added my little contribution to rails' knowledge of the english language.

     Inflector.inflections do |inflect|
        inflect.singular /dives/i, 'dive'
     end

I then ran the console again, just to make sure. First thing I see are all these errors on the command line:

    ./script/../config/../config/environment.rb:55:NameError: uninitialized constant Inflector
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.5/lib/action_controller/assertions/selector_assertions.rb:525:NoMethodError:      undefined method `camelize' for "top":String
    ..../app/controllers/application.rb:4:NameError: uninitialized constant ActionController::Base

On the prompt, I try the 'dives'.singularize again, but only get an error.

    >> "dives".singularize
    NoMethod Error: undefined method `singularize' for "dives":String
      from (irb):2

I look in the awdwr book and some other places, and find that what I added really does seem to be correct. A little googling on the error finally takes me to someone else that ran into the same problem.

It's really simple. The Inflector block needs to be outside the whole 'Rails::Initializer.run' block, not inside. Once you move it to the right place, you get what you expect:

    >> "dive".pluralize
    => "dives"
    >> "dives".singularize
    => "dive"

I've got a few projects that I work on when I get the time. Since I usually work on all of them at the same time, it seems none of them moves forward very fast. I got curious to see how much work I am actually doing over time, and came up with a few little SVN hacks.

First, get the svn logs, pipe into a file:

% cd <head_of_the_svn_tree>
% svn log -q | egrep '^r' > activity.csv

Right, that gives us a file with all of the project checkins. The 'egrep' part strips out all of the annoying dashes that come with the svn log. The data looks like of like this:

r2 | danielw | 2006-12-20 00:38:13 +0200 (Wed, 20 Dec 2006)
r1 | danielw | 2006-12-20 00:33:41 +0200 (Wed, 20 Dec 2006)

Now, with some command-line tricks I can break down the activity a little more:

% svn log -q | egrep '^r' | cut -d '|' -f 2 | sort | uniq -c | sort -n

This breaks down the log and counts the number of checkins per person. You can point it to a URL as well. Results on one of my SVN trees gives something like this:

6  carl 
123  danielw

What I am really interested in is how this activity progresses over time. I don't know how to do this on the command line, but SQL could do this in no time. We need to create a database and a table to hold the data. In postgres, like this:

 % createdb work_activity
 % psql -d work_activity
 work_activity => create table svn_activity (revision varchar, who varchar, date timestamp);

Now we need to populate this with data. Since the end of that SVN line has got some funny timestamps, we'll get AWK to strip that out for us. Also, since the standard postgres column delimiter is the tab (\t), we'll delimit our records like that. Also, let's use the rails project to get more interesting stats.

% svn log -q http://svn.rubyonrails.org/rails/trunk > activity_rails.txt
% cat activity_rails.txt | egrep '^r' | awk '{print $1"\t"$3"\t"$5}' > activity_rails.data

This puts all of the data into a file, which we can now load into the DB in a single easy command:

% psql -d work_activity -c 'COPY svn_activity FROM STDIN' < activity_rails.data

Now it's all in the database, and we can do loads of fancy queries on it:

% psql -d work_activity -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1;"

     date_trunc      | count 
---------------------+-------
 2004-11-01 00:00:00 |    30
 2004-12-01 00:00:00 |   259
 2005-01-01 00:00:00 |   218
 2005-02-01 00:00:00 |   219
 2005-03-01 00:00:00 |   227
 2005-04-01 00:00:00 |   199
 2005-05-01 00:00:00 |    99
 2005-06-01 00:00:00 |   172
 2005-07-01 00:00:00 |   304
 2005-08-01 00:00:00 |    63
 2005-09-01 00:00:00 |   263
 2005-10-01 00:00:00 |   306
 2005-11-01 00:00:00 |   265
 2005-12-01 00:00:00 |    93
 2006-01-01 00:00:00 |    79
 2006-02-01 00:00:00 |   163
 2006-03-01 00:00:00 |   347
 2006-04-01 00:00:00 |   162
 2006-05-01 00:00:00 |    60
 2006-06-01 00:00:00 |   116
 2006-07-01 00:00:00 |    96
 2006-08-01 00:00:00 |   162
 2006-09-01 00:00:00 |   216
 2006-10-01 00:00:00 |   130
 2006-11-01 00:00:00 |   139
 2006-12-01 00:00:00 |    97
 2007-01-01 00:00:00 |   155
 2007-02-01 00:00:00 |    92
 2007-03-01 00:00:00 |   101
 2007-04-01 00:00:00 |    65
 2007-05-01 00:00:00 |   192
 2007-06-01 00:00:00 |   115
 2007-07-01 00:00:00 |    39
 2007-08-01 00:00:00 |    43
 2007-09-01 00:00:00 |   278
 2007-10-01 00:00:00 |   236
 2007-11-01 00:00:00 |   105

Looks like a very healthy project. Ok, let's find out on what day of the week rails developers have been most prolific:

psql -d work_activity -c "select extract(dow from date) as day, count(*) from svn_activity group by 1 order by 1;"  

 day | count 
-----+-------
   0 |  1040
   1 |   969
   2 |   874
   3 |   755
   4 |   790
   5 |   688
   6 |   789
(7 rows)

Day 0 is sunday! Thanks for the hard work, guys.

If you are using google maps in your apps, you get special key from them that works for the URL where you want to use the maps (ie www.yoursite.com). Since you probably test your code on localhost before you deploy it, you have a problem. The key they give you only works at that URL. What you need to do is get another key that works with your testing on your localhost, for example for localhost:3000.

What this means that when you deploy, you have to switch them. Of course, you do not want to do this by hand.

Here's a simple way to do this with capistrano:

    task :after_update_code, :roles => [:app, :db, :web] do

       layout_file = File.read('app/views/layouts/standard.rhtml')
       google_map_key = File.read("config/google_map_key.txt")
       layout_file.sub!(/file=api&v=2&key=(\w+)/, "file=api&v=2&key=#{google_map_key}")
       put(layout_file,
           "#{release_path}/app/views/layouts/standard.rhtml",
           :mode => 0444)
    end

So, what's going on?

First, save the production key on some file, like config/googlemapkey.txt. Don't think that is is like saving database passwords in SVN, since this key appears on all your webpages anyway.

I put my key in the site layout I use, standard.rhtml. By default, you want to keep development version of the key in there. When we deploy, we read the key from googlemapkey.txt, use a regexp to switch the development key with the production key. Then we send that file over with the put command.

You could be more clever by changing the file that's already on the server, instead of changing the local file and sending it over, but this works well enough.

I just noticed that google has published another cool API, this time for charts. It makes very fine looking charts, and has a simple API. Getting the values into the chart is very strange at first, but the constraints that you have on the values actually help in that your chart ends up looking better.

Here's how to use it. First we need some values to chart. A couple of weeks ago I wrote about how to extract some data out of subversion and put in into postgres. I still have those values around, so I for values I can use the number of commits per month in the rails project since it began. If you want to know more about that, read svn tricks. The strange part, as I mentioned before, is that you don't pass numbers to the chart. You pass some kind of encoding which has either 62, 1000, or 4096 values. For many uses, the first one is good enough.

Ok, so first we need to generate the encoding for the values. The Google maps API webpage shows a sample javascript function to generate that. I translated that to ruby. I think it's more or less correct, and goes like this:

    def simpleEncode(values, maxValue)
      simpleEncoding = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';

      chartData = ['s:'];
      values.each do |v|
        val = Float(v)
        if (!val.nan? && v >= 0)   
          chartData.push(simpleEncoding[( (simpleEncoding.length-1) * val / maxValue).round, 1]);
        else
          chartData.push('_');
        end
      end
      return chartData.join('');
    end

Right, so now how do we get our values to this function. As mentioned before, our values would come out of the database like this:

    % psql -d work_activity -t -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1;"
     2004-11-01 00:00:00 |    30
     2004-12-01 00:00:00 |   259
     2005-01-01 00:00:00 |   218
     etc
     ...

In order to generate the graph, we need to stuff those values into the funny URL google wants. So we hack a little chunk of code to parse the stuff.

    a = STDIN.read
    values = []
    dates = []
    a.each do |line|
      next if line.chomp.length < 1
      date, value = line.split('|')
      value.chomp!; date.chomp! #get rid of all the chars
      values << value.to_i 
      dates << date.split[0].split('-')[1] # just take month, legend too busy otherwise
    end

That gives us two arrays, one with the values, and one with the dates. Notice that I chopped off the year and the day off the dates. Before I did that, the labels on the X axis were so busy with text that you could not even make sense out of any of them.

Now that we have our values, we make the img tag with all the funny values in it. We also decide the size of the chart and the 'alt' value here. There are plenty of other things that you can modify, but this does most of what you want:

    image_str =<<-START
    <img src="http://chart.apis.google.com/chart?
    chs=500x225
    &amp;chd=#{simpleEncode(values, values.max)}
    &amp;cht=lc
    &amp;chxt=x,y
    &amp;chxl=0:|#{dates.join('|')}|1:||#{values.max}"
    alt="Rails Subversion Commits" />
    START
    puts image_str

Now we put all those bits of code above into a single file, and call it makegoogleurl.rb. We run the query as before, and pipe it to our nifty new program like this:

    psql -d work_activity -t -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1"| ruby make_google_url.rb

This is our result:

    <img src="http://chart.apis.google.com/chart?
    chs=500x225
    &amp;chd=s:FummojRe1Lu2vQOd9cLURcmXYRbQSLiUHIxpS
    &amp;cht=lc
    &amp;chxt=x,y
    &amp;chxl=0:|11|12|01|02|03|04|05|06|07|08|09|10|11|12|01|02|03|04|05|06|07|08|09|10|11|12|01|02|03|04|05|06|07|08|09|10|11|1:||347"
    alt="Rails Subversion Commits" />

And if you put that in your page, it looks like:

I bundled the code from the previous post into a plugin, called Chartr. This makes it easy to create graphs within rails.

Here's how to use it.

First, install the plugin. (It's probably a good idea to install it with '-x' since it's likely to be updated. Also, I should mention that this is my first plugin.)

    ruby script/plugin install -x svn://syvera.com/plugins/chartr

Right, so now we need to graph something. You can put a graph into any page, but let's create a page that definitely deserves a graph:

    ruby script/generate controller graphs index

This gives you an index.html.erb page where you can put your graph. In that page, we're going to use Chartr to give us a random graph. This is the code:

    <%= Chartr.make_simple_line_chart Array.new(5) {|i| rand(10)}, 
                                      ['one', 'two', 'three', 'four', 'five'],
                                      'stuff to graph' %>

So that's going to give you a graph like this:

Sure, that's cool. But what the heck were those arguments?

The first one is for your values:

    Array.new(5) {|i| rand(10)}  # returns an array of 5 random values between 0 and 9

The second, also an array, is for the legend. No explanation needed for that. The last is for the html 'alt' tag.

You could provide a fourth one for the size of the graph. If not, the default is 200x100.

A problem that I recently started running into is a strange Postgres FK constraints error, which only shows up when running tests. The error looks something like this:

PGError: ERROR:  permission denied: "RI_ConstraintTrigger_XXXXX" is a system trigger

This is something that came in with FoxyFixtures, and there's a discussion about this problem and a patch in the rails trac. So what's this about?

When loading fixtures, rails tries to disable all foreign keys so that there are no problems inserting the data in whatever order. The calls work like this:

ALTER TABLE table_name DISABLE TRIGGER ALL
insert a bunch of fixtures 
ALTER TABLE table_name ENABLE TRIGGER ALL

However, postgres keeps some kind of metadata regarding the FKs to be enforced called RI_Constraint_something. Disabling the triggers on a table causes these to be deleted as well. The problems is that these other triggers belong to the superuser and not you. So, you got this permissions problem.

You may not have superuser permissions for your postgres instance, but it's probably best that the database you work with does not have superuser permissions, since that's the way it's going to be in production.

One workaround for to disable the foreign key checking in a different way. Instead of running the DISABLE TRIGGER, you can make the constraints DEFERRED postgres docs. This way, you run like this:

SET CONSTRAINTS ALL DEFERRED
insert a bunch of fixtures 
SET CONSTRAINTS ALL IMMEDIATE

This allows you to load your fixtures, but keep all contraint-checking.

The easiest way to patch this is to override the behavior of the disable_referential_integrity method. I created a new file, active_record_fk_hack.rb, stuck this in there:

module ActiveRecord
  module ConnectionAdapters
    class PostgreSQLAdapter < AbstractAdapter
      def disable_referential_integrity(&block)
         transaction {
           begin
             execute "SET CONSTRAINTS ALL DEFERRED"
             yield
           ensure
             execute "SET CONSTRAINTS ALL IMMEDIATE"
           end
         }
      end
    end
  end
end

then, in your environment.rb, add this at the end:

require 'active_record_fk_hack'

I recently tried to start using metric-fu for code statistics. This makes use of the rcov gem, as well as a few other gems. It generates the statistics with the rake task rake metrics:all_with_migrate. Somehow, this caused a slew of errors in the functional tests, all looking like this:

NoMethodError: undefined method `to_plain_segments' for #<ActionController::Routing::RouteSet:0x2aaaacad5648>

On the Mac, everything worked fine. On Fedora 6 hosts running 1.8.5 or 1.8.6 I would get the error. A bit of googling found that some rspec people are getting it too, but it's not rspec's fault. It's rcov.

I updated my rcov to the yet-unofficially unreleased version 0.8.3.1, thanks to a github branch of rcov.

sudo gem install spicycode-rcov -s http://gems.github.com

That fixes it, and tests run fine now.