Rorra's blog » Ruby

Updating sitemap_generator to upload your files to s3 with aws-s3 and aws-sdk

admin — Mon, 30 Jan 2012 07:35:15 +0000

Recently I had to do a job where I was suppose to either find a gem that was able to generate sitemaps in rails 3 and upload it to S3, or update the current behavior of the gem sitemap_generator (which is a great gem), so I ended updated the behavior of the gem.

The gem already supports uploads to S3, but the problem is that it ask you to install carrierwave, and if you are already using paperclip, aws-s3 and aws-sdk, that’s not the best way to go, so what I did was to enhance the rake tasks of the gem in order to add behavior for the S3 upload and notification to the different search engines as well.

You can check my code in this here: https://gist.github.com/1693860

Something important is to add this line of code to your file config/sitemap.rb

SitemapGenerator::Sitemap.public_path = File.join(Rails.root, "public", "system", "sitemaps").to_s

And here’s the code

require 'aws'

class Rake::Task
  def replace &block
    @actions.clear
    prerequisites.clear
    enhance &block
  end
end

namespace 'sitemap' do
  desc 'Upload the sitemap files to S3 (using your configuration in config/s3.yml)'
  task :upload_to_s3 => :environment do
    if File.exist?(File.join(Rails.root, "config", "s3.yml"))

      # Load credentials
      s3_options = YAML.load_file(File.join(Rails.root, "config", "s3.yml"))[Rails.env].symbolize_keys
      bucket_name = s3_options[:bucket]
      s3_options.delete(:bucket)

      # Establish S3 connection
      AWS.config(s3_options)

      Dir.entries(File.join(Rails.root, "public", "system", "sitemaps")).each do |file_name|
        next if ['.', '..'].include? file_name
        path = "sitemaps/#{file_name}"
        file = File.join(Rails.root, "public", "system", "sitemaps", file_name)

        begin
          s3 = AWS::S3.new
          bucket = s3.buckets.create(bucket_name)

          object = bucket.objects[path]
          object.write(:file => file)
        rescue Exception => e
          raise
        end
        puts "Saved #{file_name} to S3"
      end
    end
  end
end

Rake::Task["sitemap:create"].enhance do
  Rake::Task["sitemap:upload_to_s3"].invoke
end

Rake::Task[:'sitemap:refresh'].replace do
  if File.exist?(File.join(Rails.root, "config", "s3.yml"))
    s3_options = YAML.load_file(File.join(Rails.root, "config", "s3.yml"))[Rails.env].symbolize_keys
    bucket_name = s3_options[:bucket]
    SitemapGenerator::Sitemap.ping_search_engines(:sitemap_index_url => "https://#{bucket_name}.s3.amazonaws.com/sitemaps/sitemap_index.xml.gz")
  else
    SitemapGenerator::Sitemap.ping_search_engines
  end
end

Rails tickets moved from lighthouse to github

admin — Mon, 09 May 2011 05:22:51 +0000

Until now, all the tickets created for ruby on rails were being reported on lighthouse <https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/overview>.

But now, the tickets were migrated to github and you should use github in order to report tickets on ruby on rails <https://github.com/rails/rails/issues>.

Ruby on Rails 2.3.x video tutorial

admin — Tue, 21 Sep 2010 14:02:48 +0000

While attending to the pcmasmas.com meeting, I gave an advance introduction to ruby on rails 2.3.x with haml, sass, authlogic, friendly_id and activescaffold.

I prepared a set of videos in order to introduce other developers to web development with ruby on rails.

You can find all those videos (without audio) here:

Video 1 of 10: Creating the rails application
Video 2 of 10: Setting up git
Video 3 of 10: Starting the layout with haml and sass
Video 4 of 10: Implementing authentication with authlogic
Video 5 of 10: Creating the category and product models
Video 6 of 10: Starting the administration with activescaffold
Video 7 of 10: Browsing categories and products, part 1 of 2
Video 8 of 10: Browsing categories and products, part 2 of 2
Video 9 of 10: Better navigation with friendly_id
Video 10 of 10: Improving the administration and updating it to integrate friendly_id updates

Rails 3 released, bundler 1.0 released!!!!

admin — Mon, 30 Aug 2010 13:44:30 +0000

Rails 3.0 has been underway for a good two years, so it’s with immense pleasure that we can declare it’s finally here. We’ve brought the work of more than 1,600 contributors together to make everything better, faster, cleaner, and more beautiful.

David Heinemeier Hansson

So at this moment, if you run a gem install rails, you will notice that it install the version 3 Why? Because it’s stable (or it should be (A)).

Where to start?

You should start by checking the “rails 3 screencasts” that you can find on the rails website. It’s a great introduction to see the news on the rails 3 framework. As usually, you can find good videos on railscasts about rails 3, how to migrate to rails 3, etc.

You can find all the improvements on the official website, but just to summarize them, in rails 3 you will find:

New Active Record query engine
New router for Action Controller
New Action Mailer
Manage dependencies with Bundler
XSS protection by default
Ruby 1.9 Encoding support
Active Model: Validations, callbacks, etc for all models, so you can use other ORMs with rails in a easy way
Official plugin APIs (railties)
Rewritten internals
Agnosticism with jQuery, rSpec, and Data Mapper
Documentation

Rails 3, mysql and UTF-8

admin — Fri, 30 Jul 2010 18:01:52 +0000

So, have you started to work with rails 3? Did you realize that your UTF-8 databases doesn’t work nice with mysql? This is because the mysql gem works with ASCII-8BIT encoding, but ruby 1.9 and rails 3 works with UTF-8 encoding, so when you create a model into the database, everything works fine, but when you work with those models, it doesn’t works so nice and you don’t get what you stored.
What’s the solution? Well, there are actually three solutions, the most recommended, use mysql2, in order to do this, edit your Gemfile and include:

gem "mysql2"

and then, edit your databases.yml file, and change the adapter to mysql2

development:
  adapter: mysql2
  database: fun_development
  user: root
  password:
  encoding: utf8

Another solution, is just instead of using the “mysql” gem, use the “ruby-mysql” gem, but it’s pretty slow because it’s a 100% ruby gem.

gem "ruby-mysql"

The last solution is a monkey patch, but its also slow to use it, so I really recommend using the mysql 2 gem

require 'mysql'

class Mysql::Result
  def encode(value, encoding = "utf-8")
    String === value ? value.force_encoding(encoding) : value
  end

  def each_utf8(&block)
    each_orig do |row|
      yield row.map {|col| encode(col) }
    end
  end
  alias each_orig each
  alias each each_utf8

  def each_hash_utf8(&block)
    each_hash_orig do |row|
      row.each {|k, v| row[k] = encode(v) }
      yield(row)
    end
  end
  alias each_hash_orig each_hash
  alias each_hash each_hash_utf8
end

Contributing to ruby on rails source code is so easy

admin — Thu, 29 Jul 2010 16:59:36 +0000

I finally made my first contribution to rails, although it’s not a big fix nor anything like that, is a way to get started.

If you have read my last posts, you will see how to install and works with rails edge, but now I’ll explain how to setup a good rails environment to live in the edge and to help ruby on rails community by checking the issues posted on lighthouseapp.com to see if they are a real bug or not, to create patches for bugs, or why not, to create a new functionality for the rails framework.

Supposing that you already installed the last version of rails, you will need to get the rails code checked out from github, and make your demo project to use that version, so you can modify that version to create fixes, to test fixes, etc.

git checkout http://github.com/rails/rails.git

Now that you just create an application for doing test/development

rails new my_app

Edit the Gemfile of my_app (my_app/Gemfile) to use the rails code you are going to work with, by giving the rails gem a parameter :path where your code of ruby on rails is installed

gem 'rails', :path => File.expand_path('../../rails',  __FILE__)

And that’s all, now you can follow this guide to know how to work with git in order to post valid patches.

Rails 3.0 RC Released!!!!

admin — Tue, 27 Jul 2010 15:00:13 +0000

So finally, rails 3.0 RC was released, cool news because rails 3.0 brings a lot of updates and it will be very fun to develop for rails 3.0. I wonder how much work would be out there migrating rails 2.x to rails 3.x versión.
Rails 3.0 RC is out just after 21 hours bunder 1.0.0.RC.1 was released, was it really a coincidence? Yeah sure…
So how to start playing with rails 3.0.RC? Pretty easy, first, install rvm, after that, lets install ruby 1.9.2 RC 2

rvm install 1.9.2

Now let’s use ruby 1.9.2

rvm 1.9.2

Then we will install bundler, because we are using rvm, we won’t use sudo at anytime

gem install bundler -v 1.0.0.rc.1

And now we can install rails rc 1

gem install rails --pre

We can start playing with rails 3.0

rails app my_app_name
cd my_app_name
bundle install

But finally, instead of locking your development on rails 3.0.RC 1, work with the latest version of rails that is being developed (because we will find bugs in the RC), edit the Gemfile inside your application, and do these modifications

#gem 'rails', '3.0.0.rc'

# Bundle edge Rails instead:
gem 'rails', :git => 'git://github.com/rails/rails.git'

Now update your app to work with the latest version of rails

bundle install

And that’s all
You can report any bugs you find on https://rails.lighthouseapp.com/projects/8994-ruby-on-rails, and also check the bugs there, reproduce them and confirm that is really a bug or not, and don’t forget, you are also testing bundler 1.0.0.RC.1, so if you find any issue with bundler, you can report your issues on http://github.com/carlhuda/bundler/issues.
Remember, this is your opportunity to give something back to the open source community.

Installing RubyDebug with ruby 1.9.2 and rails 3 to use RubyMine

admin — Sat, 24 Jul 2010 20:26:55 +0000

So you are trying to install ruby-debug to debug rails applications (or perhaps ruby-debug-ide for RubyMine) and you are having problems with it, because you have ruby 1.8.7 for your system, and you are using rvm in order to user ruby 1.9.2 to test rails edge.

The first thing to notice, is that all of the sources that you have installed with rvm are located in $rvm_path/src, so in case you are using ruby 1.9.2 rc 2 as I’m doing right now, the installation is pretty easy

gem install ruby-debug-ide19 -- --with-ruby-include=$rvm_path/src/ruby-1.9.2-rc2/

But for sure, if you are using ruby 1.9.1 or any other version, you can easily do a ls $rvm_path/src to find the right location to compile ruby-debug-ide19 into your system.

Now, you can add the gem to the Gemfile if you are planning to debug from the console

gem 'ruby-debug19', :require => 'ruby-debug', :group => :development

And then run

rails s --debug

Or if you are using rubymine, the debug will start although it’s not working yet, hopefully it will be working before rubymine 2.3.5 is released, because nobody wants to use rails3 without ruby 1.9.2, or at least I don’t want to do that

Integrating radiant 0.9, globalize and GeoIP

admin — Sun, 10 Jan 2010 09:28:59 +0000

After downloading and using radiant 0.9 and globalize, I realized that I wanted to redirect the users by default with their geo location.
So I installed GeoIP and then I had to do these little modifications to the Globalize plugin.
First, I edit the file config/environment.rb and added this line, in order to load GeoIP with the application

config.gem 'geoip'

Then I updated the file vendor/extensions/globalize2/lib/globalize2/application_controller_extensions.rb

module Globalize2
  module SiteControllerExtensions
    def self.included(base)
      base.class_eval do
        alias_method_chain :find_page, :globalize

        alias_method_chain :show_page, :homepage_redir
      end
    end

    def show_page_with_homepage_redir
      url = params[:url]
      if Array === url
        url = url.join('/')
      else
        url = url.to_s
      end

      if url == '/'
        locale = params[:locale] || cookies[:locale] || session[:locale] || Globalize2Extension.ip_lookup(request.remote_ip)
        redirect_to CGI.unescape('/' + locale + '/') and return
      end

      show_page_without_homepage_redir
    end

    def find_page_with_globalize(url)
      globalized_url = '/' + I18n.locale + '/' + url
      find_page_without_globalize(globalized_url)
    end
  end
end

So, as you can see on the code, I also check if the user has the language selected in a cookie, and when the user selects (or get a locale by default), I set the cookie with the locale, so the next time the user came into the site, he will see the last language selected.
Finally, I updated the file vendor/extensions/globalize2/globalize2_extension.rb and added the following method, where I get the country code from the user, and I set the locale based on his country ip, or the default locale.

  def ip_lookup(ip)
    g = GeoIP.new( File.join(File.dirname(__FILE__), '..', '..', 'gems', 'geoip-0.8.6', 'GeoIP.dat'))
    r = g.country(ip)
    country_code = r[3]
    case country_code
      when 'DE'
        'de'
      when 'GB'
        'uk'
      when 'AU'
        'au'
      when 'FR'
        'fr'
      when ' US'
        'us'
      else
        'us'
    end
  end

And finally, we need to tell Radiant to load the GeoIP library, so in the same file, we add

  extension_config do |config|
    config.gem 'geoip', :source => 'http://github.com'
  end

So that’s all, now you have GeoIP working with globalize and Radiant 0.9

Radiant & Globalize: Redirecting the user to their language site

admin — Sat, 09 Jan 2010 09:30:53 +0000

After installing Radiant 0.9 and Globalize, I realized I had to redirect the user to their language site.

So, if the user has been assigned to the locale en, when he vist http://mysite.com he should be redirected to http://mysite.com/en

Why? Because I had some files under images with locales, and I had some flash issues that were not going to work from the homepage address.

So I ended updating the file vendor/extensions/globalize2/lib/globalize2/site_controller_extensions.rb

module Globalize2
  module SiteControllerExtensions
    def self.included(base)
      base.class_eval do
        alias_method_chain :find_page, :globalize

        alias_method  riginal_show_page, :show_page
        alias_method :show_page, :check_homepage_redir

      end
    end

    def check_homepage_redir
      url = params[:url]
      if Array === url
        url = url.join('/')
      else
        url = url.to_s
      end

      if url == '/'
        locale = params[:locale] || cookies[:locale] || session[:locale] || Globalize2Extension.ip_lookup(request.remote_ip)
        redirect_to CGI.unescape('/' + locale + '/') and return
      end

      original_show_page
    end

    def find_page_with_globalize(url)
      globalized_url = '/' + I18n.locale + '/' + url
      find_page_without_globalize(globalized_url)
    end
  end
end