Category Archives: News

Episode #521 – January 13th, 2015

This episode covers Ruby methods you should be using, code metrics that matter, using ES6 on the Rails asset pipeline, tampering with network connections, automatically prefixing email subject lines and phone verification using SMS.

Listen to this episode on Ruby5

Useful Methods

Ben Lewis wrote an article about Five Ruby Methods you should be using. He covers Object#tap, Array#bsearch, Enumerable#flat_map, Array.new with a block, and the sort operator (a.k.a. spaceship).

Metrics That Matter

Chris Kottom published an article covering a few of the code metrics that he feels don’t necessarily matter, like 100% code coverage. He focuses on a few that do, including performance, complexity, and security.

sprockets-traceur

The sprockets-traceur gem by Adam Pohorecki allows us to use the new features from ECMAScript 6 in Rails apps today, before the next version of JavaScript is officially released. Since a lot of Rails apps tend to make heavy use of JavaScript, it’s good to be able to try out some of these features sooner than later.

Toxiproxy

The team at Shopify recently released a new library for tampering with network connections called Toxiproxy. It provides a TCP proxy system which can be used to simulate network and system conditions.

email_prefixer

The email_prefixer gem by Ryan Sonnek automatically adds a prefix to the subject line of all delivered emails in development and staging. Great use of ActionMailer::Interceptor, a feature added in Rails 3.1.

Phone verification using SMS via Twilio

Santosh Wadghule published a guide on how to perform phone number verification in your Ruby application using the Twilio API. Who needs CAPTCHA? I’m just gonna TEXTYA!

Episode #520 – January 9th, 2015

Rails Misconceptions, performance and minecraft

Listen to this episode on Ruby5

Sponsored by NewRelic
most popular posts for 2014

Why Do They Say Rails Doesn’t Scale?
Why Do They Say Rails Doesn’t Scale?Noah Gibbs posted 4 reasons
…

Read more at the source

Episode #519 – January 6th, 2015

The holidays brought us Ruby 2.2.0 and Rails 4.2.0, Finishing Moves, the end/beginning of RubySpec, a nifty way to bind data from JavaScript to Rails, tips on testing controllers, and the tragic news of James Golick’s death.

Listen to this episode on Ruby5

Ruby 2.2.0 Released

Ruby 2.2.0 was released on Christmas day, containing a bunch of new features and improvements. The new garbage collector is now able to collect Symbol type objects, reducing memory usage of Symbols. This new version has reduced pause time for garbage collection thanks to the Incremental Garbage Collector. Rails 5.0 will take full advantage of both of these features, when it eventually comes out sometime in the Fall of 2015, requiring Ruby 2.2 or greater.

Rails 4.2.0

Rails 4.2.0 was also released over the holidays. There’s AdequateRecord, the painstaking multi-year rewrite of ActiveRecord led by Aaron Patterson to improve the performance of ActiveRecord beyond what it was in Rails 2.3 days, which means a lot.There’s also ActiveJob, which was originally planned for Rails 4. It brings a centralized interface for queuing jobs for any queueing systems like Resque, Sidekiq or Delayed Job. There’s also Web Console which gives you the ability to run queries you would usually run in the rails console, but right inside the browser when an exception occurs, which is similar to the better_errors gem. And now, adding and removing foreign keys is properly supported in Rails migrations.

Finishing Moves

Every now and then you find yourself in a situation where Ruby doesn’t have that one method you need. For example, a nil_chain method, which allows you to write method chains without fear of tripping over a NoMethodError and NameError exception when something throws out a nil value. Sometimes you just want to call something if it exists, and not if it’s nil. The folks over at Forge Software wrapped up a bunch of these useful methods into a gem called “Finishing Moves”.

RubySpec

On new year’s eve Ruby Hero Brian Shirai wrote a blog post explaining why he is ending the RubySpec project. RubySpec is a set of specifications, or tests, on how Ruby is supposed to run that was taken out of the Rubinius project. The objective was to have a common set of tests, not written in C, that could be run against any Ruby implementation to see if it had all the right behavior for Ruby. The reason why RubySpec is no longer is quite contentious but it seems like collaboration on the project wasn’t optimal between Ruby implementation. There was already a fork from people working on the MRI implementation that tackles some of the issues with RubySpec itself. At the very least we can, we can all agree that knowing how Ruby should behave matters.

Broader discussion available at: https://gist.github.com/nateberkopec/11dbcf0ee7f2c08450ea

Databound

Domas Bitvinskas dropped us a line yesterday to let us know about Databound, a Ruby gem that provides JavaScript a simple API to the Ruby on Rails CRUD. Basically it provides you a simple way to write ActiveRecord like functions directly from your JavaScript. So you can do things like User.find(1) from your JavaScript methods, and Databound will take care of calling into your Rails app to find the user with an ID of 1. You don’t even need a controller for the models you want to be able to query for, if you don’t already have one. All you need to do is write a line in your routes file specifying the model you want to be able to query for, and the columns you’re permitted to set from your JavaScript calls. This could be really useful if you want to write more code on the front-end, and less on the back-end.

How to test controllers in Rails

Occasionally it’s nice to cover a beginner tutorial, and we’d like to mention Cezar Halmagean’s tutorial on testing controllers in Rails. Controllers can be pretty easy to test assuming you know how to stub out calls and set expectations properly. Otherwise they can mimic your integration tests, which is not very useful.

Remembering James Golick

Thanks for everything you did James.

Episode #518 – December 19, 2014

Eastward Ho, Git ours/theirs and where does your code go?

Listen to this episode on Ruby5

Read more at the source

Episode #517 – December 16, 2014

This week we have Streem, Rails 4.2.rc3, Papercrop, and RubyMotion 3.0.

Listen to this episode on Ruby5

Streem

Ruby’s creator is working on a new stream based language. It’s in its early days but is clearly influenced by Ruby.
It is a concurrent scripting language based on a programming model similar to shell, with influences from Ruby and other functional programming languages. Currently, it’s still in the design stages.

Rails 4.2.0 Release Candidate

The Rails team has just released the third Release Candidate of Rails 4.2 today. Major changes include – Active Jobs, Asychnonous mails, Adequate record, web console, and foreign key support. So far, there haven’t been any major regressions for this release, meaning that we are still on track for our target to release the final version of 4.2.0 before Christmas time.

Papercrop

Papercrop is a gem from rsantamaria. It’s an easy extension for Paperclip to crop your image uploads using jCrop. It’s quite simple to use – there are 2 form helpers (cropbox and crop_preview). Cropbox places an adjustable crop area over the image being cropped and crop_preview renders a small preview of your finished image. Just a note – make sure you allow for additional params when using strong parameters

RubyMotion 3.0

Last Friday, RubyMotion announced the official release of version 3.0. Since the release of the beta, they’ve been hard at work adding some updates. Most notably, they now support all versions of Android. There’s a bunch of new features too. A debugger interface and a spec framework have been added but it doesn’t stop there. 3.0 let’s you write Apple Watch apps with Watchkit, create iOS 8 integrations, iOS 64-bit support, and an overall application size reduction.

Episode #516 – December 9, 2014

This week we have a jRuby security release, a new Command API for ROM, Traveling Ruby, early validations, easy rewrites with Ruby and Science!, and a Rails Camp not too far from Hobbiton.

Read more at the source

Episode #515 – December 5th, 2014

RubyConf 2014 on Confreaks, browser geolocation with Spyme, referential integrity with foreign keys, forwarding messages with tell, and free SecCasts all in this episode of the Ruby5.

Read more at the source

Episode #514 – December 2, 2014

In this episode we yap about Rails 4.2 RC1, Opal, Deployment, Interpol, Sportdb, Service Objects, and Emails.

Read more at the source

Episode #513 – November 25th, 2014

This episode covers the future of Ruby 3.0, the flip flop operator, a gem to ferry your data away, speeding up your Capybara specs, and a new place to find Ruby meet up resources.

Read more at the source

Episode #512 – November 21st, 2014

RubyConf 2014 wrap-up, accelerating your Rails app, RubyGems.org gets redesigned, Heroku wants you to try HTTP Git, and new versions of Rails were released

Read more at the source

Forklift: Moving Big Databases Around

April 12, 2013

What?

Forklift is a ruby gem that can help you collect, augment, and save copies of your mySQL databases. This is often called an “ETL” tool as the steps in this process mirror the actions of “Extracting the data,” “Transforming the data,” and finally “Loading the data” into its final place.

With Forklift, you create a Plan which describes how to manipulate your data. The process for this involves (at least) three databases:

Live Set
Working Database
Final Database

The “Live Set” is first loaded into the “Working Set” to create a copy of your production data we can manipulate without fear of breaking replication. Then, any transformations/manipulations are run on the data in the working set. This might include normalizing or cleaning up data which was great for production but hard for analysts to use. Finally, when all of your transformations are complete, that data is loaded into the final database.

Forklift is appropriate to use by itself or integrated within a larger project. Forklift aims to be as fast as can be by using native mySQL copy commands and eschewing all ORMs and other RAM hogs.

Features

Can extract data from both local and remote databases
Can perform integrity checks on your source data to determine if this run of Forklift should be executed
Can run each Extract either each run or at a frequency
Can run each Transform either each run or at a frequency
Data kept in the woking database after each run to be used on subsequent transformations
Only ETL’d tables will be copied into the final database, leaving other tables untouched
Emails sent on errors

What does TaskRabbit use this for?

At TaskRabbit, the website you see at www.taskrabbit.com is actually made up of many smaller rails applications. When analyzing our site, we need to collect all of this data into one place so we can easily join across it.

We replicate all of our databases into one server in our office, and then use Forklift to extract the data we want into a common place. This gives us the option to both look at live data and to have a more accessible transformed set which we create on a rolling basis. Our “Forklift Loop” also git-pulls to check for any new transformations before each run.

Example Annotated Plan

In Forklift, you build a plan. You can add any action to the plan in any order before you run it. You can have 0 or many actions of each type.

require 'rubygems'
require 'bundler'
Bundler.require(:default)
require 'forklift/forklift' # Be sure to have installed the gem!

#########
# SETUP #
#########

forklift = Forklift::Plan.new({
  
  :local_connection => {
    :host => "localhost",
    :username => "root",
    :password => nil,
  },
  
  :remote_connections => [
    {
      :name => "remote_connection_a",
      :host => "192.168.0.0",
      :username => "XXX",
      :password => "XXX",
    },
    {
      :name => "remote_connection_b",
      :host => "192.168.0.1",
      :username => "XXX",
      :password => "XXX",
    },
  ],

  :final_database => "FINAL",
  :working_database => "WORKING",

  :do_dump? => true,
  :dump_file => "/data/backups/dump-#{Time.new}.sql.gz",

  :do_email? => true,
  :email_to => ['XXX'],
  :email_options => { 
    :via => :smtp, 
    :via_options => {
      :address              => 'smtp.gmail.com',
      :port                 => '587',
      :enable_starttls_auto => true,
      :user_name            => "XXX",
      :password             => "XXX",
      :authentication       => :plain,
    } 
  }

})

##########
# CHECKS #
##########

forklift.check_local_source({
  :name => 'CHECK_FOR_NEW_DATA',
  :database => 'test',
  :query => 'select (select max(created_at) from new_table) > (select date_sub(NOW(), interval 1 day))',
  :expected => '1'
})

forklift.check_remote_source({
  :connection_name => "remote_connection_b",
  :name => 'ANOTHER_CHECK',
  :database => 'stuff',
  :query => 'select count(1) from people',
  :expected => '100'
})

###########
# EXTRACT #
###########

forklift.import_local_database({
  :database => "database_1",
  :prefix => false,
  :frequency => 24 * 60 * 60,
})

forklift.import_local_database({
  :database => "database_2",
  :prefix => false,
  :only => ['table_1', 'table_2'],
})

forklift.import_remote_database({
  :connection_name => 'remote_connection_a',
  :database => "database_3",
  :prefix => true,
  :skip => ['schema_migrations']
})

#############
# TRANSFORM #
#############

transformation_base = File.dirname(__FILE__) + "/transformations"

forklift.transform_sql({
  :file => "#{transformation_base}/calendars/create_calendars.sql",
  :frequency => 24 * 60 * 60,
})

forklift.transform_ruby({
  :file => "#{transformation_base}/test/test.rb",
})

#######
# RUN #
#######

forklift.run

Workflow

def run
  lock_pidfile                # Ensure that only one instance of Forklift is running
  rebuild_working_database    # Ensure that the working database exists
  ensure_forklift_data_table  # Ensure that the metadata table for forklift exists (used for frequency calculations)
  
  run_checks                  # Preform any data integrity checks
  run_extractions             # Extact data from the life databases into the working database
  run_transformations         # Perform any transformations
  run_load                    # Load the manipulated data into the final database
  
  save_dump                   # mySQLdump the new final database for safe keeping
  send_email                  # Email folks the status of this forklift
  unlock_pidfile              # Clean up the pidfile so I can run next time
end

Transformations

Forklift allows you to create both Ruby transformations and SQL transformations

Ruby Transformations

SQL Transformations are kept in a file ending in .rb
Ruby Transformations should define a class which matches the name of the file (IE: class MyTransformation would be in a file called my_transformation.rb
logger.log(message) is the best way to log but logger.debug is also available
database is a string containing the name of the working database
connection is an instance of Forklift::Connection and connection.connection is a raw mysql2 connection
Classes need to define a transform(connection, database, logger) IE:

class Test

  def transform(connection, database, logger)
    logger.log "Running on DB: #{database}"
    logger.log "Counting users..."
    connection.q("USE `#{database}`")
    users_count = connection.q("count(1) as 'users_count' from `users`")
    logger.log("There were #{users_count} users")
  end

end

SQL Transformations

SQL Transformations are kept in a file ending in .sql
You can have many SQL statements per file
SQL will be executed linearly as it is written in the file

SQL Transformations can be used to generate new tables like this as well

Defaults

The defaults for a new Forklift::Plan are:

{
   :project_root => Dir.pwd,
   :lock_with_pid? => true,

   :final_database => {},
   :local_database => {},
   :forklift_data_table => '_forklift',
   
   :verbose? => true,

   :do_checks? => true,
   :do_extract? => true,
   :do_transform? => true,
   :do_load? => true,
   :do_email? => false,
   :do_dump? => false,
 }

Methods

Test

forklift.check_local_source({
  :name => STRING,     # A name for the test
  :database => STRING, # The Database to test
  :query => STRING,    # The Query to Run.  Needs to return only 1 row with 1 value
  :expected => STRING  # The response to compare against
})

forklift.check_remote_source({
  :connection_name => STRING,  # The name of the remote_connection
  :name => STRING,             # A name for the test
  :database => STRING,         # The Database to test
  :query => STRING,            # The Query to Run.  Needs to return only 1 row with 1 value
  :expected => STRING          # The response to compare against
})

Extract

forklift.import_local_database({
  :database => STRING,              # The Database to Extract
  :prefix => BOOLEAN,               # Should we prefix the names of all tables in this database when imported wight the database?
  :frequency => INTEGER (seconds),  # How often should we import this database?
  :skip => ARRAY OR STRINGS          # A list of tables to ignore and not import
  :only => ARRAY OR STRINGS          # A list of tables to ignore and not import (use :only or :skip, not both)
})

forklift.import_remote_database({
  :connection_name => STRING,       # The name of the remote_connection
  :database => STRING,              # The Database to Extract
  :prefix => BOOLEAN,               # Should we prefix the names of all tables in this database when imported wight the database?
  :frequency => INTEGER (seconds),  # How often should we import this database?
  :skip => ARRAY OR STRINGS          # A list of tables to ignore and not import
  :only => ARRAY OR STRINGS          # A list of tables to ignore and not import (use :only or :skip, not both)
})

Transform

forklift.transform_sql({
  :file => STRING,                 # The transformation file to run
  :frequency => INTEGER (seconds), # How often should we run this transformation?
})

forklift.transform_ruby({
  :file => STRING,                 # The transformation file to run
  :frequency => INTEGER (seconds), # How often should we run this transformation?
})

Debug

You can launch forklift in “debug mode” with --debug (we check ARGV["--debug"] and ARGV["-debug"]). In debug mode the following will happen: - verbose = true - no SQL will be run (extract, load) - no transforms will be run - no email will be sent - no mySQL dumps will be created

Options & Notes

email_options is a hash consumed by the Pony mail gem
Forklift’s logger is Lumberjack with a wrapper to also echo the log lines to stdout and save them to an array to be accessed later by the email system.
The connections hash will be passed directly to a mysql2 connection. Follow the link to see all the available options.

Limitations

mySQL only (the mysql2 gem specifically)

Forklift is available now. Enjoy!

[…]

The Ruby Rails Aggregator

Category Archives: News

Episode #521 – January 13th, 2015

Sponsored by Top Ruby Jobs

Useful Methods

Metrics That Matter

sprockets-traceur

Toxiproxy

email_prefixer

Phone verification using SMS via Twilio

Sponsored by Ruby5

Episode #520 – January 9th, 2015

Episode #519 – January 6th, 2015

Sponsored by Top Ruby Jobs

Ruby 2.2.0 Released

Rails 4.2.0

Finishing Moves

RubySpec

Databound

How to test controllers in Rails

Remembering James Golick

Sponsored by Ruby5

Episode #518 – December 19, 2014

Episode #517 – December 16, 2014

Sponsored by Top ruby Jobs

Streem

Rails 4.2.0 Release Candidate

Papercrop

RubyMotion 3.0

Sponsored by Gift of Code

Episode #516 – December 9, 2014

Episode #515 – December 5th, 2014

Episode #514 – December 2, 2014

Episode #513 – November 25th, 2014

Episode #512 – November 21st, 2014