Debugging an Assertion Error in Ruby

I hope nobody runs in to a problem where they need the information in this post, but in case you do, I hope this post is helpful.
(I’m talking to you, future Aaron! lol)

I committed a patch to Ruby that caused the tests to start failing.
This was the patch:

commit 1be84e53d76cff30ae371f0b397336dee934499d
Author: Aaron Patterson <tenderlove@ruby-lang.org>
Date:   Mon Feb 1 10:42:13 2021 -0800

    Don't pin `val` passed in to `rb_define_const`.
    
    The caller should be responsible for holding a pinned reference (if they
    need that)

diff --git a/variable.c b/variable.c
index 92d7d11eab..ff4f7964a7 100644
--- a/variable.c
+++ b/variable.c
@@ -3154,7 +3154,6 @@ rb_define_const(VALUE klass, const char *name, VALUE val)
     if (!rb_is_const_id(id)) {
        rb_warn("rb_define_const: invalid name `%s' for constant", name);
     }
-    rb_gc_register_mark_object(val);
     rb_const_set(klass, id, val);
 }

This patch is supposed to allow objects passed in to rb_define_const to move.
As the commit message says, the caller should be responsible for keeping the value pinned.
At the time I committed the patch, I thought that most callers of the function were marking the value passed in (as val), so we were pinning objects that something else would already pin.
In other words, this code was being wasteful by chewing up GC time by pinning objects that were already pinned.

Unfortunately the CI started to error shortly after I committed this patch.
Clearly the patch was related, but how?

In this post I am going to walk through the debugging tricks I used to find the error.

Reproduction

I was able to reproduce the error on my Linux machine by running the same command CI ran.
Unfortunately since this bug is related to GC, the error was intermittent.
To reproduce it, I just ran the tests in a loop until the process crashed like this:

$ while test $status -eq 0
    env RUBY_TESTOPTS='-q --tty=no' make -j16 -s check
  end

Before running this loop though, I made sure to do ulimit -c unlimited so that I would get a core file when the process crashed.

The Error

After the process crashed, the top of the error looked like this:

<OBJ_INFO:rb_ractor_confirm_belonging@./ractor_core.h:327> 0x000055be8657f180 [0     ] T_NONE 
/home/aaron/git/ruby/lib/bundler/environment_preserver.rb:47: [BUG] id == 0 but not shareable
ruby 3.1.0dev (2021-02-03T17:35:37Z master 6b4814083b) [x86_64-linux]

The Ractor verification routines crashed the process because a T_NONE object is “not sharable”.
In other words you can’t share an object of type T_NONE between Ractors.
This makes sense because T_NONE objects are actually empty slots in the GC.
If a Ractor, or any other Ruby code sees a T_NONE object, then it’s clearly an error.
Only the GC internals should ever be dealing with this type.

The top of the C backtrace looked like this:

-- C level backtrace information -------------------------------------------
/home/aaron/git/ruby/ruby(rb_print_backtrace+0x14) [0x55be856e9816] vm_dump.c:758
/home/aaron/git/ruby/ruby(rb_vm_bugreport) vm_dump.c:1020
/home/aaron/git/ruby/ruby(bug_report_end+0x0) [0x55be854e2a69] error.c:778
/home/aaron/git/ruby/ruby(rb_bug_without_die) error.c:778
/home/aaron/git/ruby/ruby(rb_bug+0x7d) [0x55be854e2bb0] error.c:786
/home/aaron/git/ruby/ruby(rb_ractor_confirm_belonging+0x102) [0x55be856cf6e2] ./ractor_core.h:328
/home/aaron/git/ruby/ruby(vm_exec_core+0x4ff3) [0x55be856b0003] vm.inc:2224
/home/aaron/git/ruby/tool/lib/test/unit/parallel.rb(rb_vm_exec+0x886) [0x55be856c9946]
/home/aaron/git/ruby/ruby(load_iseq_eval+0xbb) [0x55be8554f66b] load.c:594
/home/aaron/git/ruby/ruby(require_internal+0x394) [0x55be8554e3e4] load.c:1065
/home/aaron/git/ruby/ruby(rb_require_string+0x973c4) [0x55be8554d8a4] load.c:1142
/home/aaron/git/ruby/ruby(rb_f_require) load.c:838
/home/aaron/git/ruby/ruby(vm_call_cfunc_with_frame+0x11a) [0x55be856dd6fa] ./vm_insnhelper.c:2897
/home/aaron/git/ruby/ruby(vm_call_method_each_type+0xaa) [0x55be856d4d3a] ./vm_insnhelper.c:3387
/home/aaron/git/ruby/ruby(vm_call_alias+0x87) [0x55be856d68e7] ./vm_insnhelper.c:3037
/home/aaron/git/ruby/ruby(vm_sendish+0x200) [0x55be856d08e0] ./vm_insnhelper.c:4498

The function rb_ractor_confirm_belonging was the function raising an exception.

Debugging the Core File with LLDB

I usually use clang / lldb when debugging.
I’ve added scripts to Ruby’s lldb tools that let me track down problems more easily, so I prefer it over gcc / gdb.

First I inspected the backtrace in the corefile:

(lldb) target create "./ruby" --core "core.456156"
Core file '/home/aaron/git/ruby/core.456156' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'ruby', stop reason = signal SIGABRT
  * frame #0: 0x00007fdc5fc8918b libc.so.6`raise + 203
    frame #1: 0x00007fdc5fc68859 libc.so.6`abort + 299
    frame #2: 0x000056362ac38bc6 ruby`die at error.c:765:5
    frame #3: 0x000056362ac38bb5 ruby`rb_bug(fmt=<unavailable>) at error.c:788:5
    frame #4: 0x000056362ae256e2 ruby`rb_ractor_confirm_belonging(obj=<unavailable>) at ractor_core.h:328:13
    frame #5: 0x000056362ae06003 ruby`vm_exec_core(ec=<unavailable>, initial=<unavailable>) at vm.inc:2224:5
    frame #6: 0x000056362ae1f946 ruby`rb_vm_exec(ec=<unavailable>, mjit_enable_p=<unavailable>) at vm.c:0
    frame #7: 0x000056362aca566b ruby`load_iseq_eval(ec=0x000056362b176710, fname=0x000056362ce96660) at load.c:594:5
    frame #8: 0x000056362aca43e4 ruby`require_internal(ec=<unavailable>, fname=<unavailable>, exception=1) at load.c:1065:21
    frame #9: 0x000056362aca38a4 ruby`rb_f_require [inlined] rb_require_string(fname=0x00007fdc38033178) at load.c:1142:18
    frame #10: 0x000056362aca3880 ruby`rb_f_require(obj=<unavailable>, fname=0x00007fdc38033178) at load.c:838
    frame #11: 0x000056362ae336fa ruby`vm_call_cfunc_with_frame(ec=0x000056362b176710, reg_cfp=0x00007fdc5f958de0, calling=<unavailable>) at vm_insnhelper.c:2897:11
    frame #12: 0x000056362ae2ad3a ruby`vm_call_method_each_type(ec=0x000056362b176710, cfp=0x00007fdc5f958de0, calling=0x00007ffe3b552128) at vm_insnhelper.c:3387:16
    frame #13: 0x000056362ae2c8e7 ruby`vm_call_alias(ec=0x000056362b176710, cfp=0x00007fdc5f958de0, calling=0x00007ffe3b552128) at vm_insnhelper.c:3037:12

It’s very similar to the backtrace in the crash report.
The first thing that was interesting to me was frame 5 in vm_exec_core.
vm_exec_core is the main loop for the YARV VM.
This program was crashing when executing some kind of instruction in the virtual machine.

(lldb) f 5
frame #5: 0x000056362ae06003 ruby`vm_exec_core(ec=<unavailable>, initial=<unavailable>) at vm.inc:2224:5
   2221	    /* ### Instruction trailers. ### */
   2222	    CHECK_VM_STACK_OVERFLOW_FOR_INSN(VM_REG_CFP, INSN_ATTR(retn));
   2223	    CHECK_CANARY(leaf, INSN_ATTR(bin));
-> 2224	    PUSH(val);
   2225	    if (leaf) ADD_PC(INSN_ATTR(width));
   2226	#   undef INSN_ATTR
   2227	
(lldb) 

Checking frame 5, we can see that it’s crashing when we push a value on to the stack.
The Ractor function checks the value of objects being pushed on the VM stack, and in this case we have an object that is a T_NONE.
The question is where did this value come from?

The crash happened in the file vm.inc, line 2224. This file is a generated
file, so I can’t link to it, but I wanted to know which instruction was being
executed, so I pulled up that file.

Line 2224 happened to be inside the opt_send_without_block instruction.
So something is calling a method, and the return value of the method is a T_NONE object.

But what method is being called, and on what object?

Finding the called method

The value ec, or “Execution Context” contains information about the virtual machine at runtime.
On the ec, we can find the cfp or “Control Frame Pointer” which is a data structure representing the current executing stack frame.
In lldb, I could see that frame 7 had the ec available, so I went to that frame to look at the cfp:

(lldb) f 7
frame #7: 0x000056362aca566b ruby`load_iseq_eval(ec=0x000056362b176710, fname=0x000056362ce96660) at load.c:594:5
   591 	        rb_ast_dispose(ast);
   592 	    }
   593 	    rb_exec_event_hook_script_compiled(ec, iseq, Qnil);
-> 594 	    rb_iseq_eval(iseq);
   595 	}
   596 	
   597 	static inline enum ruby_tag_type
(lldb) p *ec->cfp
(rb_control_frame_t) $1 = {
  pc = 0x000056362c095d58
  sp = 0x00007fdc5f859330
  iseq = 0x000056362ca051f0
  self = 0x000056362b1d92c0
  ep = 0x00007fdc5f859328
  block_code = 0x0000000000000000
  __bp__ = 0x00007fdc5f859330
}

The control frame pointer has a pointer to the iseq or “Instruction Sequence” that is currently being executed.
It also has a pc or “Program Counter”, and the program counter usually points at the instruction that will be executed next (in other words, not the currently executing instruction).
Of other interest, the iseq also has the source location that corresponds to those instructions.

Getting the Source File

If we examine the iseq structure, we can find the source location of the code that is currently being executed:

(lldb) p ec->cfp->iseq->body->location
(rb_iseq_location_t) $4 = {
  pathobj = 0x000056362ca06960
  base_label = 0x000056362ce95a30
  label = 0x000056362ce95a30
  first_lineno = 0x0000000000000051
  node_id = 137
  code_location = {
    beg_pos = (lineno = 40, column = 4)
    end_pos = (lineno = 50, column = 7)
  }
}
(lldb) command script import -r ~/git/ruby/misc/lldb_cruby.py
lldb scripts for ruby has been installed.
(lldb) rp 0x000056362ca06960
bits [     ]
T_STRING: [FROZEN] (const char [57]) $6 = "/home/aaron/git/ruby/lib/bundler/environment_preserver.rb"
(lldb) 

The location info clearly shows us that the instructions are on line 40.
The pathobj member contains the file name, but it is stored as a Ruby string.
To print out the string, I imported the lldb CRuby extensions, then used the rp command and gave it the address of the path object.

From the output, we can see that it’s crashing in the “environment_preserver.rb” file inside of the instructions that are defined on line 40.
We’re not crashing on line 40, but the instructions are defined there.

Those instructions are this method:

    def replace_with_backup
      ENV.replace(backup) unless Gem.win_platform?

      # Fallback logic for Windows below to workaround
      # https://bugs.ruby-lang.org/issues/16798. Can be dropped once all
      # supported rubies include the fix for that.

      ENV.clear

      backup.each {|k, v| ENV[k] = v }
    end

It’s still not clear which of these method calls is breaking.
In this function we have some method call that is returning a T_NONE.

Finding The Method Call

To find the method call, I disassembled the instruction sequence and checked the program counter:

(lldb) command script import -r misc/lldb_disasm.py
lldb Ruby disasm installed.
(lldb) rbdisasm ec->cfp->iseq
PC             IDX  insn_name(operands) 
0x56362c095c20 0000 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095ee0 )
0x56362c095c38 0003 putobject( (VALUE)0x14 )
0x56362c095c48 0005 getconstant( ID: 0x807b )
0x56362c095c58 0007 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095ee0 )
0x56362c095c68 0009 opt_send_without_block( (struct rb_call_data *)0x56362c095f20 )
0x56362c095c78 0011 branchif( 15 )
0x56362c095c88 0013 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095ef0 )
0x56362c095ca0 0016 putobject( (VALUE)0x14 )
0x56362c095cb0 0018 getconstant( ID: 0x370b )
0x56362c095cc0 0020 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095ef0 )
0x56362c095cd0 0022 putself
0x56362c095cd8 0023 opt_send_without_block( (struct rb_call_data *)0x56362c095f30 )
0x56362c095ce8 0025 opt_send_without_block( (struct rb_call_data *)0x56362c095f40 )
0x56362c095cf8 0027 pop
0x56362c095d00 0028 opt_getinlinecache( 6, (struct iseq_inline_cache_entry *)0x56362c095f00 )
0x56362c095d18 0031 putobject( (VALUE)0x14 )
0x56362c095d28 0033 getconstant( ID: 0x370b )
0x56362c095d38 0035 opt_setinlinecache( (struct iseq_inline_cache_entry *)0x56362c095f00 )
0x56362c095d48 0037 opt_send_without_block( (struct rb_call_data *)0x56362c095f50 )
0x56362c095d58 0039 pop
0x56362c095d60 0040 putself
0x56362c095d68 0041 opt_send_without_block( (struct rb_call_data *)0x56362c095f60 )
0x56362c095d78 0043 send( (struct rb_call_data *)0x56362c095f70, (rb_iseq_t *)0x56362ca05178 )
0x56362c095d90 0046 leave
(lldb) p ec->cfp->pc
(const VALUE *) $9 = 0x000056362c095d58

First I loaded the disassembly helper script. It provides the rbdisasm function.
Then I used rbdisasm on the instruction sequence.
This printed out the instructions in mostly human readable form.
Printing the PC showed a value of 0x000056362c095d58.
Looking at the PC list in the disassembly shows that 0x000056362c095d58 corresponds to a pop instruction.
But the PC always points at the next instruction that will execute, not the currently executing instruction.
The currently executing instruction is the one right before the PC.
In this case we can see it is opt_send_without_block, which lines up with the information we discovered from vm.inc.

This is the 3rd from last method call in the block.
At 0041 there is another opt_send_without_block, and then at 0043 there is a generic send call.

Looking at the Ruby code, from the bottom of the method, we see a call to backup.
It’s not a local variable, so it must be a method call.
The code calls each on that, and each takes a block.
These must correspond to the opt_send_without_block and the send at the end of the instruction sequence.
Our crash is happening just before these two, so it must be the call to ENV.clear.

If we read the implementation of ENV.clear, we can see that it returns a global variable called envtbl:

VALUE
rb_env_clear(void)
{
    VALUE keys;
    long i;

    keys = env_keys(TRUE);
    for (i=0; i<RARRAY_LEN(keys); i++) {
        VALUE key = RARRAY_AREF(keys, i);
        const char *nam = RSTRING_PTR(key);
        ruby_setenv(nam, 0);
    }
    RB_GC_GUARD(keys);
    return envtbl;
}

This object is allocated here:

    envtbl = rb_obj_alloc(rb_cObject);

And then it calls rb_define_global_const to define the ENV constant as a global:

    /*
     * ENV is a Hash-like accessor for environment variables.
     *
     * See ENV (the class) for more details.
     */
    rb_define_global_const("ENV", envtbl);

If we read rb_define_global_const we can see that it just calls rb_define_const:

void
rb_define_global_const(const char *name, VALUE val)
{
    rb_define_const(rb_cObject, name, val);
}

Before my patch, any object passed to rb_define_const would be pinned.
Once I removed the pinning, that allowed the ENV variable to move around even though it shouldn’t.

I reverted that patch here, and then sent a pull request to make rb_gc_register_mark_object a little bit smarter here.

Conclusion

TBH I don’t know what to conclude this with.
Debugging errors kind of sucks, but I hope that the LLDB scripts I wrote make it suck a little less.
Hope you’re having a good day!!!

Read more at the source

Counting Write Barrier Unprotected Objects

This is just a quick post mostly as a note to myself (because I forget the jq commands).
Ruby objects that are not protected with a write barrier must be examined on every minor GC.
That means that any objects in your system that live for a long time and don’t have write barrier protection will cause unnecessary overhead on every minor collection.

Heap dumps will tell you which objects have a write barrier.
In Rails apps I use a small script to get a dump of the heap after boot:

require 'objspace'
require 'config/environment'

GC.start

File.open("heap.dump", "wb") do |f|
  ObjectSpace.dump_all(output: f)
end

The heap.dump file will have a list of all of the objects in the heap.

Here is an example of an object with a write barrier:

{"address":"0x7fec1b2ff940", "type":"IMEMO", "class":"0x7fec1b2ffd50", "imemo_type":"ment", "references":["0x7fec1b314908", "0x7fec1b2ffcd8"], "memsize":48, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}}

Here is an example of an object without a write barrier:

{"address":"0x7fec1b2ff760", "type":"ICLASS", "class":"0x7fec1a8c0f60", "references":["0x7fec1a8c9250", "0x7fec1b2fefe0"], "memsize":40}

Objects with a write barrier will have "wb_protected":true in their flags section.

I like to use jq to process heap dumps.
Here is a command to find all of the unprotected objects, group them by type, then count them up:

$ jq 'select(.flags.wb_protected | not) | .type' heap.dump  | sort | uniq -c | sort -n
   1 "MATCH"
   2 "ARRAY"
   5 "ROOT"
   9 "FILE"
 323 "MODULE"
 927 "ICLASS"
1631 "DATA"

All of the objects listed here will be examined on every minor GC.
If my Rails app is spending a lot of time in minor GCs, this is a good place to look.

Ruby 2.8 (or 3.0) will eliminate ICLASS from this list (here is the commit).

Read more at the source

Guide to String Encoding in Ruby

Encoding issues don’t seem to happen frequently, but that is a blessing and a curse.
It’s great not to fix them very frequently, but when you do need to fix them, lack
of experience can leave you feeling lost.

This post is meant to be a sort of guide about what to do when you encounter different types
of encoding errors in Ruby.
First we’ll cover what an encoding object is, then we’ll look at common encoding exceptions
and how to fix them.

What are String encodings in Ruby?

In Ruby, strings are a combination of an array of bytes, and an encoding object.
We can access the encoding object on the string by calling encoding on the
string object.

For example:

>> x = 'Hello World'
>> x.encoding
=> #<Encoding:UTF-8>

In my environment, the default encoding object associated with a string us the “UTF-8” encoding object.
A graph of the object relationship looks something like this:

string points at encoding

Changing a String’s Encoding

We can change encoding by two different methods:

  • String#force_encoding
  • String#encode

The force_encoding method will mutate the string object and only change which encoding object the string points to.
It does nothing to the bytes of the string, it merely changes the encoding object associated with the string.
Here we can see that the return value of encoding changes after we call the force_encode method:

>> x = 'Hello World'
>> x.encoding
=> #<Encoding:UTF-8>
>> x.force_encoding "US-ASCII"
=> "Hello World"
>> x.encoding
=> #<Encoding:US-ASCII>

The encode method will create a new string based on the bytes of the old string and associate the encoding object with the new string.

Here we can see that the encoding of x remains the same, and
calling encode returns a new string y which is associated with the new encoding:

>> x = 'Hello World'
>> x.encoding
=> #<Encoding:UTF-8>
>> y = x.encode("US-ASCII")
>> x.encoding
=> #<Encoding:UTF-8>
>> y.encoding
=> #<Encoding:US-ASCII>

Here is a visualization of the difference:

changing encoding

Calling force_encoding mutates the original string, where encode creates a new string with a different encoding.
Translating a string from one encoding to another is probably the “normal” use of encodings.
However, developers will rarely call the encode method because Ruby will typically handle any necessary translations automatically.
It’s probably more common to call the force_encoding method, and that is because strings can be associated with the wrong encoding.

Strings Can Have the Wrong Encoding

Strings can be associated with the wrong encoding object, and that is the source of most if not all encoding related exceptions.
Let’s look at an example:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.valid_encoding?
=> false

In this case, Ruby associated the string "Hello \x93\xfa\x96\x7b" with the default encoding UTF-8.
However, many of the bytes in the string are not valid Unicode characters.
We can check if the string is associated with a valid encoding object by calling valid_encoding? method.
The valid_encoding? method will scan all bytes to see if they are valid for that particular encoding object.

So how do we fix this?
The answer depends on the situation.
We need to think about where the data came from and where the data is going.
Let’s say we’ll display this string on a webpage, but we do not know the correct encoding for the string.
In that case we probably want to make sure the string is valid UTF-8, but since we don’t know the correct encoding for the string, our only choice is to remove the bad bytes from the string.

We can remove the unknown bytes by using the scrub method:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.valid_encoding?
=> false
>> y = x.scrub
>> y
=> "Hello ���{"
>> y.encoding
=> #<Encoding:UTF-8>
>> y.valid_encoding?
=> true

The scrub method will return a new string associated with the encoding but with all of the invalid bytes replaced by a replacement character, the diamond question mark thing.

What if we do know the encoding of the source string?
Actually the example above is using a string that’s encoding using Shift JIS.
Let’s say we know the encoding, and we want to display the string on a webpage.
In that case we tag the string by using force_encoding, and transcode to UTF-8:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.force_encoding "Shift_JIS"
=> "Hello \x{93FA}\x{967B}"
>> x.valid_encoding?
=> true
>> x.encode "UTF-8" # display as UTF-8
=> "Hello 日本"

The most important thing to think about when dealing with encoding issues is “where did this data come from?” and “what will we do with this data?”
Answering those two questions will drive all decisions about which encoding to use with which string.

Encoding Depends on the Context

Before we look at some common errors and their remediation, let’s look at one more example of the encoding context dependency.
In this example, we’ll use some user input as a cache key, but we’ll also display the user input on a webpage.
We’re going to use our source data (the user input) in two places: as a cache key, and something to display on a web page.

Here’s the code:

require "digest/md5"
require "cgi"

# Make a checksum
def make_checksum string
  Digest::MD5.hexdigest string
end

# Not good HTML escaping (don't use this)
# Returns a string with UTF-8 compatible encoding for display on a webpage
def display_on_web string
  string.gsub(/>/, "&gt;")
end

# User input from an unknown source
x = "Hello \x93\xfa\x96\x7b"
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?

p display_on_web x
p make_checksum x

If we run this code, we’ll get an exception:

$ ruby thing.rb
{:ENCODING=>#<Encoding:UTF-8>}
{:VALID_ENCODING=>false}
Traceback (most recent call last):
        2: from thing.rb:20:in `<main>'
        1: from thing.rb:12:in `display_on_web'
thing.rb:12:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)

The problem is that we have a string of unknown input with bytes that are not valid UTF-8 characters.
We know we want to display this string on a UTF-8 encoded webpage, so lets scrub the string:

require "digest/md5"
require "cgi"

# Make a checksum
def make_checksum string
  Digest::MD5.hexdigest string
end

# Not good HTML escaping (don't use this)
# Returns a string with UTF-8 compatible encoding for display on a webpage
def display_on_web string
  string.gsub(/>/, "&gt;")
end

# User input from an unknown source
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?

p display_on_web x
p make_checksum x

Now when we run the program, the output is like this:

$ ruby thing.rb
{:ENCODING=>#<Encoding:UTF-8>}
{:VALID_ENCODING=>true}
"Hello ���{"
"4dab6f63b4d3ae3279345c9df31091eb"

Great! We’ve build some HTML and generated a checksum.
Unfortunately there is a bug in this code (of course the mere fact that we’ve written code means there’s a bug! lol)
Let’s introduce a second user input string with slightly different bytes than the first input string:

require "digest/md5"
require "cgi"

# Make a checksum
def make_checksum string
  Digest::MD5.hexdigest string
end

# Not good HTML escaping (don't use this)
# Returns a string with UTF-8 compatible encoding for display on a webpage
def display_on_web string
  string.gsub(/>/, "&gt;")
end

# User input from an unknown source
x = "Hello \x93\xfa\x96\x7b".scrub
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?

p display_on_web x
p make_checksum x

# Second user input from an unknown source with slightly different bytes
y = "Hello \x94\xfa\x97\x7b".scrub
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?

p display_on_web y
p make_checksum y

Here is the output from the program:

$ ruby thing.rb
{:ENCODING=>#<Encoding:UTF-8>}
{:VALID_ENCODING=>true}
"Hello ���{"
"4dab6f63b4d3ae3279345c9df31091eb"
{:ENCODING=>#<Encoding:UTF-8>}
{:VALID_ENCODING=>true}
"Hello ���{"
"4dab6f63b4d3ae3279345c9df31091eb"

The program works in the sense that there is no exception.
But both user input strings have the same checksum despite the fact that the original strings clearly have different bytes!
So what is the correct fix for this program?
Again, we need to think about the source of the data (where did it come from), as well as what we will do with it (where it is going).
In this case we have one source, from a user, and the user provided us with no encoding information.
In other words, the encoding information of the source data is unknown, so we can only treat it as a sequence of bytes.
We have two output cases, one is a UTF-8 HTML the other output is the input to our checksum function.
The HTML requires that our string be UTF-8 so making the string valid UTF-8, in other words “scrubbing” it, before displaying makes sense.
However, our checksum function requires seeing the original bytes of the string.
Since the checksum is only concerned with the bytes in the string, any encoding including an invalid encoding will work.
It’s nice to make sure all our strings have valid encodings though, so we’ll fix this example such that everything has a valid encoding.

require "digest/md5"
require "cgi"

# Make a checksum
def make_checksum string
  Digest::MD5.hexdigest string
end

# Not good HTML escaping (don't use this)
# Returns a string with UTF-8 compatible encoding for display on a webpage
def display_on_web string
  string.gsub(/>/, "&gt;")
end

# User input from an unknown source
x = "Hello \x93\xfa\x96\x7b".b
p ENCODING: x.encoding
p VALID_ENCODING: x.valid_encoding?

p display_on_web x.encode("UTF-8", undef: :replace)
p make_checksum x

# Second user input from an unknown source with slightly different bytes
y = "Hello \x94\xfa\x97\x7b".b
p ENCODING: y.encoding
p VALID_ENCODING: y.valid_encoding?

p display_on_web y.encode("UTF-8", undef: :replace)
p make_checksum y

Here is the output of the program:

$ ruby thing.rb
{:ENCODING=>#<Encoding:ASCII-8BIT>}
{:VALID_ENCODING=>true}
"Hello ���{"
"96cf6db2750fd4d2488fac57d8e4d45a"
{:ENCODING=>#<Encoding:ASCII-8BIT>}
{:VALID_ENCODING=>true}
"Hello ���{"
"b92854c0db4f2c2c20eff349a9a8e3a0"

To fix our program, we’ve changed a couple things.
First we tagged the string of unknown encoding as “binary” by using the .b method.
The .b method returns a new string that is associated with the ASCII-8BIT encoding.
The name ASCII-8BIT is somewhat confusing because it has the word “ASCII” in it.
It’s better to think of this encoding as either “unknown” or “binary data”.
Unknown meaning we have some data that may have a valid encoding, but we don’t know what it is.
Or binary data, as in the bytes read from a JPEG file or some such binary format.
Anyway, we pass the binary string in to the checksum function because the checksum only cares about the bytes in the string, not about the encoding.

The second change we made is to call encode with the encoding we want (UTF-8) along with undef: :replace meaning that any time Ruby encounters bytes it doesn’t know how to convert to the target encoding, it will replace them with the replacement character (the diamond question thing).

SIDE NOTE: This is probably not important, but it is fun!
We can specify what Ruby uses for replacing unknown bytes.
Here’s an example:

>> x = "Hello \x94\xfa\x97\x7b".b
>> x.encoding
=> #<Encoding:ASCII-8BIT>
>> x.encode("UTF-8", undef: :replace, replace: "Aaron")
=> "Hello AaronAaronAaron{"
>> x.encode("UTF-8", undef: :replace, replace: "🤣")
=> "Hello 🤣🤣🤣{"
>> [_.encoding, _.valid_encoding?]
=> [#<Encoding:UTF-8>, true]

Now lets take a look at some common encoding errors in Ruby and what to do about them.

Encoding::InvalidByteSequenceError

This exception occurs when Ruby needs to examine the bytes in a string and the bytes do not match the encoding.
Here is an example of this exception:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.encode "UTF-16"
Traceback (most recent call last):
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        2: from (irb):4
        1: from (irb):4:in `encode'
Encoding::InvalidByteSequenceError ("\x93" on UTF-8)
>> x.encoding
=> #<Encoding:UTF-8>
>> x.valid_encoding?
=> false

The string x contains bytes that aren’t valid UTF-8, yet it is associated with the UTF-8 encoding object.
When we try to convert x to UTF-16, an exception occurs.

How to fix Encoding::InvalidByteSequenceError

Like most encoding issues, our string x is tagged with the wrong encoding.
The way to fix this issue is to tag the string with the correct encoding.
But what is the correct encoding?
To figure out the correct encoding, you need to know where the string came from.
For example if the string came from a Mime attachment, the Mime attachment should specify the encoding (or the RFC will tell you).

In this case, the string is a valid Shift JIS string, but I know that because I looked up the bytes and manually entered them.
So we’ll tag this as Shift JIS, and the exception goes away:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.force_encoding "Shift_JIS"
=> "Hello \x{93FA}\x{967B}"
>> x.encode "UTF-16"
=> "\uFEFFHello \u65E5\u672C"
>> x.encoding
=> #<Encoding:Shift_JIS>
>> x.valid_encoding?
=> true

If you don’t know the source of the string, an alternative solution is to tag as UTF-8 and then scrub the bytes:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.force_encoding "UTF-8"
=> "Hello \x93\xFA\x96{"
>> x.scrub!
=> "Hello ���{"
>> x.encode "UTF-16"
=> "\uFEFFHello \uFFFD\uFFFD\uFFFD{"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.valid_encoding?
=> true

Of course this works, but it means that you’ve lost data.
The best solution is to figure out what the encoding of the string should be depending on its source and tag it with the correct encoding.

Encoding::UndefinedConversionError

This exception occurs when a string of one encoding can’t be converted to another encoding.

Here is an example:

>> x = "四\u2160"
>> x
=> "四Ⅰ"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.valid_encoding?
=> true
>> x.encode "Shift_JIS"
Traceback (most recent call last):
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        2: from (irb):23
        1: from (irb):23:in `encode'
Encoding::UndefinedConversionError (U+2160 from UTF-8 to Shift_JIS)

In this example, we have two characters: “四”, and the Roman numeral 1 (“Ⅰ”).
Unicode Roman numeral 1 cannot be converted to Shift JIS because there are two codepoints that represent that character in Shift JIS.
This means the conversion is ambiguous, so Ruby will raise an exception.

How to fix Encoding::UndefinedConversionError

Our original string is correctly tagged as UTF-8, but we need to convert to Shift JIS.
In this case we’ll use a replacement character when converting to Shift JIS:

>> x = "四\u2160"
>> y = x.encode("Shift_JIS", undef: :replace)
>> y
=> "\x{8E6C}?"
>> y.encoding
=> #<Encoding:Shift_JIS>
>> y.valid_encoding?
=> true
>> y.encode "UTF-8"
=> "四?"

We were able to convert to Shift JIS, but we did lose some data.

ArgumentError

When a string contains invalid bytes, sometimes Ruby will raise an ArgumentError exception:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.downcase
Traceback (most recent call last):
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        2: from (irb):34
        1: from (irb):34:in `downcase'
ArgumentError (input string invalid)
>> x.gsub(/ello/, "i")
Traceback (most recent call last):
        6: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        3: from (irb):34
        2: from (irb):35:in `rescue in irb_binding'
        1: from (irb):35:in `gsub'
ArgumentError (invalid byte sequence in UTF-8)

Again we use our incorrectly tagged Shift JIS string.
Calling downcase or gsub both result in an ArgumentError.
I personally think these exceptions are not great.
We didn’t pass anything to downcase, so why is it an ArgumentError?
There is nothing wrong with the arguments we passed to gsub, so why is it an ArgumentError?
Why does one say “input string invalid” where the other gives us a slightly more helpful exception of “invalid byte sequence in UTF-8”?
I think these should both result in Encoding::InvalidByteSequenceError exceptions, as it’s a problem with the encoding, not the arguments.

Regardless, these errors both stem from the fact that the Shift JIS string is incorrectly tagged as UTF-8.

Fixing ArgumentError

Fixing this issue is just like fixing Encoding::InvalidByteSequenceError.
We need to figure out the correct encoding of the source string, then tag the source string with that encoding.
If the encoding of the source string is truly unknown, scrub it.

>> x = "Hello \x93\xfa\x96\x7b"
>> x.force_encoding "Shift_JIS"
=> "Hello \x{93FA}\x{967B}"
>> x.downcase
=> "hello \x{93FA}\x{967B}"
>> x.gsub(/ello/, "i")
=> "Hi \x{93FA}\x{967B}"

Encoding::CompatibilityError

This exception occurs when we try to combine strings of two different encodings and those encodings are incompatible.
For example:

>> x = "四\u2160"
>> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
>> [x.encoding, x.valid_encoding?]
=> [#<Encoding:UTF-8>, true]
>> [y.encoding, y.valid_encoding?]
=> [#<Encoding:Shift_JIS>, true]
>> x + y
Traceback (most recent call last):
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        2: from (irb):50
        1: from (irb):50:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and Shift_JIS)

In this example we have a valid UTF-8 string and a valid Shift JIS string.
However, these two encodings are not compatible, so we get an exception when combining.

Fixing Encoding::CompatibilityError

To fix this exception, we need to manually convert one string to a new string that has a compatible encoding.
In the case above, we can choose whether we want the output string to be UTF-8 or Shift JIS, and then call encode on the appropriate string.

In the case we want UTF-8 output, we can do this:

>> x = "四"
>> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
>> x + y.encode("UTF-8")
=> "四Hello 日本"
>> [_.encoding, _.valid_encoding?]
=> [#<Encoding:UTF-8>, true]

If we wanted Shift JIS, we could do this:

>> x = "四"
>> y = "Hello \x93\xfa\x96\x7b".force_encoding("Shift_JIS")
>> x.encode("Shift_JIS") + y
=> "\x{8E6C}Hello \x{93FA}\x{967B}"
>> [_.encoding, _.valid_encoding?]
=> [#<Encoding:Shift_JIS>, true]

Another possible solution is to scrub bytes and concatenate, but again that results in data loss.

What is a compatible encoding?

If there are incompatible encodings, there must be compatible encodings too (at least I would think that).
Here is an example of compatible encodings:

>> x = "Hello World!".force_encoding "US-ASCII"
>> [x.encoding, x.valid_encoding?]
=> [#<Encoding:US-ASCII>, true]
>> y = "こんにちは"
>> [y.encoding, y.valid_encoding?]
=> [#<Encoding:UTF-8>, true]
>> y + x
=> "こんにちはHello World!"
>> [_.encoding, _.valid_encoding?]
=> [#<Encoding:UTF-8>, true]
>> x + y
=> "Hello World!こんにちは"
>> [_.encoding, _.valid_encoding?]
=> [#<Encoding:UTF-8>, true]

The x string is encoded with “US ASCII” encoding and the y string UTF-8.
US ASCII is fully compatible with UTF-8, so even though these two strings have different encoding, concatenation works fine.

String literals may default to UTF-8, but some functions will return US ASCII encoded strings.
For example:

>> require "digest/md5"
=> true
>> Digest::MD5.hexdigest("foo").encoding
=> #<Encoding:US-ASCII>

A hexdigest will only ever contain ASCII characters, so the implementation tags the returned string as US-ASCII.

Encoding Gotchas

Let’s look at a couple encoding gotcha’s.

Infectious Invalid Encodings

When a string is incorrectly tagged, Ruby will typically only raise an exception when it needs to actually examine the bytes.
Here is an example:

>> x = "Hello \x93\xfa\x96\x7b"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.valid_encoding?
=> false
>> x + "ほげ"
=> "Hello \x93\xFA\x96{ほげ"
>> y = _
>> y
=> "Hello \x93\xFA\x96{ほげ"
>> [y.encoding, y.valid_encoding?]
=> [#<Encoding:UTF-8>, false]

Again we have the incorrectly tagged Shift JIS string.
We’re able to append a correctly tagged UTF-8 string and no exception is raised.
Why is that?
Ruby assumes that if both strings have the same encoding, there is no reason to validate the bytes in either string so it will just append them.
That means we can have an incorrectly tagged string “infect” what would otherwise be correctly tagged UTF-8 strings.
Say we have some code like this:

def append string
  string + "ほげ"
end

p append("ほげ").valid_encoding? # => true
p append("Hello \x93\xfa\x96\x7b").valid_encoding? # = false

When debugging this code, we may be tempted to think the problem is in the append method.
But actually the issue is with the caller.
The caller is passing in incorrectly tagged strings, and unfortunately we might not get an exception until the return value of append is used somewhere far away.

ASCII-8BIT is Special

Sometimes ASCII-8BIT is considered to be a “compatible” encoding and sometimes it isn’t.
Here is an example:

>> x = "\x93\xfa\x96\x7b".b
>> x.encoding
=> #<Encoding:ASCII-8BIT>
>> y = "ほげ"
>> y + x
Traceback (most recent call last):
        5: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `<main>'
        4: from /Users/aaron/.rbenv/versions/ruby-trunk/bin/irb:23:in `load'
        3: from /Users/aaron/.rbenv/versions/ruby-trunk/lib/ruby/gems/2.7.0/gems/irb-1.2.0/exe/irb:11:in `<top (required)>'
        2: from (irb):89
        1: from (irb):89:in `+'
Encoding::CompatibilityError (incompatible character encodings: UTF-8 and ASCII-8BIT)

Here we have a binary string stored in x.
Maybe it came from a JPEG file or something (it didn’t, I just typed it in!)
When we try to concatenate the binary string with the UTF-8 string, we get an exception.
But this may actually be an exception we want!
It doesn’t make sense to be concatenating some JPEG data with an actual string we want to view, so it’s good we got an exception here.

Now here is the same code, but with the contents of x changed somewhat:

>> x = "Hello World".b
>> x.encoding
=> #<Encoding:ASCII-8BIT>
>> y = "ほげ"
>> y + x
=> "ほげHello World"

We have the same code with the same encodings at play.
The only thing that changed is the actual contents of the x string.

When Ruby concatenates ASCII-8BIT strings, it will examine the contents of that string.
If all bytes in the string are ASCII characters, it will treat it as a US-ASCII string and consider it to be “compatible”.
If the string contains non-ASCII characters, it will consider it to be incompatible.

This means that if you had read some data from your JPEG, and that data happened to all be ASCII characters, you would not get an exception even though maybe you really wanted one.

In my personal opinion, concatenating an ASCII-8BIT string with anything besides another ASCII-8BIT string should be an exception.

Anyway, this is all I feel like writing today. I hope you have a good day, and remember to check your encodings!

Read more at the source

Important information about our Elixir and Ruby Open Source projects

You may have heard that Nubank has acqui-hired Plataformatec. Plataformatec has been working with Nubank over the past few months and Nubank saw great value on the practices and expertise shown by our teams. According to Nubank leaders, Plataformatec consultants have provided restructured rituals and new working agreements to its teams, and also brought improvements … »

The post Important information about our Elixir and Ruby Open Source projects first appeared on Plataformatec Blog.

Read more at the source

OKR: lições aprendidas para você começar a aplicá-lo de forma efetiva

Depois do sucesso do livro Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs de John Doerr, praticamente toda organização vem buscando utilizar OKR como forma de desdobrar seus objetivos e medir os avanços dos resultados. Assim como qualquer modelo, framework ou ferramenta, existe uma tendência natural das pessoas acreditarem que o artefato … »

The post OKR: lições aprendidas para você começar a aplicá-lo de forma efetiva first appeared on Plataformatec Blog.

Read more at the source
close