Skip to content

Move C and Java code to Ruby and fix #133#135

Merged
tagomoris merged 17 commits intomsgpack:masterfrom
tenderlove:remove-global-c
Jun 13, 2017
Merged

Move C and Java code to Ruby and fix #133#135
tagomoris merged 17 commits intomsgpack:masterfrom
tenderlove:remove-global-c

Conversation

@tenderlove
Copy link
Contributor

Hi,

This PR moves lots of the encoding / decoding logic from C to Ruby. It fixes #133 because the C code doesn't need to reference the class anymore. I prefer this PR over #134 because it removes lots of C code.

I've run the benchmarks to compare this branch against the master branch, and benchmark times don't change (as I expected) even though we are using more Ruby code.

Here are the benchmarks from master running on Ruby 2.4:

[aaron@TC msgpack-ruby (master)]$ RUBYOPT=-Ilib sh bench/run.sh 
pack
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+----------+---------+----------+----------+----------+----------+
| :bench   | :runs   | :user    | :system  | :total   | :real    |
+----------+---------+----------+----------+----------+----------+
| :strings |   10000 | 0.000000 | 0.000000 | 0.000000 | 0.009500 |
| :strings |  100000 | 0.100000 | 0.000000 | 0.100000 | 0.107401 |
| :strings | 1000000 | 1.600000 | 0.010000 | 1.610000 | 1.682810 |
| :symbols |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.011381 |
| :symbols |  100000 | 0.180000 | 0.000000 | 0.180000 | 0.188534 |
| :symbols | 1000000 | 1.630000 | 0.020000 | 1.650000 | 1.744726 |
+----------+---------+----------+----------+----------+----------+
unpack
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+----------+---------+----------+----------+----------+----------+
| :bench   | :runs   | :user    | :system  | :total   | :real    |
+----------+---------+----------+----------+----------+----------+
| :strings |   10000 | 0.030000 | 0.000000 | 0.030000 | 0.041100 |
| :strings |  100000 | 0.290000 | 0.010000 | 0.300000 | 0.303145 |
| :strings | 1000000 | 3.110000 | 0.030000 | 3.140000 | 3.212419 |
| :symbols |   10000 | 0.030000 | 0.000000 | 0.030000 | 0.034893 |
| :symbols |  100000 | 0.370000 | 0.010000 | 0.380000 | 0.387672 |
| :symbols | 1000000 | 3.430000 | 0.030000 | 3.460000 | 3.570336 |
+----------+---------+----------+----------+----------+----------+
pack log
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+------------+---------+----------+----------+----------+----------+
| :bench     | :runs   | :user    | :system  | :total   | :real    |
+------------+---------+----------+----------+----------+----------+
| :plain     |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.010122 |
| :plain     |  100000 | 0.110000 | 0.000000 | 0.110000 | 0.117346 |
| :plain     | 1000000 | 1.270000 | 0.030000 | 1.300000 | 1.343877 |
| :structure |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.014418 |
| :structure |  100000 | 0.160000 | 0.010000 | 0.170000 | 0.168190 |
| :structure | 1000000 | 1.610000 | 0.060000 | 1.670000 | 1.714063 |
+------------+---------+----------+----------+----------+----------+
unpack log
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+------------+---------+----------+----------+----------+----------+
| :bench     | :runs   | :user    | :system  | :total   | :real    |
+------------+---------+----------+----------+----------+----------+
| :plain     |   10000 | 0.030000 | 0.030000 | 0.060000 | 0.051582 |
| :plain     |  100000 | 0.260000 | 0.030000 | 0.290000 | 0.294509 |
| :plain     | 1000000 | 2.440000 | 0.160000 | 2.600000 | 2.709385 |
| :structure |   10000 | 0.070000 | 0.000000 | 0.070000 | 0.069416 |
| :structure |  100000 | 0.620000 | 0.010000 | 0.630000 | 0.638155 |
| :structure | 1000000 | 6.540000 | 0.050000 | 6.590000 | 6.785341 |
+------------+---------+----------+----------+----------+----------+

Here are the benchmarks run against this PR using Ruby 2.4:

[aaron@TC msgpack-ruby (remove-global-c)]$ RUBYOPT=-Ilib sh bench/run.sh 
pack
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+----------+---------+----------+----------+----------+----------+
| :bench   | :runs   | :user    | :system  | :total   | :real    |
+----------+---------+----------+----------+----------+----------+
| :strings |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.016913 |
| :strings |  100000 | 0.130000 | 0.010000 | 0.140000 | 0.126337 |
| :strings | 1000000 | 1.170000 | 0.000000 | 1.170000 | 1.205452 |
| :symbols |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.013735 |
| :symbols |  100000 | 0.140000 | 0.000000 | 0.140000 | 0.146469 |
| :symbols | 1000000 | 1.450000 | 0.010000 | 1.460000 | 1.492300 |
+----------+---------+----------+----------+----------+----------+
unpack
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+----------+---------+----------+----------+----------+----------+
| :bench   | :runs   | :user    | :system  | :total   | :real    |
+----------+---------+----------+----------+----------+----------+
| :strings |   10000 | 0.040000 | 0.010000 | 0.050000 | 0.044605 |
| :strings |  100000 | 0.310000 | 0.010000 | 0.320000 | 0.322814 |
| :strings | 1000000 | 3.190000 | 0.020000 | 3.210000 | 3.310599 |
| :symbols |   10000 | 0.030000 | 0.000000 | 0.030000 | 0.034799 |
| :symbols |  100000 | 0.340000 | 0.010000 | 0.350000 | 0.364677 |
| :symbols | 1000000 | 3.380000 | 0.020000 | 3.400000 | 3.493115 |
+----------+---------+----------+----------+----------+----------+
pack log
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+------------+---------+----------+----------+----------+----------+
| :bench     | :runs   | :user    | :system  | :total   | :real    |
+------------+---------+----------+----------+----------+----------+
| :plain     |   10000 | 0.010000 | 0.000000 | 0.010000 | 0.017635 |
| :plain     |  100000 | 0.140000 | 0.000000 | 0.140000 | 0.148023 |
| :plain     | 1000000 | 1.300000 | 0.020000 | 1.320000 | 1.343853 |
| :structure |   10000 | 0.010000 | 0.010000 | 0.020000 | 0.014556 |
| :structure |  100000 | 0.170000 | 0.000000 | 0.170000 | 0.174690 |
| :structure | 1000000 | 1.770000 | 0.040000 | 1.810000 | 1.857091 |
+------------+---------+----------+----------+----------+----------+
unpack log
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Fixnum is deprecated
/Users/aaron/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/myrrha-1.2.2/lib/myrrha/to_ruby_literal.rb:6: warning: constant ::Bignum is deprecated
+------------+---------+----------+----------+----------+----------+
| :bench     | :runs   | :user    | :system  | :total   | :real    |
+------------+---------+----------+----------+----------+----------+
| :plain     |   10000 | 0.020000 | 0.020000 | 0.040000 | 0.042739 |
| :plain     |  100000 | 0.250000 | 0.040000 | 0.290000 | 0.290547 |
| :plain     | 1000000 | 2.560000 | 0.200000 | 2.760000 | 2.846415 |
| :structure |   10000 | 0.070000 | 0.000000 | 0.070000 | 0.070674 |
| :structure |  100000 | 0.670000 | 0.000000 | 0.670000 | 0.687090 |
| :structure | 1000000 | 6.670000 | 0.050000 | 6.720000 | 6.972385 |
+------------+---------+----------+----------+----------+----------+

They look almost the same. We are able to move the C code to Ruby, prevent SEGV, and performance stays the same.

I'm sorry the patch is so large. 🙍

This commit allows us to pass the `DefaultFactory` object in to the C
code so that we don't need to use a C global for storing the factory
object.
This commit pulls the is-a checks up in to Ruby, constructs the
unpacker, then passes that unpacker in to the _load function so the C
code doesn't need to know about the `DefaultFactory` object.
We shouldn't need to reference the C global anymore
@tenderlove tenderlove changed the title Move C code to Ruby and fix #133 Move C and Java code to Ruby and fix #133 Mar 10, 2017
tenderlove added a commit to github/ruby that referenced this pull request Mar 10, 2017
This is a work around for a bug in msgpack.  It should be fixed by:

 * msgpack/msgpack-ruby#134
 * msgpack/msgpack-ruby#135
@iconara
Copy link
Member

iconara commented Mar 16, 2017

TypeConverter.checkType is not available in JRuby 1.7.x. Since Encoder will use instanceof to determine the type of the objects later anyway I think something like this should be good enough:

private void checkType(ThreadContext ctx, IRubyObject obj, Class<? extends IRubyObject> expectedType) {
  if (!expectedType.isInstance(obj)) {
    String expectedName = expectedType.getName().substring("org.jruby.Ruby".length());
    throw ctx.runtime.newTypeError(String.format("wrong argument type %s (expected %s)", obj.getMetaClass().toString(), expectedName));
  }
}

@JRubyMethod(name = "write_float")
public IRubyObject writeFloat(ThreadContext ctx, IRubyObject obj) {
  checkType(ctx, obj, RubyFloat.class);
  return write(ctx, obj);
}

@tagomoris
Copy link
Member

@tenderlove How do you think about JRuby 1.7.x support mentioned by @iconara 's comment?

@tenderlove
Copy link
Contributor Author

I'm going to update this patch to use @iconara's code. I've just been a little busy. I'll update the PR this week. Thank you!

@tenderlove
Copy link
Contributor Author

@iconara thanks for the example! I've implemented that in be50d62 and now everything is working! 😊

def to_msgpack(packer_or_io = nil)
if packer_or_io
if packer_or_io.is_a?(MessagePack::Packer)
_to_msgpack packer_or_io
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe name _to_msgpack –> to_msgpack_with_packer? And perhaps make it private? :bowtie:

@tagomoris
Copy link
Member

LGTM.
@frsyuki @iconara any concern?

@tagomoris tagomoris merged commit 1b531b2 into msgpack:master Jun 13, 2017
@tagomoris
Copy link
Member

I'm so sorry that I missed to merge this patch.
Thank you!

@tenderlove
Copy link
Contributor Author

@tagomoris no problem at all, thank you!! 😊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setting DefaultFactory to nil will cause a segv

5 participants