About Me

My Photo
I am a Software Architect and Developer based in Bangalore, India. I have experience in building (scalable) applications using Java, JSP, JSF, JBoss Drools, Spring Framework, Hibernate, Ajax, JavaScript, MySQL, NoSQL (HBase, Project Voldemort). I am also a fan of Ruby on Rails, and have done some experimental work with it.

Sunday, June 26, 2011

HBase Benchmarking for multi-threaded environment

This weekend I attempted to figure out how HBase writes perform in multi-threaded environments.  To do so,  I wrote a Java program that writes records in a HBase table, and which can be configured to run to write N number of records using M Threads. 

I used three variants to write records into HBase:
  1. Use one HTable for every write
    1. HTable is created using a singleton instance of HBaseConfiguration
    2. HTable is created using new instance of HBaseConfiguration (Why? - I wanted to check how the behavior changes if connections to servers are not shared.  Reference: HTable and HConnectionManager documentation)
  2. Use HTablePool

Here are the various results:

HBase 0.20.6 Standalone mode  (all figures in milliseconds)
Scenario Num Threads Puts Mean Standard Deviation Max Min     80th Percentile 90th Percentile
HTable from HTablePool (pool size: 100) 1 100 4.86 1.49 12.00 3.00 5.00 6.90
HTable with same configuration instance 1 100 4.52 1.12 9.00 3.00 5.00 6.90
HTable with new Hbase configuration instance 1 100 4.15 0.88 8.00 3.00 4.00 5.00
HTable from HTablePool (pool size: 100) 1 1000 4.19 3.60 108.00 3.00 4.00 5.00
HTable with same configuration instance 1 1000 4.02 1.79 47.00 3.00 4.00 5.00
HTable with new Hbase configuration instance 1 1000 3.84 3.57 108.00 3.00 4.00 4.00
HTable from HTablePool (pool size: 100) 1 10000 3.95 4.13 211.00 2.00 4.00 5.00
HTable with same configuration instance 1 10000 4.28 10.02 571.00 2.00 4.00 5.00
HTable with new Hbase configuration instance 1 10000 3.80 5.46 244.00 2.00 4.00 4.00
HTable from HTablePool (pool size: 100) 10 100 24.59 22.73 204.00 4.00 38.60 44.00
HTable with same configuration instance 10 100 20.48 28.05 233.00 5.00 22.00 36.80
HTable with new Hbase configuration instance 10 100 21.44 28.05 241.00 5.00 26.80 50.90
HTable from HTablePool (pool size: 100) 10 1000 26.10 21.94 246.00 5.00 40.00 49.00
HTable with same configuration instance 10 1000 23.80 25.31 238.00 3.00 35.80 44.00
HTable with new Hbase configuration instance 10 1000 22.55 21.96 209.00 4.00 35.00 44.00
HTable from HTablePool (pool size: 100) 10 10000 22.06 18.81 279.00 3.00 36.00 43.00
HTable with same configuration instance 10 10000 22.67 18.44 287.00 3.00 37.00 45.00
HTable with new Hbase configuration instance 10 10000 24.43 26.67 507.00 3.00 38.00 46.00
HTable from HTablePool (pool size: 100) 100 100 106.45 68.53 422.00 5.00 164.60 199.40
HTable with same configuration instance 100 100 102.81 68.45 421.00 4.00 167.80 194.60
HTable with new Hbase configuration instance 100 100 25.73 20.58 209.00 4.00 29.80 35.90
HTable from HTablePool (pool size: 100) 100 1000 195.38 189.46 696.00 3.00 375.80 488.00
HTable with same configuration instance 100 1000 189.86 193.69 688.00 3.00 384.00 492.00
HTable with new Hbase configuration instance 100 1000 79.80 110.55 525.00 4.00 151.00 252.80
HTable from HTablePool (pool size: 100) 100 10000 219.65 214.10 1277.00 3.00 433.00 543.00
HTable with same configuration instance 100 10000 277.60 473.36 4129.00 3.00 453.80 609.00
HTable with new Hbase configuration instance 100 10000 214.80 230.87 1386.00 3.00 435.00 566.00


HBase 0.20.6 Distributed Mode with 3 Region Servers (Hadoop 0.20) (all figures in milliseconds)
Scenario Num Threads Puts Mean Standard Deviation Max Min     80th Percentile 90th Percentile
HTable from HTablePool (pool size: 100) 1 100 3.25 1.96 18.00 2.00 3.00 3.90
HTable with same configuration instance 1 100 2.98 1.18 9.00 2.00 3.00 4.00
HTable with new Hbase configuration instance 1 100 2.91 1.43 10.00 1.00 3.00 4.90
HTable from HTablePool (pool size: 100) 1 1000 4.56 3.77 40.00 1.00 6.00 11.00
HTable with same configuration instance 1 1000 4.98 4.04 20.00 1.00 11.00 11.00
HTable with new Hbase configuration instance 1 1000 2.28 1.20 9.00 1.00 3.00 3.00
HTable from HTablePool (pool size: 100) 1 10000 2.48 1.76 41.00 1.00 3.00 3.00
HTable with same configuration instance 1 10000 2.43 1.78 49.00 1.00 3.00 3.00
HTable with new Hbase configuration instance 1 10000 2.84 2.79 52.00 1.00 3.00 5.00
HTable from HTablePool (pool size: 100) 10 100 15.88 21.62 204.00 2.00 23.80 32.90
HTable with same configuration instance 10 100 13.09 20.50 204.00 2.00 18.00 20.00
HTable with new Hbase configuration instance 10 100 8.78 19.64 197.00 2.00 8.80 13.00
HTable from HTablePool (pool size: 100) 10 1000 12.36 10.76 203.00 2.00 19.00 22.00
HTable with same configuration instance 10 1000 12.39 11.49 202.00 2.00 20.00 26.00
HTable with new Hbase configuration instance 10 1000 9.13 10.79 204.00 1.00 14.00 17.90
HTable from HTablePool (pool size: 100) 10 10000 32.41 59.62 1785.00 1.00 50.00 80.00
HTable with same configuration instance 10 10000 37.87 143.47 4911.00 2.00 51.00 84.00
HTable with new Hbase configuration instance 10 10000 12.93 11.26 212.00 2.00 20.00 24.00
HTable from HTablePool (pool size: 100) 100 100 296.71 132.66 710.00 47.00 404.80 467.50
HTable with same configuration instance 100 100 263.60 135.97 738.00 16.00 389.40 411.80
HTable with new Hbase configuration instance 100 100 7.11 8.33 48.00 2.00 10.00 12.00
HTable from HTablePool (pool size: 100) 100 1000 108.58 105.80 419.00 2.00 215.00 268.00
HTable with same configuration instance 100 1000 107.31 105.14 362.00 2.00 218.00 267.00
HTable with new Hbase configuration instance 100 1000 10.24 13.45 203.00 1.00 11.00 20.00
HTable from HTablePool (pool size: 100) 100 10000 149.96 197.28 1527.00 1.00 267.00 333.00
HTable with same configuration instance 100 10000 134.47 144.55 646.00 2.00 276.00 358.00
HTable with new Hbase configuration instance 100 10000 247.84 692.83 11043.00 1.00 370.00 668.00


HBase 0.90.1 CDH3 Standalone installation (all figures in milliseconds)
Scenario Num Threads Puts Mean Stdev Max Min 80 Pctl 90 Pctl
HTable from HTablePool (pool size: 100) 1 100 6.54 4.24 38.00 4.00 7.00 8.00
HTable with same configuration instance 1 100 5.77 1.34 10.00 4.00 7.00 8.00
HTable with new Hbase configuration instance 1 100 5.23 1.33 12.00 4.00 5.00 6.90
HTable from HTablePool (pool size: 100) 1 1000 5.50 1.62 26.00 4.00 6.00 7.00
HTable with same configuration instance 1 1000 4.95 1.35 22.00 3.00 6.00 6.00
HTable with new Hbase configuration instance 1 1000 4.94 1.31 16.00 3.00 6.00 6.00
HTable from HTablePool (pool size: 100) 1 10000 4.84 2.57 92.00 3.00 5.00 6.00
HTable with same configuration instance 1 10000 4.59 4.88 229.00 3.00 5.00 6.00
HTable with new Hbase configuration instance 1 10000 4.30 4.72 231.00 3.00 4.00 5.00
HTable from HTablePool (pool size: 100) 10 100 26.44 27.85 205.00 4.00 34.80 39.00
HTable with same configuration instance 10 100 27.50 28.68 214.00 7.00 35.00 40.00
HTable with new Hbase configuration instance 10 100 31.55 31.22 205.00 4.00 37.00 70.40
HTable from HTablePool (pool size: 100) 10 1000 25.80 14.84 204.00 3.00 36.00 41.00
HTable with same configuration instance 10 1000 26.15 15.21 212.00 3.00 37.00 41.90
HTable with new Hbase configuration instance 10 1000 25.72 14.31 199.00 3.00 37.00 43.00
HTable from HTablePool (pool size: 100) 10 10000 26.12 16.09 280.00 3.00 36.00 41.00
HTable with same configuration instance 10 10000 27.45 16.93 274.00 3.00 38.00 43.00
HTable with new Hbase configuration instance 10 10000 25.57 13.90 263.00 3.00 36.00 40.00
HTable from HTablePool (pool size: 100) 100 100 22.55 17.28 50.00 5.00 44.80 46.90
HTable with same configuration instance 100 100 132.36 77.87 481.00 18.00 197.80 241.30
HTable with new Hbase configuration instance 100 100 29.23 22.76 97.00 4.00 52.60 59.90
HTable from HTablePool (pool size: 100) 100 1000 175.42 272.73 2987.00 3.00 307.60 356.00
HTable with same configuration instance 100 1000 19.54 47.49 248.00 3.00 9.00 11.00
HTable with new Hbase configuration instance 100 1000 52.58 69.17 383.00 4.00 79.80 150.90
HTable from HTablePool (pool size: 100) 100 10000 173.08 1294.93 24606.00 4.00 117.00 148.00
HTable with same configuration instance 100 10000 57.95 43.75 414.00 4.00 93.00 113.00
HTable with new Hbase configuration instance 100 10000 180.96 163.70 937.00 4.00 334.00 424.00

I will be adding more results here: https://github.com/srikanthps/hbase-benchmarker/blob/master/Reports.TXT


Conclusions:
So, what conclusions can be drawn from above figures:

1. Write performances seems to be good when lesser threads are used for writing.
2.  All 3 variants seems to perform more or less similarly.  In our application, we use HTablePool based approach and its nice to see that it performs equally nice.  Another thing which I observed during my tests is that pool size may not have much role to play.  With pool size of 10 too, the results are more or less similar.
3. As number of threads increase, the write performance deteriorates.  A potential reason for this behavior is that HBase opens only one connection per region server from a single JVM.  So, will increasing the number of JVMs help in horizontally scaling the writes?

Under heavy multi-threading, the avg. time increases as most of the threads seems to be blocking here:
"pool-1-thread-99" prio=6 tid=0x045a4c00 nid=0x22e4 waiting for monitor entry [0x06c6f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:471) - waiting to lock <0x291f0088> (a java.io.DataOutputStream)
 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:713)
 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
 at $Proxy0.put(Unknown Source)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3$1.call(HConnectionManager.java:1242)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3$1.call(HConnectionManager.java:1240)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1035)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247)
 at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609)
 at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474)



Source code for benchmarking application:
Available in my github repository.
https://github.com/srikanthps/hbase-benchmarker

3 comments:

fernando said...

Have you thought about integrating AsyncBase with the benchmark?

Jimson said...

Hi, Have you tested the get performance like this ever?
I'm getting some weired results with get.
Is there any additional modifications needed to use multithreaded get/scan (like maximum concurrent connections etc..) to come up with increased performance than sequential read?

Thanks and Regards,
Jimson

A_Gura said...

I'm afraid that write performance isn't depend on way of HTable creating. If you move getHTable() method invocation to block code that put records in cycle, you'll, perhaps, obtain absolutly different results. Now you use one HTable instance for all puts. Pool can win in case of using new vs pooled instance HTable.