A thinking programer's blog

Configuring LZO Compression for CDH3 HBase

2011-08-06T15:48:00.007+05:30

This post explains steps for "Configuring LZO Compression for CDH3 HBase" and it applies to CDH3 HBase 0.90.1 and above. Main source of this post is http://wiki.apache.org/hadoop/UsingLzoCompression, I have added details of steps that are needed for CDH3 (step 3) as some instructions in the wiki page did not seem to apply applicable for CDH3 HBase.

I tried below steps on Ubuntu. If you are using Linux, give Todd Lipcon's https://github.com/toddlipcon/hadoop-lzo-packager a try.

Also, adding LZO Compression may require additional configuration changes in your HBase setup. Refer http://search-hadoop.com/m/WUnLM6ojHm1/Long+client+pauses+with+compression&subj=Long+client+pauses+with+compression

Download and build http://www.oberhumer.com/opensource/lzo/. This steps are based on http://www.linuxfromscratch.org/blfs/view/cvs/general/LZO.html.

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.05.tar.gz
gunzip lzo-2.05.tar.gz
tar -xvf lzo-2.05.tar
cd lzo-2.05/
./configure --prefix=/usr --enable-shared
make
make check

All checks should pass.

make test

All tests should pass.

sudo make install

Download and build http://code.google.com/p/hadoop-gpl-compression/

svn checkout http://svn.codespot.com/a/apache-extras.org/hadoop-gpl-compression/trunk/ hadoop-gpl-compression-read-only
cd hadoop-gpl-compression-read-only/
export CFLAGS="-m64"
ant compile-native
ant jar

Copy the Hadoop GPL Compression JAR to HBase lib

cp build/hadoop-gpl-compression-0.2.0-dev.jar /path/to/hbase/lib/

Copy the native files to /usr/local/lib/

sudo cp build/native/Linux-amd64-64/lib/* /usr/local/lib/

Edit /path/to/hbase/conf/hbase-env.sh, and add /usr/local/lib to environment variable. This is needed as hbase/lib does not seem to have "native" folder anymore (Refer: http://cdh3u0.cloudera.com/cdh/3/hbase-0.90.1+15.18.releasenotes.html and https://issues.apache.org/jira/browse/HBASE-3533)

vi /path/to/hbase/conf/hbase-env.sh
Add following line at the end:

export HBASE_LIBRARY_PATH=/usr/local/lib/

Start the HBase. You should see that native lzo library is successfully loaded in logs. I was trying in stand alone mode, need to confirm whether below logs will be seen in master or region server logs.

2011-08-06 14:54:08,776 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2011-08-06 14:54:08,782 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library

Using your own log4j.properties with GridGain

2011-08-06T14:11:00.001+05:30

By default, GridGain comes with its own copy of log4j configuration in the form of "default-log4j.xml" GridGain does not allow usage of log4j.properties as GridLog4jLogger only accepts XML file as source of configuration.

However, like me, if you are adding GridGain to an application that has logging already configured using log4j.properties, then, you can use below workaround.

In the below example, we are setting gridLogger property of Grid Configuration by using an
instance of GridLogger that is constructed using a regular log4j Logger object.

Then, we have log4jInitialization bean that initialzes log4j using
log4j.properties with the help of org.springframework.util.Log4jConfigurer.

Power of Ruby and HBase shell

2011-07-17T23:28:00.001+05:30

Well, its a well known fact that Hbase shell is a Ruby IRB shell with lots of HBase convenience method loaded into it.

I had a requirement for my project wherein I wanted to check whether a row, identified by a given row key, exists in table or not. Though "get" kind of works, but it does not return true or false, instead dumps the output to standard out.

However, I wanted to read a file which consisted of a set of row keys and find out whether those row keys were present in the table or not. Since, "get" would not return a boolean, I was not able to build a logic around it.

So, I started looking into ruby code of HBase shell, and came up with following. Looks like neat little hack. But it also demonstrates the opportunities at offer to write really nice shell utilities for your project's needs.

HBase Benchmarking for multi-threaded environment

2011-06-26T22:51:00.019+05:30

This weekend I attempted to figure out how HBase writes perform in multi-threaded environments. To do so, I wrote a Java program that writes records in a HBase table, and which can be configured to run to write N number of records using M Threads.

I used three variants to write records into HBase:

Use one HTable for every write

HTable is created using a singleton instance of HBaseConfiguration
HTable is created using new instance of HBaseConfiguration (Why? - I wanted to check how the behavior changes if connections to servers are not shared. Reference: HTable and HConnectionManager documentation)

Use HTablePool

Here are the various results:

HBase 0.20.6 Standalone mode (all figures in milliseconds)
Scenario	Num Threads	Puts	Mean	Standard Deviation	Max	Min	80th Percentile	90th Percentile
HTable from HTablePool (pool size: 100)	1	100	4.86	1.49	12.00	3.00	5.00	6.90
HTable with same configuration instance	1	100	4.52	1.12	9.00	3.00	5.00	6.90
HTable with new Hbase configuration instance	1	100	4.15	0.88	8.00	3.00	4.00	5.00
HTable from HTablePool (pool size: 100)	1	1000	4.19	3.60	108.00	3.00	4.00	5.00
HTable with same configuration instance	1	1000	4.02	1.79	47.00	3.00	4.00	5.00
HTable with new Hbase configuration instance	1	1000	3.84	3.57	108.00	3.00	4.00	4.00
HTable from HTablePool (pool size: 100)	1	10000	3.95	4.13	211.00	2.00	4.00	5.00
HTable with same configuration instance	1	10000	4.28	10.02	571.00	2.00	4.00	5.00
HTable with new Hbase configuration instance	1	10000	3.80	5.46	244.00	2.00	4.00	4.00
HTable from HTablePool (pool size: 100)	10	100	24.59	22.73	204.00	4.00	38.60	44.00
HTable with same configuration instance	10	100	20.48	28.05	233.00	5.00	22.00	36.80
HTable with new Hbase configuration instance	10	100	21.44	28.05	241.00	5.00	26.80	50.90
HTable from HTablePool (pool size: 100)	10	1000	26.10	21.94	246.00	5.00	40.00	49.00
HTable with same configuration instance	10	1000	23.80	25.31	238.00	3.00	35.80	44.00
HTable with new Hbase configuration instance	10	1000	22.55	21.96	209.00	4.00	35.00	44.00
HTable from HTablePool (pool size: 100)	10	10000	22.06	18.81	279.00	3.00	36.00	43.00
HTable with same configuration instance	10	10000	22.67	18.44	287.00	3.00	37.00	45.00
HTable with new Hbase configuration instance	10	10000	24.43	26.67	507.00	3.00	38.00	46.00
HTable from HTablePool (pool size: 100)	100	100	106.45	68.53	422.00	5.00	164.60	199.40
HTable with same configuration instance	100	100	102.81	68.45	421.00	4.00	167.80	194.60
HTable with new Hbase configuration instance	100	100	25.73	20.58	209.00	4.00	29.80	35.90
HTable from HTablePool (pool size: 100)	100	1000	195.38	189.46	696.00	3.00	375.80	488.00
HTable with same configuration instance	100	1000	189.86	193.69	688.00	3.00	384.00	492.00
HTable with new Hbase configuration instance	100	1000	79.80	110.55	525.00	4.00	151.00	252.80
HTable from HTablePool (pool size: 100)	100	10000	219.65	214.10	1277.00	3.00	433.00	543.00
HTable with same configuration instance	100	10000	277.60	473.36	4129.00	3.00	453.80	609.00
HTable with new Hbase configuration instance	100	10000	214.80	230.87	1386.00	3.00	435.00	566.00

HBase 0.20.6 Distributed Mode with 3 Region Servers (Hadoop 0.20) (all figures in milliseconds)
Scenario	Num Threads	Puts	Mean	Standard Deviation	Max	Min	80th Percentile	90th Percentile
HTable from HTablePool (pool size: 100)	1	100	3.25	1.96	18.00	2.00	3.00	3.90
HTable with same configuration instance	1	100	2.98	1.18	9.00	2.00	3.00	4.00
HTable with new Hbase configuration instance	1	100	2.91	1.43	10.00	1.00	3.00	4.90
HTable from HTablePool (pool size: 100)	1	1000	4.56	3.77	40.00	1.00	6.00	11.00
HTable with same configuration instance	1	1000	4.98	4.04	20.00	1.00	11.00	11.00
HTable with new Hbase configuration instance	1	1000	2.28	1.20	9.00	1.00	3.00	3.00
HTable from HTablePool (pool size: 100)	1	10000	2.48	1.76	41.00	1.00	3.00	3.00
HTable with same configuration instance	1	10000	2.43	1.78	49.00	1.00	3.00	3.00
HTable with new Hbase configuration instance	1	10000	2.84	2.79	52.00	1.00	3.00	5.00
HTable from HTablePool (pool size: 100)	10	100	15.88	21.62	204.00	2.00	23.80	32.90
HTable with same configuration instance	10	100	13.09	20.50	204.00	2.00	18.00	20.00
HTable with new Hbase configuration instance	10	100	8.78	19.64	197.00	2.00	8.80	13.00
HTable from HTablePool (pool size: 100)	10	1000	12.36	10.76	203.00	2.00	19.00	22.00
HTable with same configuration instance	10	1000	12.39	11.49	202.00	2.00	20.00	26.00
HTable with new Hbase configuration instance	10	1000	9.13	10.79	204.00	1.00	14.00	17.90
HTable from HTablePool (pool size: 100)	10	10000	32.41	59.62	1785.00	1.00	50.00	80.00
HTable with same configuration instance	10	10000	37.87	143.47	4911.00	2.00	51.00	84.00
HTable with new Hbase configuration instance	10	10000	12.93	11.26	212.00	2.00	20.00	24.00
HTable from HTablePool (pool size: 100)	100	100	296.71	132.66	710.00	47.00	404.80	467.50
HTable with same configuration instance	100	100	263.60	135.97	738.00	16.00	389.40	411.80
HTable with new Hbase configuration instance	100	100	7.11	8.33	48.00	2.00	10.00	12.00
HTable from HTablePool (pool size: 100)	100	1000	108.58	105.80	419.00	2.00	215.00	268.00
HTable with same configuration instance	100	1000	107.31	105.14	362.00	2.00	218.00	267.00
HTable with new Hbase configuration instance	100	1000	10.24	13.45	203.00	1.00	11.00	20.00
HTable from HTablePool (pool size: 100)	100	10000	149.96	197.28	1527.00	1.00	267.00	333.00
HTable with same configuration instance	100	10000	134.47	144.55	646.00	2.00	276.00	358.00
HTable with new Hbase configuration instance	100	10000	247.84	692.83	11043.00	1.00	370.00	668.00

HBase 0.90.1 CDH3 Standalone installation (all figures in milliseconds)
Scenario	Num Threads	Puts	Mean	Stdev	Max	Min	80 Pctl	90 Pctl
HTable from HTablePool (pool size: 100)	1	100	6.54	4.24	38.00	4.00	7.00	8.00
HTable with same configuration instance	1	100	5.77	1.34	10.00	4.00	7.00	8.00
HTable with new Hbase configuration instance	1	100	5.23	1.33	12.00	4.00	5.00	6.90
HTable from HTablePool (pool size: 100)	1	1000	5.50	1.62	26.00	4.00	6.00	7.00
HTable with same configuration instance	1	1000	4.95	1.35	22.00	3.00	6.00	6.00
HTable with new Hbase configuration instance	1	1000	4.94	1.31	16.00	3.00	6.00	6.00
HTable from HTablePool (pool size: 100)	1	10000	4.84	2.57	92.00	3.00	5.00	6.00
HTable with same configuration instance	1	10000	4.59	4.88	229.00	3.00	5.00	6.00
HTable with new Hbase configuration instance	1	10000	4.30	4.72	231.00	3.00	4.00	5.00
HTable from HTablePool (pool size: 100)	10	100	26.44	27.85	205.00	4.00	34.80	39.00
HTable with same configuration instance	10	100	27.50	28.68	214.00	7.00	35.00	40.00
HTable with new Hbase configuration instance	10	100	31.55	31.22	205.00	4.00	37.00	70.40
HTable from HTablePool (pool size: 100)	10	1000	25.80	14.84	204.00	3.00	36.00	41.00
HTable with same configuration instance	10	1000	26.15	15.21	212.00	3.00	37.00	41.90
HTable with new Hbase configuration instance	10	1000	25.72	14.31	199.00	3.00	37.00	43.00
HTable from HTablePool (pool size: 100)	10	10000	26.12	16.09	280.00	3.00	36.00	41.00
HTable with same configuration instance	10	10000	27.45	16.93	274.00	3.00	38.00	43.00
HTable with new Hbase configuration instance	10	10000	25.57	13.90	263.00	3.00	36.00	40.00
HTable from HTablePool (pool size: 100)	100	100	22.55	17.28	50.00	5.00	44.80	46.90
HTable with same configuration instance	100	100	132.36	77.87	481.00	18.00	197.80	241.30
HTable with new Hbase configuration instance	100	100	29.23	22.76	97.00	4.00	52.60	59.90
HTable from HTablePool (pool size: 100)	100	1000	175.42	272.73	2987.00	3.00	307.60	356.00
HTable with same configuration instance	100	1000	19.54	47.49	248.00	3.00	9.00	11.00
HTable with new Hbase configuration instance	100	1000	52.58	69.17	383.00	4.00	79.80	150.90
HTable from HTablePool (pool size: 100)	100	10000	173.08	1294.93	24606.00	4.00	117.00	148.00
HTable with same configuration instance	100	10000	57.95	43.75	414.00	4.00	93.00	113.00
HTable with new Hbase configuration instance	100	10000	180.96	163.70	937.00	4.00	334.00	424.00

I will be adding more results here: https://github.com/srikanthps/hbase-benchmarker/blob/master/Reports.TXT

Conclusions:
So, what conclusions can be drawn from above figures:

1. Write performances seems to be good when lesser threads are used for writing.
2. All 3 variants seems to perform more or less similarly. In our application, we use HTablePool based approach and its nice to see that it performs equally nice. Another thing which I observed during my tests is that pool size may not have much role to play. With pool size of 10 too, the results are more or less similar.
3. As number of threads increase, the write performance deteriorates. A potential reason for this behavior is that HBase opens only one connection per region server from a single JVM. So, will increasing the number of JVMs help in horizontally scaling the writes?

Under heavy multi-threading, the avg. time increases as most of the threads seems to be blocking here:
"pool-1-thread-99" prio=6 tid=0x045a4c00 nid=0x22e4 waiting for monitor entry [0x06c6f000]
java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:471) - waiting to lock <0x291f0088> (a java.io.DataOutputStream)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:713)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy0.put(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3$1.call(HConnectionManager.java:1242)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3$1.call(HConnectionManager.java:1240)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1035)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$3.doCall(HConnectionManager.java:1239)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1161)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1247)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:609)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:474)

Source code for benchmarking application:
Available in my github repository.
https://github.com/srikanthps/hbase-benchmarker

HBase Issue - HMaster not starting with error "Cannot assign requested address"

2011-06-26T18:35:00.000+05:30

Today I ran into an issue with my working HBase standalone node after my Ubuntu PC got a new IP address due to restart.

HBase master logs showed:
2011-06-26 17:38:13,125 INFO org.apache.hadoop.hbase.master.HMaster: My address is ubuntu.ubuntu-domain:60000
2011-06-26 17:38:13,250 ERROR org.apache.hadoop.hbase.master.HMaster: Can not start master
java.net.BindException: Problem binding to /192.168.1.2:60000 : Cannot assign requested address
    at org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java:179)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.(HBaseServer.java:242)

I noticed that my current IP address was 192.168.1.3, not 1.2 which HBase was trying to bind to.

Ping to my host name showed IP address which HBase was trying to use, but "ifconfig" showed the new IP Address:

srikanth@ubuntu:~/hadoop/hbase-0.20.6$ ping ubuntu.ubuntu-domain
PING ubuntu.ubuntu-domain (192.168.1.2) 56(84) bytes of data.
64 bytes from ubuntu.ubuntu-domain (192.168.1.2): icmp_seq=1 ttl=64 time=12.2 ms
64 bytes from ubuntu.ubuntu-domain (192.168.1.2): icmp_seq=2 ttl=64 time=1.17 ms

srikanth@ubuntu:~$ ifconfig
eth0      Link encap:Ethernet HWaddr 00:1c:c0:4d:74:86
          inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0
          inet6 addr: fe80::21c:c0ff:fe4d:7486/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:6518 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6501 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:6421057 (6.4 MB) TX bytes:865876 (865.8 KB)
          Memory:e0400000-e0420000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:421 errors:0 dropped:0 overruns:0 frame:0
          TX packets:421 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:52862 (52.8 KB) TX bytes:52862 (52.8 KB)

I tried following after some googling but had no success:
1. Disabled IPv6
2. Try to clear the DNS Cache. And in this process, also learned that Ubuntu does not do any DNS caching.
3. Tried to assign static IP Address so that HBase is happy, though this worked for HBase, but my machine went out of network (no internet, and my laptop could not SSH to it).
4. Called up my friend who is Linux expert.

Later I decided to update /etc/hosts to map my host name to my new IP address. It was only then did I realize that it already had an entry that was pointing my host name to my previous IP address which was causing all the mess. I updated it to use my current IP address, and things started working fine again. Phew!!

Ruby Application to tweet messages using twitter

2011-06-18T23:42:00.001+05:30

Using twitter_oauth gem, I wrote an application that can be used in non-web-based application to integrate twitter. Using the concepts demonstrated in this sample application, one can generate tweets on whenever an interesting application events occurs. For example, a shopping site can tweet about a new category of products that gets available.

I will try to make it into a gem soon, and may be rails plugin too, something like "ActionTweeter" similar to "ActionMailer".

Here is the GIST URL: https://gist.github.com/1033288

98th Thing every programmer should know

2010-11-15T12:06:00.000+05:30

If you provide a code review comment, follow it up with other developer to confirm whether it was implemented or not. Many a times, a comment gets lost in email chain and never gets implemented.

It is better if you can use a code review tool that integrates well with the IDE.

99th thing every programmer should know

2010-11-15T10:35:00.000+05:30

Avoid unnecessary acronmym-ization of words.

Eg: This is the name of the constant CHART_DTL_MAIN some one used in the code.

Here "DTL" is actually an acronmy invented by programmer for the word "DETAIL". By saving 3 characters, programmer has made variable not easy to interpret.

It's better to avoid using unnecssary acronyms and use as descriptive words as possible as there are typically no restrictions in the language.

Ruby Frameworks that I wish to learn more about..

2010-09-29T00:25:00.000+05:30

Paperclip
Activemerchant
Attachment_fu
cancan
factory girl
arel
radiant cms
devise
authlogic
mongomapper

Ruby Eigen Class

2010-08-16T22:38:00.000+05:30

Ruby Eigen Classes are created in following scenarios:

1. When you define a instance method (singleton method) on an object, that object's class is an Eigen class.
2. When you include a module in a class, that class's super class is an Eigen class.

Any other??

A readable piece of source code

2010-07-20T21:54:00.001+05:30

Other day, I wrote a piece of code similar to one below, when I had to intialize the password column in Users table with a default password.

String initialPassword = hashOf(firstSixChars(userName));

I liked the idea that code reads like a prose. I have been constantly practicing to write such code.

After all, code should be written less for machines and more for other programmers who are going to read it.

Sending emails using gmail SMTP Server (in Java)

2010-07-17T22:10:00.000+05:30

After doing a bit of search on google, I found here a code snippet in the comments section that can be used to send emails through gmail. There was a minor typo in the code snippet in the "main" function, where class name was wrongly spelt as "MailUtils" instead of "SimpleMail".

This code depends on Java Mail API. I picked the "mail.jar" from Spring distribution's "lib" directory.

Thats it! I was sending emails from Java in no time.

Code sample is available as my github GIST here. http://gist.github.com/479631

Basecamp like domain names for your SaaS Users

2010-07-12T23:31:00.003+05:30

I was wondering how one would go about implementing a custom domain for each user of a SaaS application. For example, I need to visit https://srikanthps.basecamphq.com/login if I need to login to BaseCamp. This is very neat.

After doing a bit of googling, I found following blog entries which gave me some idea on how one can achieve something similar to base camp.
Changes needed in a RAILS App (Post by DHH): This should be enough if you are hosting the app on your own servers.
http://37signals.com/svn/posts/1512-how-to-do-basecamp-style-subdomains-in-rails

If you are planning to use a hosting provider, such as Heroku, then, read this post, http://docs.heroku.com/custom-domains, especially the part about wild card domains.

Sending emails in RAILS using gmail SMTP Server

2010-07-11T22:06:00.003+05:30

I implemented email support in Bug Manager (http://bugmanager.heroku.com/), my application I developed to learn RAILS.

I followed the instructions from here http://blog.heroku.com/archives/2009/11/9/tech_sending_email_with_gmail/ and here (SMTP error while using Gmail in Rails | Inspired? No).

Works like charm!

Hosting RAILS app on Heroku can't be any simpler. I just followed the instructions here (http://docs.heroku.com/quickstart).

My github repositories

2009-12-08T01:41:00.002+05:30

Started adding stuff to my github repository (http://github.com/srikanthps).

Migrated my "bugs on rails" project from Google code to github

Also, started a new adventure in terms of "cvsstat" project. Was fed up today using the other cvsstat which kept reporting me wrong data, so I thought let me write a simple program to extract the data I need from cvs logs. It's current state is such that even I can't use it, but I hope to polish it in a day or two.

My Macbook Pro

2009-09-27T19:05:00.026+05:30

After thinking about it for more than a month now, I could not resist, and bought myself a Macbook Pro this Saturday. It's a 13" one, with 2.53 GHz Intel Core2 Duo, and 4GB RAM. I also got "Snow Leopard" upgrade disk with it, so my Macbook Pro is now running the latest and greatest Mac OS.

Bought it at Imagine Store, Forum Mall, Bangalore. The staff there was very friendly and helped me with registering the Mac (which I did at the store), and also showed me how to go about upgrading the OS.

Here are some pictures.

]

At last, I have become productive in using RAILS.

2009-07-31T01:57:00.007+05:30

I started working on a RAILS project to solve a problem at work. I was finding it difficult to manage tracking of defects. So, I started putting together a small tool that can help me share information on bugs within the team so that team members can pick up bugs to fix, and everyone is at same page when it comes to status.

My this little adventure with RAILS is shaping into something interesting. At present, I have made this application Open source on Google Code. My friend, Anish, and I are trying to make it a general purpose application. We are still a long way to go, but its a start!

To justify the blog title, I was able to put together a basic "Task board" in a matter of just 2 hours. At present, I am just punching the code without worrying much about automated test cases. Cucubmer, WebRat, RSpec are in my TODO list. My initial goal is to attain some mastery in doing basic RAILS right!.

I am following Elisabeth Hendrikson's suggestion on how to do TDD on a new technology. She suggests that one should build a throw-away app first to get a good hang of a new technology before one can do TDD right in that technology.

Here is her answer to my query on how to do TDD on a new technology:

Elisabeth responds:
Yes, I ran into this problem recently in my own work when using a new library. The recipe for TDDing something in a new technology where you’re still trying to figure out how the technology works is to spike, learn, shred, and then TDD it right. That means you start by hacking together something that works well enough so you can learn how the technology works. Then you shred that initial solution, ripping (or commenting) out all that hacked together code. And start over test-first. Does it take longer? Sure. But the result is more maintainable, testable software with fewer bugs, and also much deeper learning.

Practices for becoming a better programmer

2009-01-30T00:17:00.002+05:30

Today, at my work place, I presented a on the topic "Practices for becoming a better programmer" to the new campus recruits. This presentation was aimed at sharing some of the best practices which every programmer should know about.

Examples in these slides came from two major sources: Essential Skills for Agile Development and Clean Code. Topics themselves have already been discussed and blogged about by many on the internet.

You are free to use these slides to present on similar topics. My slides are available here.

Scrum in 15 Minutes

2008-12-11T23:37:00.003+05:30

Scrum and 15 minutes are related to each other.
Daily Scrum meetings are time boxed to 15 minutes.

So, when I could not get enough audience for my 1.5 hour presentation on Scrum at my organization, I realized that if I want to be heard, I will have to talk less.

Hence, I cut down all the fluff from the presentation slides to come up with a bare minimum set of slides that can introduce Scrum in just 15 minutes.

You can download the slides from here.

Slides are based on Mike Cohn's work and I do not claim any credits for these slides.
Like Mike's slides, these are also free for everyone to use as long as credits to Mike Cohn's work is not removed from the slides.

TDD Session

2008-11-19T01:44:00.005+05:30

Today, I had a real TDD session while I was coding a piece for upcoming release.

I did a classical TDD (I think):
1. Wrote failing test first.
2. Coded a bit. Made the test pass.
3. Wrote another test(s).
4. Coded some more. Made the test(s) pass.
5. Left the day with a failing test.

I think, it was Kent Beck, who in his book, Test Driven Development - By Example, has mentioned that it is a good practice to leave the day with a failing test. The failing test is a good way to remember "what were you upto yesterday" when you start the day next morning.

I used Agitar JUnit runner, and it showed me code coverage done by my tests. This helped me to not only write more tests to have 100% code coverage, but also made me remove some redundant conditional checks as the those checks were evaluating to "true" always.

Scrum Adoption

2008-11-19T01:12:00.010+05:30

I have initated adoption of Scrum in my team. To start with, we are trying to bring Scrum way of doing things in release and iteration planning.

We have had 2-week iterations in our project, and we have been having a release every 2 months. However, I felt that we have not been planning the work for iterations very effectively.

Now we have started to do following:

Do a story point estimate of high level deliverables (Stories) for the 2 month phase of the project.
Split the stories based on story points across the three 2-week iterations which we will have in 2 month release. One of the goals of allocating stories to various iterations is to ensure that we have equal distribution of story points across iterations.
Break-up the stories selected for current iteration into tasks. For this, we create tasks on post-it notes, estimate each task in ideal hours, and then put it on task board.
Team signs-up for tasks and moves them from "Not checked-out" to "Checked-out" status in Task boards.
We have daily standup at afternoon 2.30PM. We discuss the status, plan for the day and also, move the completed tasks to "Done" status.
We are doing pair-programming selectively wherever there is a need for knowledge exchange or tasks which we think need could be done faster if two members looked at it.

Yesterday, I presented the Scrum to the team. The slides were based on Mike Cohn's slides. While preparing for the presentation, I had a chance to read the Harvard Business Review , The New New Product Development Game, by Hirotaka Takeuchi and Ikujiro Nonaka. It's a great paper and I liked the parts where they talk about "Self-organizing teams" and "Multilearnings".

Notes from Ken Schwaber presentation on Scrum

2008-10-26T22:47:00.044+05:30

Source: Google Video Presentation by Ken Schroder on Scrum

* Everything in Scrum is time-boxed.
* It's the team which has to get things done by the end of timebox, and that should be potentially shippable.
* Scrum is about managing product development, and XP is about engineering practices for product development.
* Scrum is not a methodology, Scrum is like a simple process framework, analogous to rules of chess (which are simple, but can be used to develop complex strategies).
* Scrum brings in transparency, you know where you are and where you are heading at any given point of time.

* Product manager defines the backlog.

* Team decides the contents of a Sprint along with product manager by picking items from Product backlog.

* Sprint is somewhere between 2 to 4 weeks.
* Team should have minimum or no interupptions during the sprint.
* After the sprint, the backlog is revised and re-prioritized.

* Burndown chart as indicators for velocity gives transparent view on whether we are going to meet the marketing's deadline or not.

* Traditionally, Pressure to increase velocity can result in developers reducing the quality and/or working for long hours, and that will only result in much more slower velocity.

* More than 65% of functionality is rarely used or not used at all. Hence, product manager should select only high value items and leave out the rest from sprint. Hence, instead of trying to increase the velocity, try to re-prioritize the backlog or see if some requirements can be trimmed down.

* Architecture and design are driven through Non-functional requirements. This type of work can be done in first, second and other sprints, but all sprints much deliver business value, and should demonstrate that architecture infrastructure is working.

Avoid using names for HTML input elements that match HTML FORM object's attribute

2008-10-04T01:10:00.005+05:30

At work, other day we encounted a strange issue.

We have this web application which is based on Spring Web Flow, and each of the views in the flow have process buttons such as "Back", "Skip", "Cancel", and "Submit" (for main action).

In order to resolve an issue, We modified handling of Back, and Cancel buttons from using HTTP POST to use HTTP GET. In other words, instead of submitting the form to the Spring WEB Flow controller with one of the request parameter being _eventId (represented through a hidden field in form), We changed the Back and Cancel button's click handlers to use "window.location = document.forms[0].action + '&_eventId=back'" and "window.location = document.forms[0].action + '&_eventId=cancel'" respectively.

For most of the pages, this code just worked fine. However, for few pages related to certain functionality, the above code did not work. With the help of alert messages, We realized that browser was not treating document.forms[0].action as string containing action URL, but was tricked into thinking that document.forms[0].action was an object.

That behaviour seemed pretty strange as the same code was working fine for 90% of pages.

After the investigations, we found that there was an HTML input element in the page with name "action". The browser hence was returning reference to that object instead of the value of "action" URL.

This was a nice learning. And the lesson is to avoid using names for any of the HTML input elements (Input box, check box, selects, and other types of input elements) that matches the FORM object's attributes.

http://www.srikanthps.com is online now

2008-09-04T00:40:00.004+05:30

Thanks to a enlightening blog entry by Jay Fields - Be your own startup , I got my domain registered and now it's pointing to my blog. First step taken care, it's now my turn to follow on other two advices:

Start presenting
Start contributing to open source projects.

Let me see how far I will be succesful in following the above two advices.

Got running first RAILS sample app

2008-09-04T00:06:00.008+05:30

It's not really an achievement to get a sample app running. But it almost became frustrating difficult to get it running.

I have been reading RUBY FOR RAILS book by David A. Black. Since I was nearing the end, and the section four was talking about improving the RAILS app developed in chapter 2, I just wanted to get the sample app from chapter 2 running.

I already had latest version of RAILS installed on my computer. I followed the instructions in the book to create an app, and controllers and typed in the contents for the all controller .rb files and view .(r)html files. In essence, I did pretty much what the book said (or so I had thought).

When I ran the application, as usual, it started throwing errors at me. I had to following modifications to run the app:

RAILS was configured to use Sqlite3. I initially thought its a driver (thanks to my Java backround) and did not care much. But after googling, I found that I had to change config/database.yml to use MySQL. I had to google again to get hold of typical MySQL configuration. Here it is for your reference:
```
development:
  adapter: mysql
  host: localhost
  database: r4rmusic1_development
  username: railsuser
  password: railsuser
```
My first mistake was that I had copy pasted the line helper :all line already present in the app/controllers/application.rb to define the default layout. I had used layout :base. This was causing undefined method `base' for # error. I did google, but no help. Somehow, I stumbled upon the mistake I had done, and fixed it by using layout "base" in application.rb.
I thought I have got everything right now, but application started reporting now a different error. Missing layout layouts/base.html.erb in view path C:/srikanth-development/rails-workspace/r4rmusic1/app/views I did google again, but not any "bulls-eye" type of help. So, I broke my head for 15 to 20 minutes to get this working. I observed that RAILS was complaining about base.html.erb and I had base.rhtml. It took me some time but I figured that RAILS had changed extension convention for ERB files to .html.erb. My Windows Vista File explorer had that option turned on "Hide extensions of well known files" which kept me confused for some time as I observed only .html files in RAILS generated views. But once, I had .html.erb extension for all view files, things started progressing.
There were many typo in my code and I had to fix all of them to get the app finally up and running.

Getting RAILS installed and getting an app running always seems challenging. But I guess its a matter of getting used to its nuances.