Moment of Inertia [ Hiran ]

Stupid FUTEX and glibc

I have a multi-threaded C++ application, used as a component end for one of the integration channels. Since last 2-3 weeks, I have been facing a strange issue of application getting hanged and not responding.

Debugging started with `strace' and I found that the application hangs in futex as:

futex(0x5ac9df, FUTEX_WAIT, ....

I thought, it had something to do with pthread_mutex that I was using, to share a queue across the application. I ensured that, pthread_mutex_lock and pthread_mutex_unlock are happening correctly, since problem solving always starts with the assumption that, the problem is in your code.

Well, I did that and also put more debug statements around usage of `mutex'. Unfortunately, that did not help, since the problem persisted.

I was on the verge of restructuring the entire application, when on googling `futex_wait hangs', I stuck upon a link, where there is a discussion about the same issue what I was facing. Some text from the link:

Unfortunately, ctime() is not defined on this list. So, glibc does not guarantee the sane behavior when one uses ctime() in signal handler. BTW, I'm surprised that sysklogd calls some functions in signal handler.

Unfortunately, my application was doing the same, i.e., using a function `localtime' (which calls __libc_lock_lock() in glibc), in a signal handler. I couldn't believe it. Though the purpose of the signal handler was to clean up resources and exit the application, I was logging some data. The logging function was calling `localtime'.

Pathetically, it has not yet been fixed, as it seems that this is a problem with glibc on 2.6 kernel, and not application programmers are at the disposal of glibc or kernel developers to fix this.

An important week ahead

Next week is very important, to me and to INXS. All the hard work put into by me, by entire INXS for the last 8-10 weeks is going to see some light. It is not the end of the tunnel. But a small break in the tunnel to expose what the team has done till now.

Keeping fingers crosses, as someone said, being `cautiously optimistic'.

All the best to INXS.

GnuTLS, OpenSSL, and libgcrypt

I had written a multi-threaded application in C++, which was using libcurl and GnuTLS for making https post requests to a webservice call. Now, when I was making individual requests, it was working fine and faster than an equivalent Ruby program. When I did a load test of 300 requests per milli second, it crashed with the following error:

ath.c:184: _gcry_ath_mutex_lock: Assertion `*lock == ((ath_mutex_t) 0)' 
failed.
Aborted!!

From the assertion, it seemed to be an issue with variables not being initialized. Digging more, I found people suggesting to use libgcrypt > 1.2, which was stable and has fixed this error. Well, I was not sure, that GnuTLS was using it and neither did I know that, a multi-threaded application will have to make some `init' calls before using the library calls.

Now, I found this link, which looked to me like the solution. But, it didn't look convincing. My idea was, if a library uses another library, it should do all these initialization tasks, rather than expecting the application to do it. :)

Well, I changed from GnuTLS to OpenSSL and, my load test seemed to work, without crashing the application.

Now, I am not sure, where will I end up, if I start exploring `GnuTLS versus OpenSSL`. That is not my current priority.

Back and weak

Was away from work for almost a week due to viral fever. 2 days stuck in Chennai, and 5 days stuck in Kerala. I had gone to Kerala with that fever, for an extended weekend, hoping to return to work on Tuesday. But couldn't.

Now, I have a small pile of work, waiting for me to be consumed, and I have my health, which I do not want to take chances with. Have lost almost 3-4 kilos in the last 1 week. I can recover that in no time, but what about time. Have to answer that....!!

Making it work for different version

Few days ago, I found the solution for QuickFIX to make it work with g++ 4.3.X. I had to edit the source for QuickFIX and add two headers in src/C++/Utility.h.

#include <cstdlib>
#include <cstring>

This is fixed in the office laptop, which I am using at home. The next problem, I faced was making my FIX application to run with g++ 4.2 and 4.3 version. For this, I added the following lines:

#if ((__GNUC__ == 4) && (__GNUC_MINOR__ == 3))
#include <cstdlib>
#include <cstring>
#endif

That solved it!!

Now, whether I have a machine with quickfix installed, my application will work, irrespective of the version of g++. Hope g++ does not give more surprises.

Beginning the year with mixed events

Before, I mention, how 2009 has begun for me, I will list down something about my eventful 2008.

• my sister as well as myself fell for chicken pox, one after the other.
• 4 of my very close relatives, committing suicide.
• one of my cousin sisters getting married
• one of my best friends, escaping a not so fatal accident
• one of my best friends, effected by recession
• one of my best friends, going the family way
• some of my colleagues, including my CEO, going the family way
• INXS tying up with TD Ameritrade, and stamping on our vision for Unified Brokerage Access Channel
• launching our Mobile application for trading with TD Ameritrade.
• one of my cousins getting into PhD Sanskrit, following in the footsteps of my grandpa.
• Amma, succumbing to office politics and resigning, later coming to stay with me.

I know the list is endless.

After all this, I thought, may be this year I will see more good news than bad news. But, I was in for a surprise.

• expanded our device coverage for the trading application.
• in talk with some more big brokers, for expansion of business
• one of my cousin's marriage getting fixed
• one of my cousins, getting a divorce notice (shockingly), after 1.5 years of marriage

It has been a month in 2009, with full of mixed emotions. A prayer for giving courage and peace of mind, for things beyond our control, and a constructive attitude towards things that are is in our control, is what I am looking forward to.

I have been selfish, in mentioning events, only having direct relation to me. Those are things, which I feel, everyone is effected and does not need to be explicit.

Still, positive for the year ahead.

FIX, third party software and adoption.

Financial Information eXchange or in short FIX, is an application layer protocol, adopted by a lot of organizations for financial information exchange. Refer, FIX Protocol for references.

I was introduced to it, in October, and a months work went into integrating FIX initiator application, to communicate with FIX gateways implemented by various Brokers. I struggled a bit, since there wasn't a straight forward documentation, to some developer level faqs. More so, because, I was writing a custom implementation for a broker (name cannot be disclosed), who is supposed to have the best FIX gateway among the US Brokers (ofcourse among those, who have adopted FIX).

During development, I was using g++-4.2 version, and I didn't have issues with compiling quickfix. Today, I was trying to deploy the application in my office laptop (for testing), with a fresh ubuntu installation, with g++-4.3. Compiling quickfix gave error like:

../Utility.h:179: error: 'std::strcmp' has not been declared
../Utility.h:181: error: 'std::strlen' has not been declared
../Utility.h:184: error: 'std::memcpy' has not been declared
../Utility.h:185: error: 'std::memset' has not been declared
../Utility.h:189: error: 'std::strerror' has not been declared

I searched further and found this link. I was surprised. I saw this document on porting issues to gcc 4.3.

In the office, I have always been skeptical about using third party software, and try to follow some simple check list before adopting to one of those.

• Use if the software is well documented, with reference to existing bugs, reliability and performance issues.
• Don't use the software, if it is not being maintained for more than last 6 months.
• Avoid using application level frameworks, which is heavy weight and try to abstract a lot of programming constructs.
• See business perspective, as in if a GPL licensed software is used, then what will happen to business, when you are distributing the software.
• Use the software, if time to market is minimal.
• Use it if you cannot think of writing the same piece of software in the near future (for e.g: the linux kernel or gcc compiler.)

cache programming

The type of applications that I am writing, use enormous amounts of RAM space, for storing processed data as cached data, so as to be used by various other applications. During this time is when I remembered about the cache (L1 and L2) used by the system processor. Though both are entirely different, it is the same as seen by a user. It is memory!!!

It is known that L1 and L2 cache are faster than the RAM, which is in turn faster than disk I/O. It is also know that L1, L2 caches are expensive than RAM.

This is when I came up with the problem of what If I wanted to make an implicit write to L1/L2 cache instead of the RAM, from a user space application.

I came up with quite a few interesting reads, and a patent for it. Check them out:

• Pentium 4 L2 cache optimization for cached writes
• Computer system cache performance...

Abhiyum Naanum is Excellent....!!

Yesterday, I happened to see the movie `Abhiyum Naanum` along with Amma, at INOX. Well, I never used to get tickets for a movie, for a show, on the second day of the release. This time, I was lucky. Probably, everyone wanted to watch something else, or probably, the casting of the movie, didn't attract people. I wanted to see it because of, 1. Prakash Raj, 2. Amma is a fan of Prakash Raj, 3. The Mozhi magic, 4. was a story about a Father and a daughter.

One may feel it is a copy of `Father of the Bride 1`. Not taking any credits from Steve Martin, I would say that Prakash Raj had donned the role much better than Steve. May be the language (Tamil) and the element of the culture, and the Mozhi effect have influenced this liking.

It was a script, which can be seen through in a practical life as far as I am concerned. The journey of a father bringing up his only daughter, until she gets married. There is so much of life in the charaterization of every one. As for Trisha, this was a variety from the `glamorous ones` she has been doing until this one.

Hypocrisy in Sanskrit

Sampoorna kumbho na karoti shabdam
ardho ghato ghoshamupaiti noonam
Vidvan kuleeno na karoti garvam
gunairviheenaaha bahu jalpayanti

It is straight forward. One with complete knowledge will not make noise (or speak foolishly) and will not show pride in his ability. One with incomplete knowledge of who has no good qualities, will speak foolishly, and a lot.

In English language, it is like:

Empty vessels make the most noise.

I heard this in `Sree Guruvayurappan` serial in Surya TV. I like the serial and, try hard not to miss it.