Moment of Inertia [ Hiran ]

DES with openssl v/s DESCryptoServiceProvider in .NET

Recently, I had to deal with some encryption, for which the documentation included some piece of code in VB, which used DESCryptoServiceProvider class for encryption. After spending almost a day trying to understand the implementation, I have come up with the equivalent implementation in C with openssl.

The stupidest thing, which I had to scratch my head for majority of the time spent on was the problem that, why is the VB code generating a 16-byte encrypted data for an input of length 8-bytes. I found the answer to this here.

Until now, I had been using the DES_* functions. After realizing that 8-byte padding is required, I had to use low-level EVP_* functions.

Some piece of code is given here:

EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
EVP_EncryptInit_ex(&ctx, EVP_des_cbc(), NULL, key, iv);

EVP_CIPHER_CTX_set_padding(&ctx, 8);
//output_length and output_length1 need to be different.
EVP_EncryptUpdate(&ctx, output, &output_length, input, input_length);
EVP_EncryptFinal_ex(&ctx, output+output_length, &output_length1);
output_length += output_length1;

Phew...!!

gcc 4.5.1

GCC 4.5.1 was released last week. See the release notes for the changes.

One thing which caught was eyes was the introduction of the experimental profile mode.

It is true that programmers do misuse the STL. There should be some way to correct it. Now, need to get my hands dirty with the profile mode and see the usefulness.

be64toh - Another problem with gcc versions

Recently, I have been integrating another API, in which data transmission is in binary format, and as usual, the format is big-endian. Sounds familiar!!

Generally, in C, you have functions ntohl, ntohs, for converting long integer and short integer values from big-endian to the format of host machine. And obviously, they are more portable way of handling the conversion, rather that your test and shift method of conversion.

Now this time, I had to deal with some 64-bit data and was wondering if there were any functions, and I found the manual pages for endian. This was cool. A week later, I was ready to deploy for testing and got stuck with the problem of the function be64toh not being defined.

The problem was my development environment used g++ 4.4.1 and the test environment used g++ 4.3.2. Ohh god!!! What now? I tried searching for equivalent ones, and found some macros in /usr/include/byteswap.h.

So, now I modified my C++ code to look like this:

double readDouble(void * & data)
{
        uint64_t *ptr = (uint64_t *)data;
#if ( (__GNUC__ == 4) && (__GNUC_MINOR__ == 3) && \
             (__GNUC_PATCHLEVEL__ > 2) )
        uint64_t value = be64toh( *ptr );
#elif __BYTE_ORDER == __LITTLE_ENDIAN
        uint64_t value = bswap_64( *ptr );
#else
        uint64_t value = *ptr;
#endif
        ptr++;
        data = ptr;
        double *dptr = (double *)&value;
        return *dptr;
}

Code is now deployed for testing. :-)

ntohl and the left shift

Recently, I got into a small argument about an efficient code in converting big-endian data to little-endian. I was using `ntohl' library call and the argument against it was to use left shift and sum operation.

//Use ntohl
void *readUint32(void *ptr, uint32_t *value)
{
	uint32_t *uint_ptr = (uint32_t *)ptr;
	*value = (uint32_t)ntohl( *ptr );
	ptr += sizeof(uint32_t);
	return ptr;
}

//Use left shift
unsigned char *readUint32ByShift(unsigned char *p,unsigned int *n) 
{
        unsigned int v = 0;
        v <<= 8; v += *p++;
        v <<= 8; v += *p++;
        v <<= 8; v += *p++;
        v <<= 8; v += *p++;
        *n = v;
        return p;
}

This is with the implementation of a streaming market data with responses in network-byte (big-endian) order. Since it is time critical market data, the application should be faster in parsing and dispatching the data to end client.

The argument was that using `ntohl' was unnecessary, since the code is always going to run in x86 machine (little-endian) and will be faster than the library call implementation. Well, at that time, I didn't know how `ntohl' is implemented, and neither did I know if a left-shift will be faster than the library call. I did a benchmark and found that the library call implementation was twice as fast as the left-shift implementation.

Well, I got my answer.

Stupid FUTEX and glibc

I have a multi-threaded C++ application, used as a component end for one of the integration channels. Since last 2-3 weeks, I have been facing a strange issue of application getting hanged and not responding.

Debugging started with `strace' and I found that the application hangs in futex as:

futex(0x5ac9df, FUTEX_WAIT, ....

I thought, it had something to do with pthread_mutex that I was using, to share a queue across the application. I ensured that, pthread_mutex_lock and pthread_mutex_unlock are happening correctly, since problem solving always starts with the assumption that, the problem is in your code.

Well, I did that and also put more debug statements around usage of `mutex'. Unfortunately, that did not help, since the problem persisted.

I was on the verge of restructuring the entire application, when on googling `futex_wait hangs', I stuck upon a link, where there is a discussion about the same issue what I was facing. Some text from the link:

Unfortunately, ctime() is not defined on this list. So, glibc does not guarantee the sane behavior when one uses ctime() in signal handler. BTW, I'm surprised that sysklogd calls some functions in signal handler.

Unfortunately, my application was doing the same, i.e., using a function `localtime' (which calls __libc_lock_lock() in glibc), in a signal handler. I couldn't believe it. Though the purpose of the signal handler was to clean up resources and exit the application, I was logging some data. The logging function was calling `localtime'.

Pathetically, it has not yet been fixed, as it seems that this is a problem with glibc on 2.6 kernel, and not application programmers are at the disposal of glibc or kernel developers to fix this.

An important week ahead

Next week is very important, to me and to INXS. All the hard work put into by me, by entire INXS for the last 8-10 weeks is going to see some light. It is not the end of the tunnel. But a small break in the tunnel to expose what the team has done till now.

Keeping fingers crosses, as someone said, being `cautiously optimistic'.

All the best to INXS.

GnuTLS, OpenSSL, and libgcrypt

I had written a multi-threaded application in C++, which was using libcurl and GnuTLS for making https post requests to a webservice call. Now, when I was making individual requests, it was working fine and faster than an equivalent Ruby program. When I did a load test of 300 requests per milli second, it crashed with the following error:

ath.c:184: _gcry_ath_mutex_lock: Assertion `*lock == ((ath_mutex_t) 0)' 
failed.
Aborted!!

From the assertion, it seemed to be an issue with variables not being initialized. Digging more, I found people suggesting to use libgcrypt > 1.2, which was stable and has fixed this error. Well, I was not sure, that GnuTLS was using it and neither did I know that, a multi-threaded application will have to make some `init' calls before using the library calls.

Now, I found this link, which looked to me like the solution. But, it didn't look convincing. My idea was, if a library uses another library, it should do all these initialization tasks, rather than expecting the application to do it. :)

Well, I changed from GnuTLS to OpenSSL and, my load test seemed to work, without crashing the application.

Now, I am not sure, where will I end up, if I start exploring `GnuTLS versus OpenSSL`. That is not my current priority.

Back and weak

Was away from work for almost a week due to viral fever. 2 days stuck in Chennai, and 5 days stuck in Kerala. I had gone to Kerala with that fever, for an extended weekend, hoping to return to work on Tuesday. But couldn't.

Now, I have a small pile of work, waiting for me to be consumed, and I have my health, which I do not want to take chances with. Have lost almost 3-4 kilos in the last 1 week. I can recover that in no time, but what about time. Have to answer that....!!

Making it work for different version

Few days ago, I found the solution for QuickFIX to make it work with g++ 4.3.X. I had to edit the source for QuickFIX and add two headers in src/C++/Utility.h.

#include <cstdlib>
#include <cstring>

This is fixed in the office laptop, which I am using at home. The next problem, I faced was making my FIX application to run with g++ 4.2 and 4.3 version. For this, I added the following lines:

#if ((__GNUC__ == 4) && (__GNUC_MINOR__ == 3))
#include <cstdlib>
#include <cstring>
#endif

That solved it!!

Now, whether I have a machine with quickfix installed, my application will work, irrespective of the version of g++. Hope g++ does not give more surprises.

Beginning the year with mixed events

Before, I mention, how 2009 has begun for me, I will list down something about my eventful 2008.

• my sister as well as myself fell for chicken pox, one after the other.
• 4 of my very close relatives, committing suicide.
• one of my cousin sisters getting married
• one of my best friends, escaping a not so fatal accident
• one of my best friends, effected by recession
• one of my best friends, going the family way
• some of my colleagues, including my CEO, going the family way
• INXS tying up with TD Ameritrade, and stamping on our vision for Unified Brokerage Access Channel
• launching our Mobile application for trading with TD Ameritrade.
• one of my cousins getting into PhD Sanskrit, following in the footsteps of my grandpa.
• Amma, succumbing to office politics and resigning, later coming to stay with me.

I know the list is endless.

After all this, I thought, may be this year I will see more good news than bad news. But, I was in for a surprise.

• expanded our device coverage for the trading application.
• in talk with some more big brokers, for expansion of business
• one of my cousin's marriage getting fixed
• one of my cousins, getting a divorce notice (shockingly), after 1.5 years of marriage

It has been a month in 2009, with full of mixed emotions. A prayer for giving courage and peace of mind, for things beyond our control, and a constructive attitude towards things that are is in our control, is what I am looking forward to.

I have been selfish, in mentioning events, only having direct relation to me. Those are things, which I feel, everyone is effected and does not need to be explicit.

Still, positive for the year ahead.

FIX, third party software and adoption.

Financial Information eXchange or in short FIX, is an application layer protocol, adopted by a lot of organizations for financial information exchange. Refer, FIX Protocol for references.

I was introduced to it, in October, and a months work went into integrating FIX initiator application, to communicate with FIX gateways implemented by various Brokers. I struggled a bit, since there wasn't a straight forward documentation, to some developer level faqs. More so, because, I was writing a custom implementation for a broker (name cannot be disclosed), who is supposed to have the best FIX gateway among the US Brokers (ofcourse among those, who have adopted FIX).

During development, I was using g++-4.2 version, and I didn't have issues with compiling quickfix. Today, I was trying to deploy the application in my office laptop (for testing), with a fresh ubuntu installation, with g++-4.3. Compiling quickfix gave error like:

../Utility.h:179: error: 'std::strcmp' has not been declared
../Utility.h:181: error: 'std::strlen' has not been declared
../Utility.h:184: error: 'std::memcpy' has not been declared
../Utility.h:185: error: 'std::memset' has not been declared
../Utility.h:189: error: 'std::strerror' has not been declared

I searched further and found this link. I was surprised. I saw this document on porting issues to gcc 4.3.

In the office, I have always been skeptical about using third party software, and try to follow some simple check list before adopting to one of those.

• Use if the software is well documented, with reference to existing bugs, reliability and performance issues.
• Don't use the software, if it is not being maintained for more than last 6 months.
• Avoid using application level frameworks, which is heavy weight and try to abstract a lot of programming constructs.
• See business perspective, as in if a GPL licensed software is used, then what will happen to business, when you are distributing the software.
• Use the software, if time to market is minimal.
• Use it if you cannot think of writing the same piece of software in the near future (for e.g: the linux kernel or gcc compiler.)