Site With The Lamp

A couple of days back, we did an update to Deadlinedue, the Harvard reference generator, the moment the database was updated and the new code was put in place, PHP started segfaulting. It was time to decide whether to roll back or press forward. I chose the latter and it resulted in the site been offline for around an hour.

It is not possible to assign complex types to nodes in /var/www/deadline/ISBN.php on line 480, referer: http://deadlinedue.com/index.php?lookf
or=http%3A%2F%2Fraditha.com&find=find
*** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0×81cdfc78 ***

PHP is pretty old now, it’s not something that you expect to segfault, so initially I thought the culprit would be APC – Advanced PHP Cache. These PHP accelerators or caches are known to crash every once in a while. That can be easily fixed by removing the accelerator. In this case we could afford to do so since we had upgraded the server very recently and optimized the code which resulted in a speed boost and reduced memory usage. I was barking up the wrong tree,  disabling APC didn’t do any good. So guess it’s time to upgrade PHP?

Now you might ask, shouldn’t we have used the same version of PHP in our development and production servers to ensure that this sort of thing didn’t happen? Right you are but who expects PHP of all things to crash like this. It’s really rediculous that deadlinedue is hosted in the cloud. So we could easily have made a snapshot of it and started another server in less than 5 minute. We could have tested with that guinea pig and then gone live, but no, I was too cocky.

This is Centos 5.4 and there are no RPMs available for PHP 5.3.1, So it was time to compile from scratch. Which usually means you need to run ./configure about half a dozen times, fixing each of the missing dependencies it reports until it completes without error. Fortunately there is plenty of bandwidth on Amazon EC2 and the servers are blazing fast. Most deps can be installed in less time than it takes to type out the yum install command.

After all this the problem still wasn’t solved, a lot of modules we needed like JSON , DOM and heck even Mysqli were not getting added!

PHP Warning:  PHP Startup: apc: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese optio
ns need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: dbase: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese opt
ions need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: dom: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese optio
ns need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: json: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese opti
ons need to match\n in Unknown on line 0

Given enough time, I can track down their causes and fix these errors, but time was exactly what I was not having on my hands. So tried looking around to find out if there are any RPMs available from third party repositories that would update PHP to version 5.3.1. Fortunately there was.  The repo is called Web Tactic. Update using the RPMs took just seconds and good news, no more segmentation faults. The bad news is that APC is no longer available.  I tried  I tried ‘yum install php-pecl-apc’  without success. Then tried ‘pecl install apc’ and got the following error:

/tmp/pear/temp/APC/php_apc.c: In function ‘zif_apc_compile_file’:
/tmp/pear/temp/APC/php_apc.c:881: warning: unused variable ‘eg_class_table’
/tmp/pear/temp/APC/php_apc.c:881: warning: unused variable ‘eg_function_table’
/tmp/pear/temp/APC/php_apc.c: At top level:
/tmp/pear/temp/APC/php_apc.c:959: error: duplicate ’static’
make: *** [php_apc.lo] Error 1
ERROR: `make’ failed

Oh well, can live without APC for the moment.

If you add yourself to a list and then call the get list members method you might be surprised by the result.

One of the data items returned by the get list id method includes the number of members in your list. The get list members method returns 20 members at a time and you need to call the it multiple times using different cursor locations to retrieve the complete membership. When you count them and compare against the number returned by the get list id method, you will find that they do not match. Your own account is not part of the dataset that is returned.

By the time you are reading this post, the list might have changed. So I have saved a copy of the XML here. On the other hand when you access the list through twitter.com, you can see that I am included in my own list. In fact my screen name (e4c5) appears twice!

The work around I suppose is to not rely on the members count returned by the get list id method.

Amazon has been offering cloud hosting services for longer than most unfortuntely they don’t seem to have updated their Amazon Machine Images since they started operation. Well, it’s not exactly accurate, they have reasonably upto date images for some linux distributions and for windows. However many of the linux images are actually owned by third parties and not by amazon itself. As for my favourite distribution Fedora, what’s available is Fedora 8! After reading the discussion at http://developer.amazonwebservices.com/connect/message.jspa?messageID=141707 I thought to try my hand at creating my own Fedora 11 or 12 AMI.

Unfortunately I didn’t meet with a lot of success the Fedora 11 AMI refused to boot but trouble started even before, first with yum reported numerous depsolving errors. I tried to sort them out by using the –skip-broken flag to no avail.

Total size: 252 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
libcrypto.so.7 is needed by (installed) httpd-tools-2.2.14-1.fc10.i386
libssl.so.7 is needed by (installed) httpd-tools-2.2.14-1.fc10.i386
Complete!
(1, [u'Please report this error in http://yum.baseurl.org/report'])

I worked around this by uninstalling the complaining RPM (yum remove httpd-tools-2.2.14-1.fc10.i386) and putting it back after the upgrade had completed. Nevertheless the machine reused to boot up. I tried to do this as a two step process, stopping at F10, booting up that image and then upgrading to F11. Unfortunately that didn’t quite work out either. The terminal displayed by the Amazon Management Console remains blank without anything ever showing up on it. Then I tried changing the RAM Disk and Kernel,  tried  using aki-20c12649 as the kernel and ari-21c12648 as the ramdisk with the same result.

In the end I gave up on this temporarily and used a third party Centos 5.4 image (uploaded by RightScale) for the task at hand. Going to come back to this problem later. Want to try installing a new image on the loopback device.

It’s been a while since Aptana 2.0 came out. I ignored it till version 2.02 was out. As usually happens with most software, what worked in previous versions doesn’t work in the new version. Now if this had been with version 2.0, you could have just said someone forgot to do a regression test but not so with 2.02

So what doesn’t work? The Adobe AIR debugger. It gets stuck at ‘Launching New_Configuration’ and this dialog pops up without the air app starting up.

Sounds like a socket error but this is running on localhost and all my ports are open to the local machine. As usual when I run into a problem there is nothing on google and there is nothing on the log file. At this stage I would gladly switch to Netbeans. I use for almost everything other than air. Unfortunatley I never figured out how to use the netbeans debugger with AIR. Couldn’t find any howto either.

Next I came across a reference to a third party debugger that you can use with any AIR app, regardless of the IDE that you use. It looks like a great project. Unfortunately the developers forget to create some documentation. There isn’t even a getting started guide.

I am on holiday so I am doing this on my Mac the only options available are to go back to version 1.5 or to wait till I get back home next week.