Centos 5.4 , PHP 5.3 and Harvard Referencing.

2010 March 1 at 11:45 » Tagged as :virtualization, centos, cloud, deadinedue,

A couple of days back, we did an update to Deadlinedue, the Harvard reference generator, the moment the database was updated and the new code was put in place, PHP started segfaulting. It was time to decide whether to roll back or press forward. I chose the latter and it resulted in the site been offline for around an hour.

It is not possible to assign complex types to nodes in /var/www/deadline/ISBN.php on line 480, referer: http://deadlinedue.com/index.html?lookf or=http%3A%2F%2Fraditha.com&find=find *** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0x81cdfc78 *** PHP is pretty old now, it's not something that you expect to segfault, so initially I thought the culprit would be APC - Advanced PHP Cache. These PHP accelerators or caches are known to crash every once in a while. That can be easily fixed by removing the accelerator. In this case we could afford to do so since we had upgraded the server very recently and optimized the code which resulted in a speed boost and reduced memory usage. I was barking up the wrong tree,  disabling APC didn't do any good. So guess it's time to upgrade PHP? Now you might ask, shouldn't we have used the same version of PHP in our development and production servers to ensure that this sort of thing didn't happen? Right you are but who expects PHP of all things to crash like this. It's really rediculous that deadlinedue is hosted in the cloud. So we could easily have made a snapshot of it and started another server in less than 5 minute. We could have tested with that guinea pig and then gone live, but no, I was too cocky. This is Centos 5.4 and there are no RPMs available for PHP 5.3.1, So it was time to compile from scratch. Which usually means you need to run ./configure about half a dozen times, fixing each of the missing dependencies it reports until it completes without error. Fortunately there is plenty of bandwidth on Amazon EC2 and the servers are blazing fast. Most deps can be installed in less time than it takes to type out the yum install command. After all this the problem still wasn't solved, a lot of modules we needed like JSON , DOM and heck even Mysqli were not getting added!

PHP Warning:  PHP Startup: apc: Unable to initialize modulenModule compiled with module API=20050922nPHP    compiled with module API=20090626nThese optio ns need to matchn in Unknown on line 0 PHP Warning:  PHP Startup: dbase: Unable to initialize modulenModule compiled with module API=20050922nPHP    compiled with module API=20090626nThese opt ions need to matchn in Unknown on line 0 PHP Warning:  PHP Startup: dom: Unable to initialize modulenModule compiled with module API=20050922nPHP    compiled with module API=20090626nThese optio ns need to matchn in Unknown on line 0 PHP Warning:  PHP Startup: json: Unable to initialize modulenModule compiled with module API=20050922nPHP    compiled with module API=20090626nThese opti ons need to matchn in Unknown on line 0 Given enough time, I can track down their causes and fix these errors, but time was exactly what I was not having on my hands. So tried looking around to find out if there are any RPMs available from third party repositories that would update PHP to version 5.3.1. Fortunately there was.  The repo is called Web Tactic. Update using the RPMs took just seconds and good news, no more segmentation faults. The bad news is that APC is no longer available.  I tried  I tried 'yum install php-pecl-apc'  without success. Then tried 'pecl install apc' and got the following error:

/tmp/pear/temp/APC/php_apc.c: In function 'zif_apc_compile_file': /tmp/pear/temp/APC/php_apc.c:881: warning: unused variable 'eg_class_table' /tmp/pear/temp/APC/php_apc.c:881: warning: unused variable 'eg_function_table' /tmp/pear/temp/APC/php_apc.c: At top level: /tmp/pear/temp/APC/php_apc.c:959: error: duplicate 'static' make: *** [php_apc.lo] Error ERROR: `make' failed Oh well, can live without APC for the moment.