<html>

<head></head>

<body>
    <div class="item">
        <div class="item-header">



        </div>

        <div class="item-body">

            Perhaps the title should have been; 'the lack of a suitable compressed file system on linux'. A compressed file system in this case refers to a setup where the files are saved on the disk in a predefined compressed format (such as gzip or bzip2). When you read from those files they will be automatically decompressed by the file system. Similarly when you attempt to create a&nbsp; new file or modify an old file, it should be automatically compressed before saving. Such a file system is sure to be very slow for random access but for sequential access it wouldn't matter so much. It might even be faster than an uncompressed file system because hard drives continues to be the real bottleneck in most computers today.

            Linux gives you two options for creating file systems; at the kernel level or in the user space.&nbsp; <a href="http://e2compr.sourceforge.net/">e2compr</a> is a kernel patch that supports compressing an ext2 file system while the user space tools are based on <a href="http://fuse.sourceforge.net/">fuse</a>. Using kernel space drivers could be messy if the file system chosen like e2compr is always playing catchup with the kernel version. The only compressed file systems supported in the mainstream kernel are <a href="http://sources.redhat.com/jffs2/">jffs2</a> and <a href="http://squashfs.sourceforge.net/">squashfs</a>.&nbsp; The former is for flash drives and the latter is ready only. Though <a href="http://jfs.sourceforge.net/">JFS</a> is available for linux, it's compression mode is not.&nbsp; While both KDE's Dolphin and Gnome's Nautilus support mounting a gzipped tar archive of a zip file as a node in the file system in read/write mode, they are not suitable for web apps. So fuse seems&nbsp; the way to go.

            There are eight file systems listed on the <a href="https://sourceforge.net/apps/mediawiki/fuse/index.html?title=CompressedFileSystems">Fuse compressed file system page</a>. Four are read only, two are abandoned, two more are heading down that route. One is in it's early development stage. The remainder was FuseCompress.

            This one is real easy to install just do 'yum install fusecompress' setup is even easier.
            <p style="padding-left: 30px;">
                fusecompress&nbsp; /mnt/compressed&nbsp;&nbsp; /mnt/uncompressed</p>


            Copying over a set of 30,000 files (a mix of text and binary data) with a total weight of 85MB took 36 seconds. The same operation when repeated, completed in 20 seconds because of file system caching.&nbsp; When the files were copied without compression the operation completed in 1 second, so much for my assumption that a compressed system would be faster.

        </div>
    </div>


</body>

</html>

Compressed File Systems on Linux

<html>

<head></head>

<body>
    <div class="item">
        <div class="item-header">



        </div>

        <div class="item-body">

            <p></p>
            <p style="padding-left: 30px;"><span style="color: #993300;">configure error: unknown plugin: sphinx</span></p>
            <p></p>

        </div>
    </div>


</body>

</html>

Mysql Sphinx Storage Engine

<html>

<head></head>

<body>
    <div class="item">
        <div class="item-header">



        </div>

        <div class="item-body">

            A couple of days back, we did an update to <a href="http://deadlinedue.com/">Deadlinedue</a>, the Harvard reference generator, the moment the database was updated and the new code was put in place, PHP started segfaulting. It was time to decide whether to roll back or press forward. I chose the latter and it resulted in the site been offline for <a href="http://deadlinedue.com/blog/more-accurate-citations.html">around an hour</a>.
            <p style="padding-left: 30px;">I<span style="color: #993300;">t is not possible to assign complex types to nodes in /var/www/deadline/ISBN.php on line 480, referer: http://deadlinedue.com/index.html?lookf
                    or=http%3A%2F%2Fraditha.com&amp;find=find
                    *** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0x81cdfc78 ***</span>

                PHP is pretty old now, it's not something that you expect to segfault, so initially I thought the culprit would be APC - Advanced PHP Cache. These PHP accelerators or caches are known to crash every once in a while. That can be easily fixed by removing the accelerator. In this case we could afford to do so since we had upgraded the server very recently and optimized the code which resulted in a speed boost and reduced memory usage. I was barking up the wrong tree,&nbsp; disabling APC didn't do any good. So guess it's time to upgrade PHP?

                Now you might ask, shouldn't we have used the same version of PHP in our development and production servers to ensure that this sort of thing didn't happen? Right you are but who expects PHP of all things to crash like this. It's really rediculous that deadlinedue is hosted in the cloud. So we could easily have made a snapshot of it and started another server in less than 5 minute. We could have tested with that guinea pig and then gone live, but no, I was too cocky.

                This is <a href="https://centos.org/">Centos</a> 5.4 and there are no RPMs available for PHP 5.3.1, So it was time to compile from scratch. Which usually means you need to run ./configure about half a dozen times, fixing each of the missing dependencies it rep<span style="color: #000000;">o</span>rts until it completes without error. Fortunately there is plenty of bandwidth on <a href="https://aws.amazon.com/ec2/">Amazon EC2</a> and the servers are blazing fast. Most deps can be installed in less time than it takes to type out the yum install command.

                After all this the problem still wasn't solved, a lot of modules we needed like JSON , DOM and heck even Mysqli were not getting added!
            </p>
            <p style="padding-left: 30px;"><span style="color: #993300;">PHP Warning:&nbsp; PHP Startup: apc: Unable to initialize modulenModule compiled with module API=20050922nPHP&nbsp;&nbsp;&nbsp; compiled with module API=20090626nThese optio
                    ns need to matchn in Unknown on line 0
                    PHP Warning:&nbsp; PHP Startup: dbase: Unable to initialize modulenModule compiled with module API=20050922nPHP&nbsp;&nbsp;&nbsp; compiled with module API=20090626nThese opt
                    ions need to matchn in Unknown on line 0
                    PHP Warning:&nbsp; PHP Startup: dom: Unable to initialize modulenModule compiled with module API=20050922nPHP&nbsp;&nbsp;&nbsp; compiled with module API=20090626nThese optio
                    ns need to matchn in Unknown on line 0
                    PHP Warning:&nbsp; PHP Startup: json: Unable to initialize modulenModule compiled with module API=20050922nPHP&nbsp;&nbsp;&nbsp; compiled with module API=20090626nThese opti
                    ons need to matchn in Unknown on line 0</span>

                Given enough time, I can track down their causes and fix these errors, but time was exactly what I was not having on my hands. So tried looking around to find out if there are any RPMs available from third party repositories that would update PHP to version 5.3.1. Fortunately there was.&nbsp; The repo is called <a href="https://www.webtatic.com/blog/2009/06/php-530-on-centos-5/">Web Tactic</a>. Update using the RPMs took just seconds and good news, no more segmentation faults. The bad news is that APC is no longer available.&nbsp; I tried&nbsp; I tried 'yum install php-pecl-apc'&nbsp; without success. Then tried 'pecl install apc' and got the following error:
            </p>
            <p style="padding-left: 30px;"><span style="color: #993300;">/tmp/pear/temp/APC/php_apc.c: In function 'zif_apc_compile_file':
                    /tmp/pear/temp/APC/php_apc.c:881: warning: unused variable 'eg_class_table'
                    /tmp/pear/temp/APC/php_apc.c:881: warning: unused variable 'eg_function_table'
                    /tmp/pear/temp/APC/php_apc.c: At top level:
                    /tmp/pear/temp/APC/php_apc.c:959: error: duplicate 'static'
                    make: *** [php_apc.lo] Error
                    ERROR: `make' failed</span>

                Oh well, can live without APC for the moment.

            </p>
        </div>
    </div>


</body>

</html>

Monthly Archives for 2010 03

Compressed File Systems on Linux

Mysql Sphinx Storage Engine

Centos 5.4 , PHP 5.3 and Harvard Referencing.