All this started when I decided to write a small script to analyze the content of a database using python rather than my usual first choice PHP. But I haven’t really coded anything at all with Python since I learned it nearly two years ago. So I thought, why not use Python, well the time I spent researching the cause of this problem could have been utilized to write the darned thing in PHP, Perl or Java or heck even C.

The last time, I did anything with Python, I ran the scripts either from the command line or using mod_python. I can’t remember the editor or IDE I was using at the time but I pretty sure it wasn’t Netbeans. The problem pops up only when you try to run the script from inside Netbeans itself. The very same script can be executed from the shell using the python interpreter (see, what did I tell you about mysqldb being installed?)

Searching for Import Error: No module named MySQLdb netbeans came up with a blank. Digging a bit deeper, I realized that Netbeans using the jython intepreter rather than the default python 2.6 interpreter that’s available on Fedora. That in turn enabled me to dig up a post on Stackoverflow which explained that MySQLdb cannot be used with jython. You need to use something called zxJDBC instead. Damned if I am going to bother with that. Instead I just changed the python platform and solved it easily.

To change the platform, Right Click on the projects root node (the project name in the ‘projects tab’), when the dialog pops up select the ‘python’ item from the tree on the left and then change the platform.

Changing the Python Platform (click to enlarge)

If by some chance, the installed python interpreter does not show up here, click the manage button and on the new dialog, choose the ‘auto-detect’ button. If today isn’t your lucky day, netbeans will refuse to find your default python interpreter. But that’s ok, just run to the command line do a `which python` and then use the path to the interpreter as reported by which to add a ‘New’ platform.

Adding a new python platform (click to enlarge)

Hopefully you might find my older post ImportError: No module named dom.minidom also to be usefull

PHP Warning: SimpleDB::batchPutAttributes(): ServiceUnavailable: Service AmazonSimpleDB is currently unavailable. Please try again later
in /var/www/****/SimpleDB/sdb.php on line 1021
Items FAILED
ErrorCode: ServiceUnavailable

However at about the time that twitter went offline, we noticed almost every other insert producing an error. When that happens we have coded our script to sleep a few seconds and retry the insert. The second attempt was completed successfully almost everytime. I went to tweet about it to find that twitter has gone offline.

AFAIK, twitter has not publicly stated what their database platform is. They might do so at chirp so for the moment I am going to hazard a guess that they are using SimpleDB to store some of their data. All their static content is served out of Amazon S3 so this is not a wild guess (I hope). I am now going to leap to the conclusion that the two errors are related and make a fool of myself.

It is not possible to assign complex types to nodes in /var/www/deadline/ISBN.php on line 480, referer: http://deadlinedue.com/index.php?lookf
or=http%3A%2F%2Fraditha.com&find=find
*** glibc detected *** /usr/sbin/httpd: double free or corruption (fasttop): 0×81cdfc78 ***

PHP is pretty old now, it’s not something that you expect to segfault, so initially I thought the culprit would be APC – Advanced PHP Cache. These PHP accelerators or caches are known to crash every once in a while. That can be easily fixed by removing the accelerator. In this case we could afford to do so since we had upgraded the server very recently and optimized the code which resulted in a speed boost and reduced memory usage. I was barking up the wrong tree,  disabling APC didn’t do any good. So guess it’s time to upgrade PHP?

Now you might ask, shouldn’t we have used the same version of PHP in our development and production servers to ensure that this sort of thing didn’t happen? Right you are but who expects PHP of all things to crash like this. It’s really rediculous that deadlinedue is hosted in the cloud. So we could easily have made a snapshot of it and started another server in less than 5 minute. We could have tested with that guinea pig and then gone live, but no, I was too cocky.

This is Centos 5.4 and there are no RPMs available for PHP 5.3.1, So it was time to compile from scratch. Which usually means you need to run ./configure about half a dozen times, fixing each of the missing dependencies it reports until it completes without error. Fortunately there is plenty of bandwidth on Amazon EC2 and the servers are blazing fast. Most deps can be installed in less time than it takes to type out the yum install command.

After all this the problem still wasn’t solved, a lot of modules we needed like JSON , DOM and heck even Mysqli were not getting added!

PHP Warning:  PHP Startup: apc: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese optio
ns need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: dbase: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese opt
ions need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: dom: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese optio
ns need to match\n in Unknown on line 0
PHP Warning:  PHP Startup: json: Unable to initialize module\nModule compiled with module API=20050922\nPHP    compiled with module API=20090626\nThese opti
ons need to match\n in Unknown on line 0

Given enough time, I can track down their causes and fix these errors, but time was exactly what I was not having on my hands. So tried looking around to find out if there are any RPMs available from third party repositories that would update PHP to version 5.3.1. Fortunately there was.  The repo is called Web Tactic. Update using the RPMs took just seconds and good news, no more segmentation faults. The bad news is that APC is no longer available.  I tried  I tried ‘yum install php-pecl-apc’  without success. Then tried ‘pecl install apc’ and got the following error:

/tmp/pear/temp/APC/php_apc.c: In function ‘zif_apc_compile_file’:
/tmp/pear/temp/APC/php_apc.c:881: warning: unused variable ‘eg_class_table’
/tmp/pear/temp/APC/php_apc.c:881: warning: unused variable ‘eg_function_table’
/tmp/pear/temp/APC/php_apc.c: At top level:
/tmp/pear/temp/APC/php_apc.c:959: error: duplicate ’static’
make: *** [php_apc.lo] Error 1
ERROR: `make’ failed

Oh well, can live without APC for the moment.

Users don’t need any special effort for Oauth but developers do. Writing an application that uses OAuth instead of passwords is supposed to take just a wee bit more effort, (but all seasoned programmers know what exactly that means).

One of the issues we ran into was caused by Twitter returning an ‘Incorrect signature’ error for List Delete member api calls. The signature was being generated using the TwitterOauthLib. The signature was correct. It worked fine for all other methods excepts for DELETE calls. If you passed in the same parameters to a different Oauth library, you would get the exact same signature so are we seeing a bug in the twitter API?  You might also suspect that this is because the underlying HTTP client library didn’t support the DELETE HTTP method. In this case it’s curl and the DELETE method is supported. Nevertheless I tried using an HTTP POST while sending the _method=DELETE parameter as suggested by the Twitter API documentation with out any luck.

On further investigation I found that all the parameters were being posted for both DELETE and POST requests. RFC 2616 doesn’t specify how you should pass parameters (request body or query string) for DELETE requests. I decided to change the TwitterOauth code to include the parameters as query strings (get the patch). Sure enough twitter accepted the request without the incorrect signature error.  The modified code now looks like this:

  function http($url, $method, $postfields = NULL) {

    $ci = curl_init();

    /* Curl settings */

    curl_setopt($ci, CURLOPT_CONNECTTIMEOUT, $this->connecttimeout);

    curl_setopt($ci, CURLOPT_TIMEOUT, $this->timeout);

    curl_setopt($ci, CURLOPT_RETURNTRANSFER, TRUE);

    curl_setopt($ci, CURLOPT_HTTPHEADER, array('Expect:'));

    curl_setopt($ci, CURLOPT_SSL_VERIFYPEER, $this->ssl_verifypeer);

    switch ($method) {

      case 'POST':

        curl_setopt($ci, CURLOPT_URL, $url);

        curl_setopt($ci, CURLOPT_POST, TRUE);

        if (isset($postfields)) {

          curl_setopt($ci, CURLOPT_POSTFIELDS, $postfields);

        }

        break;

      case 'DELETE':

        curl_setopt($ci, CURLOPT_CUSTOMREQUEST, 'DELETE');

        if (isset($postfields)) {

            curl_setopt($ci, CURLOPT_URL, "{$url}?{$postfields}");

        }

        else

        {

            curl_setopt($ci, CURLOPT_URL, $url);

        }

        break;

      default:

            curl_setopt($ci, CURLOPT_URL, $url);

            break;

    }

    $response = curl_exec($ci);

    $this->http_code = curl_getinfo($ci, CURLINFO_HTTP_CODE);

    $this->last_api_call = $url;

    curl_close ($ci);

    //if ($this->format === 'json' && $this->decode_json) {

    //  return json_decode($response);

    //}

    return $response;

  }

Also had to do a modification to the oAuthRequest method, which was automatically appending the twitterId to the url it’s not required (and in fact leads to an incorrect link for list methods). That means if you have been using an unpatched TwitterOauth library and have been passing in the twitterId as a parameter, you will have to change that.

Oh and btw, follow me on twitter

Since PHP added support for JSON encode and decode a lot has been written about how it’s more efficient than serialize / unserialize and the have got the numbers to proove it too. In one study it was pointed out that json_encode takes 0.63 milli seconds to convert a certain object to text while the serialize function took just 0.41 milliseconds however that author suggests that all existing code that uses serialize/unserialize should be changed to json_encode/json_decode because decode/unserialize more often than encode/serialize and json_decode takes just 1.42ms vs 2.28 ms for unserialize. Another mutt had reproduced the same sort of results on eight different servers. Sure that’s about a 50% improvement in speed. And if your script just happens to be executed 1,000,000 times you will save a whopping 860 seconds! you will probably get a 100% increment at your next evaluation too.

Most often you want to convert and object to a string because you want to send it over the wire or save it in a database. If your applications appears to be slow it’s probably because the database query is slow or it takes a lot of time to retrieve the data from the remote server. If it takes two seconds to retrieve/store  the data what difference does it make to shave of 0.0005 of a second by changing from json to php serialize?

I am not saying you should not make any effort to optimize your code at all. Wordpress and Windows OS are examples that scream out for optimization. What I am saying is that you should write your code with a decent algorithm (but don’t rupture yourself looking for one), then test your app. If you find it’s slow and doesn’t scale well, run it through a profiler to find the bottleneck. You will usually find that about 20% of the code takes up about 80% of the execution time sometims it’s worse. All you need to do is to look at that part of the code and optimize it or perhaps even change the algorithm.

If you are building a web services client with PHP, you can try to combine the PHP HTTP wrapper the zlib extension to read and deflate gzip compressed data. One php.net contributed note on php.net suggests to ‘daisy chain’ the zlib:// and http:// wrappers ().

<?
echo file_get_contents(“compress.zlib://http://raditha.com/”, “r”);
?>

Another approach is to rely on gzopen.

<?

$zp = gzopen(“http://raditha.com/”, “r”);
gzpassthru($zp);

?>

They will both echo out the HTML of the web page at http://raditha.com/ – but appearances can be deceptive. If you analyze the traffic with wireshark (or any old packet sniffer), you will find that the ‘Accept-Encoding: gzip,deflate‘ header is not being sent and it’s counterpart ‘Content-Encoding: gzip‘ is not included in the response. The above code appears to work because the PHP Zlib extension will open uncompressed files without complaining. In other words, when it encounters uncompressed data, gzopen behaves exactly like fopen and gzread behaves like fread.

So if you want to benefit from the deflate module on your webserver, you will need to use your own client library. How to create a client library is beyond the scope of this post, but I will explain how you can set and get the headers and decode the data. In order to make sure that the server compresses the content you need to do:

fwrite($this->socket, “Accept-Encoding: gzip,deflate\r\n”);

Usually images and other binary files will not be gzipped, so you need to make sure that the server sends back the ‘Content-Encoding: gzip’ header before you actually try to decompress the message body.

$this->headerArray = split("\r\n", $this->headers);
foreach($this->headerArray as $head)
{
    $parts = split(": ",$head);
    if(strtolower($parts[0]) == 'content-encoding: gzip')
    {
        $gzip=true;
    }
}

It’s not a requirement to convert the headers to lower case before comparision; we are just playing safe because sometime you find weird servers not capitalizing the second word in the header. Once you have identified that the content is gzipped, you might be tempted to just use the gzinflate() method to inflate what you read in from the socket.

gzinflate(stream_get_contents($this->socket, $length-10));

you will only run into an error

PHP Warning:  gzinflate(): data error in /var/www/clients/twitter/RadHTTPClient.php on line 261
PHP Stack trace:
PHP   1. {main}() /var/www/raditha/http-deflate.php:0
PHP   2. RadHTTPClient->getMessageBody() /var/www/raditha/http-deflate.php::20
PHP   3. gzinflate() /var/www/raditha/RadHTTPClient.php:261

It happens because the gzinflate() function doesn’t expect to deal with the gzip header. The header contains the gzip magic number as well as other flags such as file modification time etc. This header has to be stripped out before you pass it into the gzinflate function.

if($useGzip)
{
    $b = fread($this->socket,2);
    if(ord($b[0]) == 0x1f && ord($b[1]) == 0x8b)
    {
        $gzHead = fread($this->socket,8);
        $s = gzinflate(stream_get_contents($this->socket, $length-10));
    }
}

if the magic number 0×1f8b is not found you can treat the data as already uncompressed data (of course you will need to add the ‘else’ to go with it).

One last thing; if you want to enable one the fly gzip compression with Apache all you need to do is to add the following directives to the httpd.conf file

LoadModule deflate_module modules/mod_deflate.so

AddOutputFilterByType DEFLATE text/html text/plain text/xml

I never made use of this feature for two reasons, the first being that my wordpress installation is subversioned (revision control with subversion) and the second being that only FTP and FTP+SSL were supported by it. FTP+SSL isn’t so bad but FTP is totally insecure. Eventually I decided to use this update system for the plugins at least and thought I should add support for SFTP. The plan was to create a plugin that would add an SSH option to the upgrader. It was then that I found that SFTP support is already there but hasn’t been enabled!

On my desktop computer (Fedora 11), where I keep a copy of all my websites, I had the SFTP option enabled in a less than a minute but on my production server it took a while longer. That was because because libssh2, which is the key requirement for the PHP SSH extension could not be installed with YUM. (This is a Centos server). While attempting to use YUM, I had to run a load of updates and worst of all reboot the server (something which hasn’t been done for more than an year).

libssh2/php ssh extention had to be installed manually and I followed the instructions in the /wp-admin/includes/class-wp-filesystem-ssh2.php file – the same set of instructions are found at http://kevin.vanzonneveld.net/techblog/article/make_ssh_connections_with_php/. Unfortunately they both contain an error.

cd /usr/src
wget http://surfnet.dl.sourceforge.net/sourceforge/libssh2/libssh2-0.14.tar.gz
tar -zxvf libssh2-0.14.tar.gz
cd libssh2-0.14/
./configure
make all install

The libssh2 download is really ancient – it wouldn’t even compile on my Centos box – worse gcc even segfaulted while trying to compile it. I kid you not. I have never seen gcc segfault before.

make[1]: Entering directory `/root/libssh2-0.14/src’
gcc -o channel.o channel.c -c -g -O2 /usr/include -I/usr/include -Wall -I../include/ -fPIC
gcc: /usr/include: linker input file unused because linking not done
channel.c:1253:10: /usr/include: No such file or directory
channel.c:71: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See for instructions.
make[1]: *** [channel.o] Error 1
make[1]: Leaving directory `/root/libssh2-0.14/src’
make: *** [all] Error 1

The correct way to go about it is to visit the libssh2 project page on sourceforge and download the latest version (happens to be 1.1 at the time of writing).  The latest version could be compiled without any problems. After that you can do the pecl installation followed by editing the php.ini file

pecl install -f ssh2

wordpress and ssh

In my case, I had to give the full path to the ssh2.so file because the engine couldn’t find the extension when only a relative path was given. That’s because pear seems to have installed the shared object at a wierd location. Well in the end after the apachectl restart I now have the PHP SSH extension enabled and wordpress update page does show the ssh2 extension.

Both these articles were written quite a few years ago and Interestingly I have received a handful of hate mail from people who spend all their time optimizing code!

If you want your code to execute fast, design a good algorithm or better still look one up in a book. There is no substitute for it. If your app is connected to a database make sure it’s properly designed. If you are retrieving data from a remote server use a cache. All this applies not just to PHP but to any language on any platform.

Normally, in order to upgrade, all you need to do is to delete the old stuff, copy over the set of scripts from the new version and invoke upgrade.php . But if you recursively delete all the old stuff you lose your version control info. So this is my approach to upgrading a customized WP installation which has been subversioned.

1) copy the files that you have modified into a safe place (themes and plugins will make up the bulk of this)

2) Download wordpress and unzip it, then change into the source directory and use rsync to copy over the new files while deleting obsolete files from the previous version. the –exclude ‘.svn’ part will ensure that subversion info is not discard. If you are using another revision control system, you can use the appropriate directory here

rsync . /var/www/raditha/blog/ -avz –exclude ‘.svn’ –delete

3) Then add new files into the repository with

svn add `svn stat | grep ‘^\?’ | cut -d’ ‘ -f8`

4) copy back the themes and plugins (inclusive of the .svn folders).

4) delete the obsolete ones with:

svn rm `svn stat | grep ‘^\!’ | cut -d’ ‘ -f7`

5) Merging – unfortunately there is no easy recipe for this, you need to manually Kompare , Meld or diff the files to see if your code needs to be merged.

6) commit.

7) run upgrade.php

httperf --hog --client=0/1 --server=radmedia --port=80 --uri=/blog/ --send-buffer=4096 --recv-buffer=16384 --num-conns=50 --num-calls=1
Maximum connect burst length: 1

Total: connections 50 requests 50 replies 50 test-duration 11.002 s

Connection rate: 4.5 conn/s (220.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 213.1 avg 220.0 max 358.2 median 217.5 stddev 20.1
Connection time [ms]: connect 0.3
Connection length [replies/conn]: 1.000

Request rate: 4.5 req/s (220.0 ms/req)
Request size [B]: 66.0

Reply rate [replies/s]: min 4.4 avg 4.5 max 4.6 stddev 0.1 (2 samples)
Reply time [ms]: response 160.5 transfer 59.2
Reply size [B]: header 245.0 content 22111.0 footer 2.0 (total 22358.0)
Reply status: 1xx=0 2xx=50 3xx=0 4xx=0 5xx=0

CPU time [s]: user 2.67 system 8.16 (user 24.2% system 74.2% total 98.4%)
Net I/O: 99.5 KB/s (0.8*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

I tested several times and too the mean which is exactly the same as the one produced by the above test : a very poor 4.5 requests per second. You can see just how bad this really is by using httperf to retrieve a static page on the same site.

httperf --hog --client=0/1 --server=radmedia --port=80 --uri=/blog.html --send-buffer=4096 --recv-buffer=16384 --num-conns=50 --num-calls=1
Maximum connect burst length: 1

Total: connections 50 requests 50 replies 50 test-duration 0.111 s

Connection rate: 450.9 conn/s (2.2 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1.3 avg 2.2 max 41.2 median 1.5 stddev 5.6
Connection time [ms]: connect 0.3
Connection length [replies/conn]: 1.000

Request rate: 450.9 req/s (2.2 ms/req)
Request size [B]: 70.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 0.5 transfer 1.5
Reply size [B]: header 274.0 content 22111.0 footer 0.0 (total 22385.0)
Reply status: 1xx=0 2xx=50 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.02 system 0.09 (user 19.8% system 80.2% total 100.1%)
Net I/O: 9887.7 KB/s (81.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

For this test, I used wget to download the blog main page and copied the resultant HTML back on the server. That way, I had an piece of static HTML that is exactly the same as the blog main page. Again, what’s produced is a typical result the mean turned out to be 447 connections a second. Yes your eyes do not lie. There is a hundred fold difference in performance.

Now I compressed the wp_posts table using a few lines of PHP code and added a plugin with a ‘posts_results’ filter to decompress it again. Here is a typical load test report:

httperf --hog --client=0/1 --server=radmedia --port=80 --uri=/blog/ --send-buffer=4096 --recv-buffer=16384 --num-conns=50 --num-calls=1
Maximum connect burst length: 1

Total: connections 50 requests 50 replies 50 test-duration 10.990 s

Connection rate: 4.5 conn/s (219.8 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 213.4 avg 219.8 max 231.6 median 219.5 stddev 4.2
Connection time [ms]: connect 0.3
Connection length [replies/conn]: 1.000

Request rate: 4.5 req/s (219.8 ms/req)
Request size [B]: 66.0

Reply rate [replies/s]: min 4.4 avg 4.5 max 4.6 stddev 0.1 (2 samples)
Reply time [ms]: response 159.8 transfer 59.7
Reply size [B]: header 245.0 content 22110.0 footer 2.0 (total 22357.0)
Reply status: 1xx=0 2xx=50 3xx=0 4xx=0 5xx=0

CPU time [s]: user 2.63 system 8.18 (user 23.9% system 74.4% total 98.4%)
Net I/O: 99.6 KB/s (0.8*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

As you can see there isn’t much of a difference. Compression of the wp_posts table doesn’t lead to any change in the number of average requests per second it’s stil 4.5

What you see here is a speed test carried out using a free tool by Speedtest.net. It’s rather predictably named as speedtest mini. At first glance these results are impressive it shows that I have a very fast Local Area Network (184 mbps down and 277 mbps up) but it’s a lie. I do have a very fast LAN but this figure is a huge understatement. My LAN is faster. Much much faster. I have gigabite ethernet and the actual speed is at least three times the speed that’s reported.

The result is inaccurate because the test method isn’t very scientific. It works by downloading several large files from the web server ( in my case hosted on a server in the LAN). File access is limited by the speed of the hard drives. Unfortunately most hard drives cannot deliver at 100 mbps – the upper limit for gigabit ethernet. Hard drives operate much slower.

For broadband speedtest which is what most people use the tool for – the hard drives will not act as a bottleneck because broadband links typically operate at much much slower speeds and will not place an undue burden on the hard drives (even if multiple users are trying the test concurrently).

For the upload test, the widget POSTs some data to the server. The data appears to be generated internally or it’s uploading a file that it just cached into memory. The result is that the hard drive bottleneck doesn’t play such a significant role  – at least not for smaller uploads which are held in memory at the server rather than being saved to disk. And speed test doesn’t try to upload a lot of data at once anyway because most servers are not equipped to handle a large incoming upload (see this link if you want to learn how to configure your server : http://www.radinks.com/upload/config.php

So how do I know that my LAN really is as fast I as I claim it to be. I used this simple PHP scripts (which mimics a 128MB file being downloaded to test it out)

$s=”0″;

for($i = 0 ; $i < 16 ; $i++)
{
$s .= $s;
}

for($i = 0 ; $i < 2048 ; $i++)
{
echo $s;
}
?>

And this is what wget reported:

Saving to: `nofile.php’

[ <=> ] 67,108,864 52.2M/s in 1.2s

2009-01-04 14:45:13 (52.2 MB/s) – `nofile.php’ saved [67108864]

52.2 MB/s that’s already three times as fast as what speedtest reports but this is not accurate either. That’s because wget is still saving the file to the local drive (which again acts as the bottleneck). Even that can be avoided by redirecting output to /dev/null/

wget http://radmedia/speedtest/nofile.php -O /dev/null

That gives a slightly faster result
Saving to: `/dev/null’

[ <=> ] 134,219,776 69.0M/s in 1.9s

2009-01-04 14:56:25 (69.0 MB/s) – `/dev/null’ saved [134219776]

This result seems a lot more realistic. Yet I am disapointed that it’s not close to the 100 MB/s mark which his about the max that you can get with gigabit ether. Do I have so much of noise in my cable to wipe off about 30% of the bandwidth?

Database error

A database query syntax error has occurred. This could be because of an illegal search query (see Searching Raditha.com), or it may indicate a bug in the software. The last attempted database query was:

INSERT INTO archive (ar_namespace,ar_title,ar_text,ar_comment,ar_user,ar_user_text,ar_timestamp,ar_minor_edit,ar_flags) SELECT cur_namespace,cur_title,cur_text,cur_comment,cur_user,cur_user_text,cur_timestamp,cur_minor_edit,0 FROM cur WHERE cur_namespace=3 AND cur_title='Alis'

from within function “Article::doDeleteArticle“. MySQL returned error “1114: The table 'archive' is full“.

There has been so much spam over the last few years that the MySQL table is full. I was tempted to clean up the database, update the mediawiki software used to power the wiki and start all over but then realized that it’s a lot of hard work for little return.

During the early days, there were a few usefull contributions to the wiki but later on they were inundated by spam. Nearly every legit contribution was accidentally deleted while cleaning up spam. Some pages would be hit even 40-50 times a minute (no wonder the database filled up). I tried combating it by using the mediawiki block list and IPtables. Entire subnets were filtered out to no avail. The spam kept increased and never showed signs of decreasing.

Trouble might have been averted if I had kept mediawiki upto date but that didn’t happen. Unlike most CMS style web applications it’s not possible to ’skip versions with mediawiki’. Well it might be possible now but it wasn’t always so. That means if you can’t keep up with all their new releases, it becomes harder and hard to do the update. I gave up years ago.

In the end I just relied on wget and a small python script that I wrote to convert all the wiki entries into PHP. There are only a few (too few really, yeah pathetically few if you would call a spade a spade) pages which means all the time and effort spent on it is just a waste.

Can’t that be handled by the template? no because this was a CSS change that really did involve changing the post’s HTML.  It was just a matter of removing an inline style (CSS) and replacing it with a CSS class. But each post only has a fragment so python minidom simply refused parse it. That forced me to look at HTMLParser which I didn’t like because it’s an old fashion event driven parser. There was a time when use to swear by event driven parsers (expat for example) but that was long ago.

Thus I had no option but to revert to the minidom API and to make my HTML fragment well formed by adding a new start and end tag to enclose the whole fragment. These enclosing tags can be stripped out later when I am saving the modified post back to the database.

With this approach the code is short and sweet. But of course I need to flesh it out by adding support for retrieving posts from the database and writing them back in, instead of working with the hardcoded bit of HTML as done during testing.

#!/usr/bin/python

from xml.dom.minidom import *

domNode = parseString(‘<xml><p align=”center”><a href=”/images/comingup.jpg”><img src=”/images/comingup-t.jpg” title=”dawn” alt=”Sunrise close to Nuwara Eliya” style=”border-color: #505050; border-width: 7px” /></a></p>Great Western, Nuwara Eliya at Dawn.</xml>’)

ele = domNode.getElementsByTagName(‘img’)
ele.item(0).removeAttribute(’style’)
ele.item(0).setAttribute(“class”,”photo”);

subNode = domNode.firstChild;

if subNode.hasChildNodes():
children = subNode.childNodes
for child in children:
print child.toxml()

]]> http://www.raditha.com/blog/archives/1763.html/feed 0 http://www.raditha.com/blog/archives/1949.html http://www.raditha.com/blog/archives/1949.html#comments Fri, 28 Nov 2008 14:46:02 +0000 admin http://www.raditha.com/blog/?p=1949 You might have noticed that I have an item named religion in my tag cloud. Does that mean I have suddenly become religious? Hello no. Exorcise that thought. Religion is the opium of the poor (well I am not exactly rich but I am not poor and don’t do opium). Heaven and hell exists only in the minds of preachers.

Lots of other geeks are religious though. They would swear by Java, PHP or some poor souls who need to be rescued even swear by primitive things like C# or .NET (windows really does sucks (No that has nothing to do with religion , it’s a scientific fact)).

So this tag, religion is about such arguments like ‘PHP is better than Java‘ and ‘perl is better than python’ (anyone want to compare Fortran and Python?).

Well what’s really better, is it PHP or Java? well it’s a matter of horses for courses (but PHP has recently lost the plot a bit with an API that is way too big, but then Java is showing it’s age too).
While still on the subject of religion and religious wars you might ask, Is Fedora better than Debian?  Yes Fedora is miles ahead of Debian (but even so, Debian is much better than Windows)

Netbeans can be used with Xdebug to debug PHP scripts. I know that xdebug can be used with eclipse too but I have never quite aquired a taste for eclipse. Of course which IDE is better is largely a matter of religious preference but in my humble opion the best IDE is Nebeans.

I am no longer a heavy PHP coder, so I use whatever editor that comes to mind when I actually do write in PHP – many of them don’t even have debug support. But Netbeans is something that is almost always on my desktop so this means I can do both PHP and Java on the same IDE.

Footnote: install xdebug with yum install php-pecl-xdebug

As for python, it doesn’t even have block quotes! Well people tell me that I don’t need block quotes, and unlike ratsnakes, blockquotes are poison. Since Pythons aren’t venomous the don’t go together. That makes Python similar to Fortran and perl. Perl has other useful features not present in python, like the here document. But when you look closer you find that triple quotes can be used as a substitute. I know python fans will fall on my like a bunch of Anacondas for comparing python with Fortran. They seem to hate it.

One great advantage of PHP is that is dynamical typed which means a great disadvantage of PHP and of Python is that they are both dynamically typed (no I am not contradicting myself it’s great when things go right, a huge pain when things go wrong). PHP has several functions to sort out the types mess but sometimes they give strange results.

I have seen programming language quizzes where they give you an expression with mixed types and ask you what the result is going to be. How stupid is that? No self respecting programmer is going to bother to figure out such code in his head.

I digress. The whole point of this was because of the trouble I was having finding the equivalent of print_r or var_dump of PHP in python. It was after some searching that I found it on a wierdly named site the function is repr. Who ever came up with that name?

I had more difficulty in finding the size of an uploaded file – sure, I know how to get the size after the file has been saved – but I need to know the size before saving – so far, the documentation hasn’t thrown up anything. Sure you can get the size indirectly by us calling len() on the file data but it seems such a waste.

PHPProjekt on the other hand has survived and has seen the release of many new versions. While so much of features have been added it hasn’t become any easier to use. Memories have faded but I think it’s actually a lot harder to use now than it was then.

For this blog, I don’t want the old entries to be redirected at all. The old links should remain the way they are (numeric) and the new ones should have the title in the link. That’s because many old posts are ranked high in google. Changing their URLs might result in them dropping off.

At first I thought of writing my own plug-in oto achieve this but a good geek doesn’t write anything if he can avoid it. So I looked around again and found the advanced permalinks migration plugin which at first sight seemed to do the job just right.

On closer inspection I found that the category archives broke the moment the plugin was enabled. Great pity because it has just the features that I am looking for and the author seems to have put in a lot of effort for it. However it’s said to be compatible with wordpress version 2.50 or older and the current version is 2.63. So there you are, it doesn’t always pay to be lazy.

That probably goes to show that wordpress really is fun to use while MT isn’t. But I will not be too critical, it served me well for more than 3 years specially at a time when my expectations of a blogger was very different. Movable type itself has changed, they are now more keen to target business bloggers rather than individual geeks.

That takes care of single posts but unfortunately the plugin doesn’t work with monthly archives. In my photo blog, the permalink format was ‘numeric’ (/archives/%post_id%) which results in the montly archives looking like this: http://photos.raditha.com/archives/date/2008/01 . Under the new ‘postname’ scheme that I adopted ( /archives/%postname%/ ) the link becomes http://photos.raditha.com/archives/2008/01 – idealy Dean’s plugin should redirect the old archives page to the new one but it doesn’t happen. However if you do click on the links above you will find that the redirection does indeed work. That’s only because I have changed my .htaccess file to rewrite them. The .htaccess file now looks like this:

It’s better to have posts identified by their names ( /%postname%/ ). I decided to go ahead and take the plunge and make the switch. Changing wordpress permalinks is a piece of cake. With the more recent versions you don’t even have to edit your .htaccess file to switch from one kind of permalink structure to another. But there is a catch: what happens to the links that are already there and have been indexed by the spiders? They will now generate 404 errors.

I was sure there would be at least one plugin that can translate the old URLs into the new scheme. If the worst comes to worst, I thougt I could just create a few URL rewrite rules to take care of it. But there was a plugin that can translate the urls. It’s named Dean’s Permalink Migration plugin and it looks like it works.

When I tried to read the documentation on variable functions. I get an internal server error. The (local) link that caused the error was:

http://localhost/php5.2/intro.var.html

And the detailed message in the apache error log was:

Syntax error in type map, no ‘:’ in /var/www/html/php5.2/intro.var.html for header </div></body></html>\n, referer: http://localhost/php5.2/refs.basic.vartype.html

I reckoned I could sort this out by commenting out the following line in the httpd.conf file.

#AddHandler type-map var

For good measure, I commented out a few other settings that control type mappings – the web server is being confused by the filename containing a .var in it (please see link above).

<Directory “/var/www/error”>
#        AllowOverride None
#        Options IncludesNoExec
#        AddOutputFilter Includes html
#        AddHandler type-map var
#        Order allow,deny
#        Allow from all
#        LanguagePriority en es de fr
#        ForceLanguagePriority Prefer Fallback
</Directory>

and lastly. ‘DirectoryIndex index.html index.html.var‘ became just ‘DirectoryIndex index.html‘. After wards apache was restarted and I am merrily on my way. So what if the apache default page is now broken? That’s only seen when there is configured website.

That fixes that but a couple of days later I ran into another error: subrequested from r->uri
Oh and by the way, it’s print_r that can send the dump to the log file instead of the browser, not var_dump

I sorted out the apache issue by uninstalling and reinstalling it. Things went smoothly for a couple of days, until I needed to run an old PHP script. The script is nothing special it just does a bit of XSL transformation. It makes use of the PHP XSLT extension. The trouble is that is not part of debian!!

There is a PHP4-XSLT module and there is a PHP5-XSL module but no PHP5-XSLT how anoying. Once upon a long ago, I quite used to enjoy compiling things from source but now I am too old forit. Nevertheless it seems that I have no choice but to manually compile PHP5. I didn’t get far – the PHP configuration script came up with this ridiculous statement

checking whether the C compiler (gcc  ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.

Turns out this is because libc6-dev is not installed. That is easily rectified but while installing libc6-dev, apt decided to uninstall xorg? how ridiculous is that. Looks like I will be limited to the terminal next time I reboot the machine.

But when I tried to compile PHP again, I ran into the same error once more. Close inspection reveals that APT had choked out with

Failed to fetch http://ftp2.jp.debian.org/debian/pool/main/g/glibc/libc6-dev_2.7-12_i386.deb

How anoying is that? when Fedora 10 is released, I am definitely switching back.

A Pythogorean triplet is the set of three integers that figure in the equation. At Project Euler, the 9th problem is to find a triplet so that a+b+c = 1000

It’s not the most difficult problem listed there.

for($a=0 ; $a < 1000 ; $a++)
{
for($b = 0 ; $b < 1000 ; $b++)
{
$c2 = pow($a,2) + pow($b,2);
$c = sqrt($c2);
//echo "$a, $b, $c \n";
if(ceil($c) == floor($c))
{
// int
if(($a+$b+$c)== 1000)
{
echo "MATCH for $a, $b, $c | product = " . ($a*$b*$c) ."\n";

}
}
}
}