A dilapidated Scrabble Power House

What does the 2003 UK Scrabble Champion and the 2005 Australian Scrabble Champion have in common?  They are both of Sri Lankan origin (Harshan and Naveen).  For good measure Naveen Fernando won the Australian title again in 2010.  Before that he had won the Gulf Championship. Ancient history you say? Then consider that Sri Lanka was the only country to have three players in the top 10 at the 2015 World Youth Scrabble Championship. One of them won the Under 13 title and Sri Lanka also won the team title. 

When the incomparable Bobby Fischer emerged as the World Chess Champion, it was likened to grass growing through the cracks on a busy sidewalk. Then Sri Lankan Scrabblers shining on the world stage is the equivalent of a tree sprouting on the middle or a busy elevated highway. Missaka Warusawithana played at the World Championship as a 15 year old. Mind you this is the senior event, not World Youth Championship. Missaka was 15th that year. A few years later Naveen bettered it moved it from 15 to 13, 11, 8 and 3 (though he wasn't representing Sri Lanka at the time). There it has languished for a long time.

Sri Lanka Scrabble team for WYSC 2016

Photo by Manjula de Silva

You would think that a tiny Indian ocean country where English is spoken as the first language by less than 50,000 persons might not be able to achieve these sort of performances but they have. Sri Lankans have done even better at WYSC consistently having one or two players in the top 10 and having three in 2015. So why hasn't Sri Lanka had a top 10 finisher in the WSC since 2005? When you read the next few posts you will find out!

If Sri Lankan eventually wins the World Scrabble Championship, it would be inspite of the Sri Lanka Scrabble League rather than because of it

foot note: This article isn't exactly tech geek but it's a prelude to a series of posts that will be geek (even though not exactly tech geek)

June 26, 2016, 1:40 p.m. » Tagged: scrabble

Sending Google Cloud Notifications with python and XMPP

Have you seen the sample code in Android Docs? for sending GCMs with python an XMPP? Sure you have. You probably know that import xmpp here refers to the xmpppy library.

xmpppy hasn't been updated for at least a decade. Heck there's even an ICQ number for the author on the project home page. And yes, the project is hosted on Source Forge!! Never mind all that. the real problem is that xmpppy doesn't play very well with modern versions of openssl and god knows what other incompatibilities there are.

  File "push/ccs.py", line 319, in 
  File "/..../push/base.py", line 140, in setup_xmpp
    self.xmpp_client.connect(server=(settings.CLOUD_MESSENGER['SERVER'],self.params['PORT']), secure=1, use_srv=False)
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/client.py", line 200, in connect
    if not CommonClient.connect(self,server,proxy,secure,use_srv) or secure<>None and not secure: return self.connected
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/client.py", line 178, in connect
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/transports.py", line 281, in PlugIn
    if now: return self._startSSL()
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/transports.py", line 316, in _startSSL
    tcpsock._sslIssuer = tcpsock._sslObj.issuer()
AttributeError: '_ssl._SSLSocket' object has no attribute 'issuer

So you might shop around for another python xmpp library, there are quite a few. But the majority of these libraries seem to be forks of xmppy itself but these forks have merely copied the files and just left them there. Twisted and SleekXMPP seem to be the only libraries and are being maintained. So let's see if we can modify google's sample to work with either of these two and in the long run that really is what you should do. But for now there is one, just one fork (https://github.com/ArchipelProject/xmpppy/) that's only two years behind. You will have to install it manually though.


source /usr/local/myvirtualenv/bin/activate
pip uninstall xmpppy
wget https://github.com/ArchipelProject/xmpppy/archive/master.zip
unzip master.zip
cd xmpppy-master
python setup.py build
python setup.py install

Hopefully the next breaking change to openssl is a long way away.

May 2, 2016, 1:31 p.m. » Tagged: python , android , xmpp , xmpppy , gcm , ccs

Updrading Postgresql 9.3 to 9.5 with Postgis.

Upgrading Postgresql minor versions is quite straght forward. You can do so with the pg_upgrade tool that's part of the binaries. But there is always a but, this is true only when Postgis extensions are not being used. The errors that I ran into while upgrading and their fixes are shared here for the benefit of anyone else who may run into the same issues.

The first error in the upgrade process had nothing to do with postgis though. It was due to the wrong encoding. 

encodings for database "postgres" do not match:  old "UTF8", new "SQL_ASCII"

This was a problem that could be easily overcome by dropping the database and recreating it with the proper encoding.

/usr/pgsql-9.5/bin/initdb -E 'UTF-8' 9.5

But you will run into trouble once again with libraries

Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the new installation.  You can add these libraries to the new installation, or remove the functions using them from the old installation.  A list of problem libraries is in the file:

Failure, exiting

The contents of the loadable_libraries.txt file is as follows:

Could not load library "$libdir/rtpostgis-2.1"
ERROR:  could not access file "$libdir/rtpostgis-2.1": No such file or directory


Could not load library "$libdir/repmgr_funcs"
ERROR:  could not access file "$libdir/repmgr_funcs": No such file or directory

Could not load library "$libdir/tablefunc"
ERROR:  could not access file "$libdir/tablefunc": No such file or directory

The requirement of tablefunc.so can be fullfilled by installing  postgresql95-contrib.x86_64 while repmgr_funcs.so is obviously from  repmgr95.x86_64

The postgis requirement isn't so easy.  What comes with postgresl-9.5 on centos is postgis-2.2 whereas 9.3 had 2.1. You have two choices; You can manually install 2.1 on the new server or you can do a hard upgrade. The first option is a little bit more work (you will invariably have to install the -devel versions of various libraries in order to be able to compile postgis from source) but it will minimize downtime. Hard upgrade involves dumping the entire database manually and importing it back.

So I decided to take the manual compile path only to find that it cannot be crossed.  When tying out the compile Postgis option it was discovered that the --prefix parameter to configure was broken. So the only choice for me is to do a hard upgrade. So it's a hard upgrade, and the hard upgrade is very hard because there are all kinds of inexplicable errors showing up. Here are a few examples:

ERROR:  invalid byte sequence for encoding "UTF8": 0xe0 0xaf 0x2e
ERROR:  value too long for type character varying(128)
ERROR:  insert or update on table "announcements_announcement" violates foreign key constraint "announcements_announcement_creator_id_fkey"
DETAIL:  Key (creator_id)=(1) is not present in table "auth_user". 

The data that doesn't fit into 128 chars now obviously fitted in the old version, this and the encoding errors indicate that some form of charset conversion is happening. More pressing is the foreign key errors. These users definitely exist on the roadlk database.

Update May 01: Finally solved this by not using postgis_restore and importing the database direct with pg_restore!

April 29, 2016, 6:21 a.m. » Tagged: postgis , postgresql , pg_dump

Modified round robin pairing.

TLDR; Do you want to organize a tournament where the pairing system is round robin but some players need to play against other players at a specific time? This gist is what you are looking for. If you want the full story, read on.

The Story

On friday evening, I got a call from the tournament director of Sri Lanka Scrabble. he wanted to know if I could write a bit of code for him to change the pairing of the Premier Players Tournament - a round robin event. Now you might read Scrabble and Sri Lanka in the same sentence and decide to move on, but hold it right there, Sri Lanka is fast on it's way to being a global Scrabble power house. For example the young scrabblers won six prizes at the World Youth Scrabble Championship three months ago. But I digress.

An odd number of players take part in this Premier Players Tournament and that means there is a bye. Another participant is Quackle. Sometimes players ask to receive the bye in a certain round so that they can rush out and attend to pressing matters in the middle of the tournament. Or if they get the bye at the start or end of the day, they can then show up later or leave early. On this occaision five of the players had wanted the timing of the bye fixed and two others wanted to play against Quackle in a specific round.

The Berger Table

If only the bye needs to be accommodated, it can be done very easily just by changing the seeds. The algorithm to generate a Berger table is described on Wikiepdia, thus it's easy to see that if someone wants to have the bye in the first round, he should be given the first seed and Bye the 10th seed (or vice verce).

As pointed out by my friend Jayendra De Silva (former Tournament director for Sri Lanka Chess), you can fix the bye even for all eight players simply by changing the seeds and it can be done manually even with pen and paper

If in addition to 'fixing the bye' one players wants to fix a time for the game against Quackle, that can be arranged by changing the seed for Quackle. Trouble arises when more than one player wants to player Quackle and Bye at certain times. Some combinations turn out to be impossible. In fact I spent quite a bit of time trying out various algorithms to try to 'flip pairings'. I soon reached the conclusion that the combination asked for was impossible but had not mathematical proof. So I ended up writing a brute force method to produce all possible combinations and to see if any one of them met the requirements.

The Numbers

For 9 or 10 players, there are 3,628,800 possible pairings but it takes only a little more than half a minute to produced them all. However this is an O(n!) problem so it might well be impossible for twenty four players. Number of possible pairings is huge and bigger than even Avagadro's number. Trying out all combinations didn't produce a match, but the code can be used to generate customized round robin pairing schedules for many situations.

Since many tournament directors probably will not be able to make much use of the code as it is, I will be releasing an online pairing generator real soon. There are quite a few of them already but none of them allow you to reschedule games as described above.


Feb. 2, 2016, 6:34 a.m. » Tagged: scrabble , chess , round robin , pairing , python

Wifi with Qualcomm Atheros Device [168c:0042] (rev 30) on Linux Mint


sudo apt-get install build-essential linux-headers-$(uname -r) git
echo "options ath10k_core skip_otp=y" | sudo tee /etc/modprobe.d/ath10k_core.conf
wget http://filebin.ca/2NIiTdhPm2Q2/backports-ath-2015-11-05.tar.bz2
tar -jxf 2015-11-05.tar.bz2
cd backports-ath10k-2015-11-05
make defconfig-ath10k
sudo make install
git clone https://github.com/kvalo/ath10k-firmware.git
sudo cp -r ath10k-firmware/ath10k/ /lib/firmware/
sudo cp -r ath10k-firmware/QCA9377 /lib/firmware/ath10k/
cp firmware-5.bin_WLAN.TF.1.0-00267-1 firmware-5.bin

This is sourced primarily from http://ubuntuforums.org/showthread.php?t=2300861&p=13386625#post13386625 however, by the time that this post is being written the original download link in the post had become invalid. Also note that grabbing the latest backport from the Linux Kernel Backports did not work either.

Right now you will want to know what this is all about. This is how you make the atheros wifi device work linux. This particular occaision it was on Linux Mint Rosa with kernerl 3.19. I had tried upgrading the kernel to the 4.2 version but wifi didn't work out of the box even though the code is supposed to have been committed. Besides isn't that where the backport comes from? Life is full of mysteries.

Jan. 8, 2016, 6:45 a.m. » Tagged: wifi , ath10k , mint

A more meaningful way to reschedule Zyzzyva cardboxes

Playing competitive Scrabble but not using Zyzzyva cardboxes? Well you should. Using Zyzzyva cardboxes but you have an unmanagable number of quizzes coming up each day? Read on.

We are not going to talk about the program's built in rescheduling feature. It's usefull if you miss your quizzes for a few days  but it's not very usefull if the number of questions grows to unmanagble levels after you have just added a large number of words to your vocabularly. As always, prevention is better than the cure and there is way to prevent or minimize it from happening; When you learn a set of new words, make sure that you do a standard quiz with them before adding them to the cardbox. That way the questions are spaced out instead of being bunched up. Even then, if you keep adding a few hundred words each day for about week such as when preparing for a tournament, you will end up with a cardbox that looks something like this.


If you are someone competing at international level you might laugh at these numbers but you might still be daunted by numbers that are tenfolds of the above So please read on.

Our strategy will be to spread these out over several days. Half the questions scheduled for the next 12 days will be pushed back by 12 days. Thus the number of questions coming up over each of the next 12 days will be halved and the number after the 12th day will increase. Twelve is a number I chose arbitarily. You can use any number but make sure it's not too large a value or the cardbox the question belongs to will need to be changed as well. How exactly do we do the change? by editing the data file. The quiz data is held in two files named ~/Zyzzyva/quiz/data/CSW12/Annagrams.db and ~/Zyzzyva/quiz/data/CSW12/Hooks.db note that ~ denotes your home folder.

Close any active instances of zyzzyva and double click on the data file. If you are nervous backup the data file before you try this out. When you click you will see something like the following pop up on your desktop.


If you don't see something like that, it means you do not have sqlite installed on your computer. No fear, you can install sqlite quite easily on your computer. But if that feels a bit daunting you can add the sqlite exention to chrome or firefox and edit the data file in your browser. Once you are ready, you just need to copy paste the following text into the sqlite console and click run.

update questions SET next_scheduled = next_scheduled + 86400*12
    where next_scheduled is not null  and next_scheduled < strftime('%s','now') + 86400*12 
    AND next_scheduled % 2 =0

Now close sqlite3 and reopen Zyzzyva you will find that the quizzes have been rescheduled in a more meaningful manner than is possible with the built in feature.


June 3, 2015, 5:59 a.m. » Tagged: zyzzyva , scrabble , cardbox , sqlite

Vim and python debug


This is the first VIM based debug connection that I managed to setup. Vebugger will stop at break points and you can use it's commands to inspect variables. But it's somewhat tedious because it doesn't seem have an interactive shell like pdb does.  Vebugger is a frontend that can be used with many different languages. In case of  python what this actually does is connect to pdb, so I am left wondering if it would be better off to just use pdb by itself.

Having said that this project looks like it's been actively maintained and hopefully in a few months time we might see some more functionality and it will become more usefull..


I tried to install it by adding Plugin 'jaredly/vim-debug' without any luck. Well the installation did happen by the :Dbg command wan't enabled. Then I uninstalled it with :PluginClean and tried a manual install with better results. In fact I could quite easily start the debugger after restarting vim just by  :Dbg . and hey presto all sorts of splits appeared with stacks, variables etc. Impressive. And just to think this software hasn't been updated for 2 years!

Now for the acid test, can it debug Django? unfortunately not. :Dbg manage.py runserver yields

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/virtual-django-15/lib/python2.7/site-packages/vim_debug/commands.py", line 19, in debugger_cmd
    return start(*shlex.split(plain))
TypeError: start() takes at most 1 argument (3 given) 

But I am hopefull that I will be able to make something out this for example you can just create another script than invokes manage.py with the set of arguments that you need. Clunky but usable. The code is in python and looks really good if you ignore the fact that it conflicts with many keymaps including Command-T. This conflict  can apparently be fixed by adding let g:vim_debug_disable_mappings = 1 to .vimrc . However in my case it didn't happen and I had no choice than to uninstall it.


It can be setup with Vundle but you need to download pydbgp from Komodo afterwards start the debug server by hitting <F5> in vim and run your script from the terminal python -S ~/bin/pydbgp -d localhost:9000 hello.py for me it didn't quite work out in the first attempt.

Traceback (most recent call last):
  File "/home/raditha/bin/pydbgp", line 71, in 
    import getopt
ImportError: No module named getopt

Turns out this is caused by the -S option for python which says don't imply 'import site' on initialization whatever that means! I am guessing it has something to do with site-packages in a virtualenv but at any rate running the script without the -S option worked! So let's leave it at that

Now the acid test. Does it work with Django? Typed python ~/bin/pydbgp -d localhost:9000 manage.py runserver and was quite pleased to see it stop at the first line inside manage.py and then hit again to goto the browser and load a url. At that stage gvim stopped responding completely. However the following message could be seen in one of the debug windows Debugger engine says it is in interactive mode,"+\ "which is not supported: closing connection"

I tracked it down to runner.py in vdebug but didn't have the time to dig any deeper. If nothing else works out I will come back here


Like Vdebug, has the Komodo remote debug client listed as a dependency. Unfortunately I couldn't elicit any response from it by pressesing or any other keys. (The fact that the plugin has been activated could be seen b typing :map)


Nextup was vimpdb. According to the documentation this shouldn't be installed as a plugin but using pip. Then you need to add import vimpdb; vimpdb.set_trace() to your code and start the script. This is similar to adding import pdb; pdb.set_trace() for use with pdb. The difference here is that a new instance of vim will be launched instead of pdb when you run the script. It doesn't seem to be able to reuse an already started vim session. The basic script could be debugged wtihout any trouble and it didn't freeze or thrown an error message when django manage.py runserver eas exectued! Now the bad news, it didn't stop at a break point when a url was loaded in the browser.


And finally vim-addon-python-pdb but not much luck with that either.

Error detected while processing function python_pdb#Setup..python_pdb#RubyBuffer:
line   11:
E117: Unknown function: async_porcelaine#LogToBuffer
Error detected while processing function python_pdb#Setup..python_pdb#UpdateBreakPoints..python_pdb#BreakPointsBuffer:
line    2:
E117: Unknown function: buf_utils#GotoBuf
E15: Invalid expression: buf_utils#GotoBuf(buf_name, {'create_cmd':'sp'} )
line    3:
E121: Undefined variable: cmd
E15: Invalid expression: cmd == 'e'
Error detected while processing function python_pdb#Setup..python_pdb#UpdateBreakPoints:
line   37:
E117: Unknown function: vim_addon_signs#Push
line   43:
E716: Key not present in Dictionary: write

May 19, 2015, 12:30 a.m. » Tagged: vim , python , django , debug

Which PostGIS function to use?

Which PostGIS function or operator do you use if you wanted to find the overlap between different lines? There are several candidates which immidiately come to mind;  ~, && , ST_Overlaps and and ST_Intersects if you are in a hurry or TLDR; use ST_Overlap. The two operators They all do job but in different ways, takes different amounts of time and return slightly different results. 

Using only operators             time  4.3237  rows  1208
Using ~ operator and ST_Overlap  time  5.5201  rows  233
Using ST_Dwithin                 time  5.5551  rows  2500
Using ST_Intersects              time  5.3128 rows  746  

ST_DWithin seems to match everything to everything else. Using ~ and &&. Usin operatings seem to be a shade quicker than the other options. But the operators only deal with bounding boxes not the lines themselves so your milage may vary depending on your context. This test was carried out with a nested for loop with the other loop having 250 objects and the inner loop having 25. The data came from the road.lk carpool tables. Picking the right sample is very important in this test since different samples could produce different results so the sample was actually changed multiple times.  The results followed the same pattern except for this one instance where all the routes were disjoint! 

Using only operators      time  0.0174  rows  0
Using operators + overlap time  0.0176  rows  0
Using ST_Dwithin          time  0.0177  rows  0
Using ST_Intersects       time  0.0177  rows  0   

I should have used number formatting but I didn't so sue me. Getting back to the first data set, the number of matches vary by a factor of eleven so we ought to check the quality of the matches. That can be done by using the overlapping distances.

Using only operators             time 7.1954  rows  1208  total overlap  0.199858778468
Using ~ operators and ST_Overlap time 6.9210  rows  233   total overlap  0.199858778468
Using ST_Dwithin                 time 10.7650 rows  2500  total overlap  0.199858778468
Using ST_Intersects              time 7.1929  rows  746   total overlap  0.199858778468 


Surprise, surprise all the overlapping distances are identical. So clearly the a.line && b.line greatly over estimates the number of overlaps and so does ST_Dwithin. Counter intuitively ST_Overlaps and ST_Intersects both seem to have the edge over the operators.

Before winding down, I wanted to do one final test. As things stand the queries look like' INNER JOIN pool_route b ON a.line ~ b.line OR b.line ~ a.line or a.line && b.line ' and ' INNER JOIN pool_route b ON a.line ~ b.line OR b.line ~ a.line or ST_Overlaps(a.line , b.line) ' will there be a difference if the contains operator (~) is removed?


Using only operators             time  7.4457  rows  1208  total overlap  0.199858778468
Using ~ operators and ST_Overlap time  6.0198  rows  64    total overlap  0.199858778468
Using ST_Dwithin                 time  10.7326 rows  2500  total overlap  0.199858778468
Using ST_Intersects              time  7.3100  rows  746   total overlap  0.199858778468     


Just ignore the timing the difference is too small to be taken seriously but look at how the number of matches has dropped dramatically from 266 to 64 for the ST_Overlap query. Yet the final overlapping distance hasn't changed. At this point I double checked and triple checked to make sure that my code doesn't have any bugs and it looks like there isn't. The only thing to do is to try with a much bigger dataset


Using only operators time  317.762  rows  83075  total overlap  116.168481483
Using ST_Overlap     time  223.555  rows  13884  total overlap  116.167709497
Using ST_Dwithin     time  416.819  rows  160000  total overlap  116.168481483
Using ST_Intersects  time  276.907  rows  46804  total overlap  116.168481483

This is with a much larger dataset, 400 objects in the outer loop, 400 objects in the inner loop. If you want to have look at the code that was used here you go:

import os, sys

if __name__ == '__main__':  #pragma nocover
    # Setup environ

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "main.settings_dev")
from django.db import connection
from pool.models import Route

import time

cursor = connection.cursor()

routes = Route.objects.order_by('id')[0:100]
routes2 = Route.objects.order_by('-id')[0:250]

t0 = time.time()
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '

                ' INNER JOIN pool_route b ON a.line && b.line '
                ' WHERE a.id = %s AND b.id = %s', [r1.id, r2.id])
        row = cursor.fetchone()
        sum += row[0]
        if row[1]:
            distance += row[1]
t1 = time.time() - t0
t0 = time.time()

print 'Using only operators time ' , t1 , ' rows ' , sum ,' total overlap ', distance
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Overlaps(a.line , b.line) '
                ' WHERE a.id = %s AND b.id = %s', [r1.id, r2.id])
        row = cursor.fetchone()
        sum += row[0]
        if row[1]:       
           distance += row[1]

t1 = time.time() - t0
t0 = time.time()
print 'Using ~ operators and ST_Overlap time ' , t1 , ' rows ' , sum ,' total overlap ', distance

sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Dwithin(a.line, b.line, 50) '
                ' WHERE a.id = %s AND b.id = %s', [r1.id, r2.id])
        row = cursor.fetchone()
        sum += row[0] 
        if row[1]:
            distance += row[1]

t1 = time.time() - t0
t0 = time.time()
print 'Using ST_Dwithin time ' , t1 , ' rows ' , sum, ' total overlap ', distance
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line)))  FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Intersects(a.line, b.line) '
                ' WHERE a.id = %s AND b.id = %s', [r1.id, r2.id])
        row = cursor.fetchone()
        sum += row[0] 
        if row[1]:
            distance += row[1]
t1 = time.time() - t0

print 'Using ST_Intersects time ' , t1 , ' rows ' , sum, ' total overlap ', distance

Update: a bug in the code was fixed and the blog post formatted (by hand)

May 16, 2015, 4:11 a.m. » Tagged: postgis , python , road.lk