Sending Google Cloud Notifications with python and XMPP

Have you seen the sample code in Android Docs? for sending GCMs with python an XMPP? Sure you have. You probably know that import xmpp here refers to the xmpppy library.

xmpppy hasn't been updated for at least a decade. Heck there's even an ICQ number for the author on the project home page. And yes, the project is hosted on Source Forge!! Never mind all that. the real problem is that xmpppy doesn't play very well with modern versions of openssl and god knows what other incompatibilities there are.

  File "push/", line 319, in 
  File "/..../push/", line 140, in setup_xmpp
    self.xmpp_client.connect(server=(settings.CLOUD_MESSENGER['SERVER'],self.params['PORT']), secure=1, use_srv=False)
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/", line 200, in connect
    if not CommonClient.connect(self,server,proxy,secure,use_srv) or secure<>None and not secure: return self.connected
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/", line 178, in connect
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/", line 281, in PlugIn
    if now: return self._startSSL()
  File "/usr/local/virtualenv-django18/lib/python2.7/site-packages/xmpp/", line 316, in _startSSL
    tcpsock._sslIssuer = tcpsock._sslObj.issuer()
AttributeError: '_ssl._SSLSocket' object has no attribute 'issuer

So you might shop around for another python xmpp library, there are quite a few. But the majority of these libraries seem to be forks of xmppy itself but these forks have merely copied the files and just left them there. Twisted and SleekXMPP seem to be the only libraries and are being maintained. So let's see if we can modify google's sample to work with either of these two and in the long run that really is what you should do. But for now there is one, just one fork ( that's only two years behind. You will have to install it manually though.


source /usr/local/myvirtualenv/bin/activate
pip uninstall xmpppy
cd xmpppy-master
python build
python install

Hopefully the next breaking change to openssl is a long way away.

May 2, 2016, 1:31 p.m. » Tagged: python , android , xmpp , xmpppy , gcm , ccs

Updrading Postgresql 9.3 to 9.5 with Postgis.

Upgrading Postgresql minor versions is quite straght forward. You can do so with the pg_upgrade tool that's part of the binaries. But there is always a but, this is true only when Postgis extensions are not being used.

The first error in the upgrade process had nothing to do with postgis though. It was due to the wrong encoding. 

encodings for database "postgres" do not match:  old "UTF8", new "SQL_ASCII"

This was a problem that could be easily overcome by dropping the database and recreating it with the proper encoding.

/usr/pgsql-9.5/bin/initdb -E 'UTF-8' 9.5

But you will run into trouble once again with libraries

Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the new installation.  You can add these libraries to the new installation, or remove the functions using them from the old installation.  A list of problem libraries is in the file:

Failure, exiting

The contents of the loadable_libraries.txt file is as follows:

Could not load library "$libdir/rtpostgis-2.1"
ERROR:  could not access file "$libdir/rtpostgis-2.1": No such file or directory


Could not load library "$libdir/repmgr_funcs"
ERROR:  could not access file "$libdir/repmgr_funcs": No such file or directory

Could not load library "$libdir/tablefunc"
ERROR:  could not access file "$libdir/tablefunc": No such file or directory

The requirement of can be fullfilled by installing  postgresql95-contrib.x86_64 while is obviously from  repmgr95.x86_64

The postgis requirement isn't so easy.  What comes with postgresl-9.5 on centos is postgis-2.2 whereas 9.3 had 2.1. You have two choices; You can manually install 2.1 on the new server or you can do a hard upgrade. The first option is a little bit more work (you will invariably have to install the -devel versions of various libraries in order to be able to compile postgis from source) but it will minimize downtime. Hard upgrade involves dumping the entire database manually and importing it back.

So I decided to take the manual compile path only to find that it cannot be crossed.  When tying out the compile Postgis option it was discovered that the --prefix parameter to configure was broken. So the only choice for me is to do a hard upgrade. So it's a hard upgrade, and the hard upgrade is very hard because there are all kinds of inexplicable errors showing up. Here are a few examples:

ERROR:  invalid byte sequence for encoding "UTF8": 0xe0 0xaf 0x2e
ERROR:  value too long for type character varying(128)
ERROR:  insert or update on table "announcements_announcement" violates foreign key constraint "announcements_announcement_creator_id_fkey"
DETAIL:  Key (creator_id)=(1) is not present in table "auth_user". 

The data that doesn't fit into 128 chars now obviously fitted in the old version, this and the encoding errors indicate that some form of charset conversion is happening. More pressing is the foreign key errors. These users definitely exist on the roadlk database.

Update May 01: Finally solved this by not using postgis_restore and importing the database direct with pg_restore!

April 29, 2016, 6:21 a.m. » Tagged: postgis , postgresql , pg_dump

Modified round robin pairing.

TLDR; Do you want to organize a tournament where the pairing system is round robin but some players need to play against other players at a specific time? This gist is what you are looking for. If you want the full story, read on.


On friday evening, I got a call from the tournament director of Sri Lanka Scrabble. he wanted to know if I could write a bit of code for him to change the pairing of the Premier Players Tournament - a round robin event. Now you might read Scrabble and Sri Lanka in the same sentence and decide to move on, but hold it right there, Sri Lanka is fast on it's way to being a global Scrabble power house. For example the young scrabblers won six prizes at the World Youth Scrabble Championship three months ago. But I digress.

An odd number of players take part in this Premier Players Tournament and that means there is a bye. Another participant is Quackle. Sometimes players ask to receive the bye in a certain round so that they can rush out and attend to pressing matters in the middle of the tournament. Or if they get the bye at the start or end of the day, they can then show up later or leave early. On this occaision five of the players had wanted the timing of the bye fixed and two others wanted to play against Quackle in a specific round.

If only the bye needs to be accommodated, it can be done very easily just by changing the seeds. The algorithm to generate a Berger table is described on Wikiepdia, thus it's easy to see that if someone wants to have the bye in the first round, he should be given the first seed and Bye the 10th seed (or vice verce).

As pointed out by my friend Jayendra De Silva (former Tournament director for Sri Lanka Chess), you can fix the by even for all eight players simply by changing the seeds and it can be done manually even with pen and paper

If in addition to 'fixing the bye' one players wants to fix a time for the game against Quackle, that can be arranged by changing the seed for Quackle. Trouble arises when more than one player wants to some combinations turn out to be impossible. In fact I spent quite a bit of time trying out various algorithms to try to 'flip pairings'. I soon reached the conclusion that the combination asked for was impossible but had not mathematical proof. So I ended up writing a brute force method to produce all possible combinations and to see if any one of them met the requirements.

For 9 or 10 players, there are 3,628,800 possible pairings but it takes only a little more than half a minute to produced them all. However this is an O(n!) problem so it might well be impossible for twenty four players. Number of possible pairings is huge and bigger than even Avagadro's number. Trying out all combinations didn't produce a match, but the code can be used to generate customized round robin pairing schedules for many situations.

Since many tournament directors probably will not be able to make much use of the code as it is, I will be releasing an online pairing generator real soon. There are quite a few of them already but none of them allow you to reschedule games as described above.


Feb. 2, 2016, 6:34 a.m. » Tagged: scrabble , chess , round robin , pairing , python

Wifi with Qualcomm Atheros Device [168c:0042] (rev 30) on Linux Mint


sudo apt-get install build-essential linux-headers-$(uname -r) git
echo "options ath10k_core skip_otp=y" | sudo tee /etc/modprobe.d/ath10k_core.conf
tar -jxf 2015-11-05.tar.bz2
cd backports-ath10k-2015-11-05
make defconfig-ath10k
sudo make install
git clone
sudo cp -r ath10k-firmware/ath10k/ /lib/firmware/
sudo cp -r ath10k-firmware/QCA9377 /lib/firmware/ath10k/
cp firmware-5.bin_WLAN.TF.1.0-00267-1 firmware-5.bin

This is sourced primarily from however, by the time that this post is being written the original download link in the post had become invalid. Also note that grabbing the latest backport from the Linux Kernel Backports did not work either.

Right now you will want to know what this is all about. This is how you make the atheros wifi device work linux. This particular occaision it was on Linux Mint Rosa with kernerl 3.19. I had tried upgrading the kernel to the 4.2 version but wifi didn't work out of the box even though the code is supposed to have been committed. Besides isn't that where the backport comes from? Life is full of mysteries.

Jan. 8, 2016, 6:45 a.m. » Tagged: wifi , ath10k , mint

A more meaningful way to reschedule Zyzzyva cardboxes

Playing competitive Scrabble but not using Zyzzyva cardboxes? Well you should. Using Zyzzyva cardboxes but you have an unmanagable number of quizzes coming up each day? Read on.

We are not going to talk about program's built in rescheduling feature. It's usefull if you miss your quizzes for a few days  but it's not very usefull if the number of questions grows to unmanagble levels after you have just added a large number of words to your vocabularly. As always, prevention is better than the cure and there is way to prevent or minimize it from happening; When you learn a set of new words, make sure that you do a standard quiz with them before adding them to the cardbox. That way the questions are spaced out instead of being bunched up. Even then, if you keep adding a few hundred words each day for about week such as when preparing for a tournament, you will end up with a cardbox that looks something like this.


Our strategy will be to spread these out over several days. Half the questions scheduled for the next 12 days will be pushed back by 12 days. Thus the number of questions coming up over each of the next 12 days will be halved and the number after the 12th day will increase. Twelve is a number I chose arbitarily. You can use any number but make sure it's not too large a value or the cardbox the question belongs to will need to be changed as well. How exactly do we do the change? by editing the data file. The quiz data is held in two files named ~/Zyzzyva/quiz/data/CSW12/Annagrams.db and ~/Zyzzyva/quiz/data/CSW12/Hooks.db note that ~ denotes your home folder.

Close any active instances of zyzzyva and double click on the data file. If you are nervous backup the data file before you try this out. When you click you will see something like the following pop up on your desktop.


If you don't see something like that, it means you do not have sqlite installed on your computer. No fear, you can install sqlite quite easily on your computer. But if that feels a bit daunting you can add the sqlite exention to chrome or firefox and edit the data file in your browser. Once you are ready, you just need to copy paste the following text into the sqlite console and click run.

update questions SET next_scheduled = next_scheduled + 86400*12
    where next_scheduled is not null  and next_scheduled < strftime('%s','now') + 86400*12 
    AND next_scheduled % 2 =0

Now close sqlite3 and reopen Zyzzyva you will find that the quizzes have been rescheduled in a more meaningful manner than is possible with the built in feature.


June 3, 2015, 5:59 a.m. » Tagged: zyzzyva , scrabble , cardbox , sqlite

Vim and python debug


This is the first VIM based debug connection that I managed to setup. Vebugger will stop at break points and you can use it's commands to inspect variables. But it's somewhat tedious because it doesn't seem have an interactive shell like pdb does.  Vebugger is a frontend that can be used with many different languages. In case of  python what this actually does is connect to pdb, so I am left wondering if it would be better off to just use pdb by itself.

Having said that this project looks like it's been actively maintained and hopefully in a few months time we might see some more functionality and it will become more usefull..


I tried to install it by adding Plugin 'jaredly/vim-debug' without any luck. Well the installation did happen by the :Dbg command wan't enabled. Then I uninstalled it with :PluginClean and tried a manual install with better results. In fact I could quite easily start the debugger after restarting vim just by  :Dbg . and hey presto all sorts of splits appeared with stacks, variables etc. Impressive. And just to think this software hasn't been updated for 2 years!

Now for the acid test, can it debug Django? unfortunately not. :Dbg runserver yields

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/virtual-django-15/lib/python2.7/site-packages/vim_debug/", line 19, in debugger_cmd
    return start(*shlex.split(plain))
TypeError: start() takes at most 1 argument (3 given) 

But I am hopefull that I will be able to make something out this for example you can just create another script than invokes with the set of arguments that you need. Clunky but usable. The code is in python and looks really good if you ignore the fact that it conflicts with many keymaps including Command-T. This conflict  can apparently be fixed by adding let g:vim_debug_disable_mappings = 1 to .vimrc . However in my case it didn't happen and I had no choice than to uninstall it.


It can be setup with Vundle but you need to download pydbgp from Komodo afterwards start the debug server by hitting <F5> in vim and run your script from the terminal python -S ~/bin/pydbgp -d localhost:9000 for me it didn't quite work out in the first attempt.

Traceback (most recent call last):
  File "/home/raditha/bin/pydbgp", line 71, in 
    import getopt
ImportError: No module named getopt

Turns out this is caused by the -S option for python which says don't imply 'import site' on initialization whatever that means! I am guessing it has something to do with site-packages in a virtualenv but at any rate running the script without the -S option worked! So let's leave it at that

Now the acid test. Does it work with Django? Typed python ~/bin/pydbgp -d localhost:9000 runserver and was quite pleased to see it stop at the first line inside and then hit again to goto the browser and load a url. At that stage gvim stopped responding completely. However the following message could be seen in one of the debug windows Debugger engine says it is in interactive mode,"+\ "which is not supported: closing connection"

I tracked it down to in vdebug but didn't have the time to dig any deeper. If nothing else works out I will come back here


Like Vdebug, has the Komodo remote debug client listed as a dependency. Unfortunately I couldn't elicit any response from it by pressesing or any other keys. (The fact that the plugin has been activated could be seen b typing :map)


Nextup was vimpdb. According to the documentation this shouldn't be installed as a plugin but using pip. Then you need to add import vimpdb; vimpdb.set_trace() to your code and start the script. This is similar to adding import pdb; pdb.set_trace() for use with pdb. The difference here is that a new instance of vim will be launched instead of pdb when you run the script. It doesn't seem to be able to reuse an already started vim session. The basic script could be debugged wtihout any trouble and it didn't freeze or thrown an error message when django runserver eas exectued! Now the bad news, it didn't stop at a break point when a url was loaded in the browser.


And finally vim-addon-python-pdb but not much luck with that either.

Error detected while processing function python_pdb#Setup..python_pdb#RubyBuffer:
line   11:
E117: Unknown function: async_porcelaine#LogToBuffer
Error detected while processing function python_pdb#Setup..python_pdb#UpdateBreakPoints..python_pdb#BreakPointsBuffer:
line    2:
E117: Unknown function: buf_utils#GotoBuf
E15: Invalid expression: buf_utils#GotoBuf(buf_name, {'create_cmd':'sp'} )
line    3:
E121: Undefined variable: cmd
E15: Invalid expression: cmd == 'e'
Error detected while processing function python_pdb#Setup..python_pdb#UpdateBreakPoints:
line   37:
E117: Unknown function: vim_addon_signs#Push
line   43:
E716: Key not present in Dictionary: write

May 19, 2015, 12:30 a.m. » Tagged: vim , python , django , debug

Which PostGIS function to use?

Which PostGIS function or operator do you use if you wanted to find the overlap between different lines? There are several candidates which immidiately come to mind;  ~, && , ST_Overlaps and and ST_Intersects if you are in a hurry or TLDR; use ST_Overlap. The two operators They all do job but in different ways, takes different amounts of time and return slightly different results. 

Using only operators             time  4.3237  rows  1208
Using ~ operator and ST_Overlap  time  5.5201  rows  233
Using ST_Dwithin                 time  5.5551  rows  2500
Using ST_Intersects              time  5.3128 rows  746  

ST_DWithin seems to match everything to everything else. Using ~ and &&. Usin operatings seem to be a shade quicker than the other options. But the operators only deal with bounding boxes not the lines themselves so your milage may vary depending on your context. This test was carried out with a nested for loop with the other loop having 250 objects and the inner loop having 25. The data came from the carpool tables. Picking the right sample is very important in this test since different samples could produce different results so the sample was actually changed multiple times.  The results followed the same pattern except for this one instance where all the routes were disjoint! 

Using only operators      time  0.0174  rows  0
Using operators + overlap time  0.0176  rows  0
Using ST_Dwithin          time  0.0177  rows  0
Using ST_Intersects       time  0.0177  rows  0   

I should have used number formatting but I didn't so sue me. Getting back to the first data set, the number of matches vary by a factor of eleven so we ought to check the quality of the matches. That can be done by using the overlapping distances.

Using only operators             time 7.1954  rows  1208  total overlap  0.199858778468
Using ~ operators and ST_Overlap time 6.9210  rows  233   total overlap  0.199858778468
Using ST_Dwithin                 time 10.7650 rows  2500  total overlap  0.199858778468
Using ST_Intersects              time 7.1929  rows  746   total overlap  0.199858778468 


Surprise, surprise all the overlapping distances are identical. So clearly the a.line && b.line greatly over estimates the number of overlaps and so does ST_Dwithin. Counter intuitively ST_Overlaps and ST_Intersects both seem to have the edge over the operators.

Before winding down, I wanted to do one final test. As things stand the queries look like' INNER JOIN pool_route b ON a.line ~ b.line OR b.line ~ a.line or a.line && b.line ' and ' INNER JOIN pool_route b ON a.line ~ b.line OR b.line ~ a.line or ST_Overlaps(a.line , b.line) ' will there be a difference if the contains operator (~) is removed?


Using only operators             time  7.4457  rows  1208  total overlap  0.199858778468
Using ~ operators and ST_Overlap time  6.0198  rows  64    total overlap  0.199858778468
Using ST_Dwithin                 time  10.7326 rows  2500  total overlap  0.199858778468
Using ST_Intersects              time  7.3100  rows  746   total overlap  0.199858778468     


Just ignore the timing the difference is too small to be taken seriously but look at how the number of matches has dropped dramatically from 266 to 64 for the ST_Overlap query. Yet the final overlapping distance hasn't changed. At this point I double checked and triple checked to make sure that my code doesn't have any bugs and it looks like there isn't. The only thing to do is to try with a much bigger dataset


Using only operators time  317.762  rows  83075  total overlap  116.168481483
Using ST_Overlap     time  223.555  rows  13884  total overlap  116.167709497
Using ST_Dwithin     time  416.819  rows  160000  total overlap  116.168481483
Using ST_Intersects  time  276.907  rows  46804  total overlap  116.168481483

This is with a much larger dataset, 400 objects in the outer loop, 400 objects in the inner loop. If you want to have look at the code that was used here you go:

import os, sys

if __name__ == '__main__':  #pragma nocover
    # Setup environ

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "main.settings_dev")
from django.db import connection
from pool.models import Route

import time

cursor = connection.cursor()

routes = Route.objects.order_by('id')[0:100]
routes2 = Route.objects.order_by('-id')[0:250]

t0 = time.time()
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '

                ' INNER JOIN pool_route b ON a.line && b.line '
                ' WHERE = %s AND = %s', [,])
        row = cursor.fetchone()
        sum += row[0]
        if row[1]:
            distance += row[1]
t1 = time.time() - t0
t0 = time.time()

print 'Using only operators time ' , t1 , ' rows ' , sum ,' total overlap ', distance
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Overlaps(a.line , b.line) '
                ' WHERE = %s AND = %s', [,])
        row = cursor.fetchone()
        sum += row[0]
        if row[1]:       
           distance += row[1]

t1 = time.time() - t0
t0 = time.time()
print 'Using ~ operators and ST_Overlap time ' , t1 , ' rows ' , sum ,' total overlap ', distance

sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line))) FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Dwithin(a.line, b.line, 50) '
                ' WHERE = %s AND = %s', [,])
        row = cursor.fetchone()
        sum += row[0] 
        if row[1]:
            distance += row[1]

t1 = time.time() - t0
t0 = time.time()
print 'Using ST_Dwithin time ' , t1 , ' rows ' , sum, ' total overlap ', distance
sum = 0
distance = 0

for r1 in routes:
    for r2 in routes2 :
        q = cursor.execute('SELECT count(*) , sum(ST_Length(ST_intersection(a.line,b.line)))  FROM pool_route a '
                ' INNER JOIN pool_route b ON ST_Intersects(a.line, b.line) '
                ' WHERE = %s AND = %s', [,])
        row = cursor.fetchone()
        sum += row[0] 
        if row[1]:
            distance += row[1]
t1 = time.time() - t0

print 'Using ST_Intersects time ' , t1 , ' rows ' , sum, ' total overlap ', distance

Update: a bug in the code was fixed and the blog post formatted (by hand)

May 16, 2015, 4:11 a.m. » Tagged: postgis , python ,

Configure the vim plugins and other stuff.

The story so far is that I have moved from pydev to vim, and then dumped the initial vim setup which used python mode in favour of setting everything up from scratch. Then struggled a bit with autocomplete for django and now it's time to tweak some of the plugins.


Right-o now it's time to configure the plugins that were installed the other day. Let's start with minibufexpl they haven't made any key mappings so if you want to use it you need to keep typing in long commands or you need to create a map in your .vimrc file. Before doing that you can type :nmap to see what the current mappings are (you dont' want a conflict do you?) I chose <leader>\b for buffer explorer :MBEToggle and I am not bothered about open and close mappings. The next thing is to change the positioning from a horizontal split to a verticle and make it 20 columns wide, and also open on the left.

let g:miniBufExplVSplit = 20   " column width in chars
let g:miniBufExplBRSplit = 0
let g:miniBufExplStatusLineText = 'Buffers'

It also changes the minibufexpl's status bar to just buffers. Wonder if the status line can be removed completely.


While minibufexpl might be cool it might actually be redundant because command-t has both a file and buffer select/search/autocomplete feature. In fact with Command-t installed you probably don't need NERDTree either. In my case the debugger took over both default mappings for the buffer and files and they had to be added back in with

nnoremap  t :CommandT
nnoremap  b :CommandTBuffer


This one didn't survie. The plugin is five years old and can be replaced with a simple vim command (which is nothing more than a comment to a stackoverflow answer)

noremap <leader>todo :noautocmd vimgrep /TODO/j **/*.py<CR>:cw<CR>


Here is another one that didn't survive. I've never really liked using snippets. You are more likely to spend time thinking up usefull snippets and putting them into a file rather than actual coding. And this is a lot of extra stuff to remember.

Oh, BTW snipmate is often linked in blogs as msanders/snipmate.vim; that's indeed the original. However there is a more upto form at garbas/vim-snipmate this one doesn't have the snippets though, but you would probably want to look up snippets specific for your programming language than the rather generic (and outdated) snippets you find in the original repo.

NERD Commenter

It's one of the things I picked up from coming home to vim. NERD Commenter allows you to add remove comments very easily. If you never had to comment out a large chunk of code temporarily your brain is probably wired differently from everyone else's



If you write HTML (and with everyone claiming to be a full stack developer who doesn't?) you would love sparkup you write what looks like CSS and the plugin will convert that to a set of tags. Pretty nifty.


The keyboard mappings

use Ctrl-s to save files when in insert mode

imap <C-S> <ESC>:w<CR>i

add Ctrl-c, v, x support and also make vim use the system clipboard

" add 'standard' copy paste stuff 
vmap <C-c> "+y1
vmap <C-x> "+c
vmap <C-v> c<ESC>"+pg
imap <C-v> <C-r><C-o>+

" use the system clipboard so that lines yanked in vim can be pasted elsewhere
set clipboard=unnamed


Before ending this post, Let's get back to rope for a bit. The first time you open a project you will need to do :RopeGenerateAutoimportCache to set things up

May 13, 2015, 5:15 a.m. » Tagged: vim , plugins