Tuesday 10 September 2013

Ubuntu Setup

A new machine, with Ubuntu installed.

sudo apt-get remove unity-lens-shopping
# Also switch off amazon results on settings >> Privacy


sudo apt-get install openssh-server

# Remember to remember pass phrase
ssh-keygen

sudo apt-get install git
git config --global user.name "Tim Pizey" 
git config --global user.email Tim.Pizey@gmail.com

sudo apt-get install gitg
sudo apt-get install ant

sudo apt-get install pip

sudo apt-get install openjdk-7-jdk

sudo apt-get install eclipse

sudo apt-get install emacs23-nox

sudo apt-get install jedit

sudo apt-get install curl

sudo apt-get install texlive
sudo apt-get install texlive-xetex
sudo apt-get install texlive-latex-extra

#scrollbars in eclipse
sudo su -c 'echo export LIBOVERLAY_SCROLLBAR=0 > /etc/X11/Xsession.d/80overlayscrollbars'

sudo apt-get install pgadmin3

sudo apt-get install scrub


Setup Chrome

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update

sudo apt-get install google-chrome-stable

Setup Haskell:

sudo apt-get install ghc
sudo apt-get install libghc6-zlib-dev
sudo apt-get install cabal-install
cabal update
cabal install cabal-install
cabal install lhs2tex

Jenkins Setup

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

Setup dropbox

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
~/.dropbox-dist/dropboxd

Setup password gorilla

sudo apt-get install password-gorilla

Checkout my stuff - manually move dot-prefixed files


sudo apt-get install cvs
cvs -d :ext:timp@paneris.org:/usr/cvsroot co timp

Add the following to /etc/ap/sources.d/postgresql.list

deb http://ppa.launchpad.net/pitti/postgresql/ubuntu precise main
deb-src http://ppa.launchpad.net/pitti/postgresql/ubuntu precise main
sudo apt-get update
sudo apt-get install postgresql-9.2

Wednesday 21 August 2013

North Brittany 2013

We booked a very last minute trip to north Brittany, to two camp sites just a few hours out of St Malo.

Saturday 27th July

Arrived just in time for ferry, fifteen minutes to spare, too close for comfort really. Great to have a cabin, had a meal and bought map etc. All slept well.

Sunday 28th July

Drove from St Malo to St Cast-le-Guildo. Drove straight to the Seymour's house, though it has been many years since I was last there.

Had a lovely walk along the beach, writing in the sand, collecting cuttle fish and Flora found two euros.

We split up and I ended up buying a coffee whilst Ellie used a loo, and spilt the whole cup in my lap, not best pleased.

We drove on to Camping le Cap Horn, where we were met with enthusiasm and great cheerfulness, which lasted the whole stay.

Our mobilhome had a covered porch with a great view over the bay. It was new this season, very clean and completely plastique!

Monday 29th July

We visited the E Leclerc at Paimpol for essentials. I spotted some cast iron crocottes, which we bought later. Bought Flora a digital watch.

Ruth knew she wanted to go to the Isle de Bréhat, so we went for a quick trip. Ellie bought the first of two thermometers. I caught eight fish from one tiny rock pool.

Tuesday 30th July

We went to an outdoor, tidal swimming pool.

Wednesday 31st July

On the way to a beach we stopped at a café

When we got to the beach

Flora and I launched the barge and coracle.

Thursday 1st August

Hung around.

Friday 2nd August

Ruth and Ellie walked to Abbaye de Beauport, Flora and I drove, via the Intermarche, where we bought some dolls house carpet.

The Abbaye is the start (km 0) of the pilgrimage to Santiago de Compostela.

Saturday 3rd August

We went for a boat trip around the Île de Beréhat, ending on the Île. We rented bikes and cycled around the whole island. We ate at a café on the main square. What a lovely place.

Our first stop was a tide mill.

We cycled all over both islands, spotting some beautiful cattle.

Sunday 4th August

Monday 5th August

Once packed and wished on our way we stopped at the Free Sculpture of Kito.

Bought some Cidre de Paimpol from the Carrefour to take home.

Stopped and horrified ourselves, at a travelling circus.

Arrived at Le Vieux Moulin Camping, Erquy.

Tuesday 6th Augst

Wednesday 7th August

Thursday 8th August

Ruth and the girls cleared out, for swim, whilst I had a 'phone interview with BlinkBox Music.

Went to Les Médievales de Fort La Latte.

Walked down to the beach at LanRuen, high tide but still very nice, through lovely woods.

Friday 9th August

Heard that BlinkBox did want to see me face to face, good end to the holiday.

Went to Pleneuf Val Andreé and had lunch at Plein des Sens. Then popped into a pottery owned by the café owners's mum.

Went down to the great beach at LanRuen, down from the camp site. It was low tide, there were fantastic rocks, a shoal of whitebait stranded in a large rock pool. We swam in the water, which was warm and a little wavy. We were sad to leave to pack up the mobilhome.

Saturday 10th August

Left the campsite at 8.20, though we had aimed for eight. No problems with the journey, arrived with fifteen minutes to spare.

Wednesday 14 August 2013

Reimplementing Length Unit Converter in Java

Back in April I did a programming exercise in Python, which I really enjoyed and resulted in PyCLU and its Jenkins node, I was slightly sorry not to set up the Jenkins node earlier in the process, so the coverage graphs do not start at zero.

Today I received the same exercise with the option of implementing in Python or Java, so I have decided to reimplement in Java, rather than reuse my python implementation.

Setup

Following Project Initiation Recipe

Choose short name

JCLU

Choose stack

This will be a pure java command line application. It will be built using Maven on OpenJDK 7, with version compatibility set to 1.6, this was initially 1.4, but revised in the light of use of annotations within the testing code. It would be possible to use 1.4 in the artefact code and 1.6 within the testing code, should that become a requirement.

Create Repo

The only decision is public or private. It will be a git repo: here.

Choose CI

My Jenkins.

Setup dev machine

My dev machine has been set up according to this Ubuntu Setup Recipe.

Checkout and create Maven project from archetype

Now we have a buildable artefact.

Setup on Jenkins

Nice and early on, so that we can see test coverage evolve. Here

Diversion to reimplement RDD in Java

In the Python implementation I added an iteration of Requirements Driven Development (RDD), a utility born out of the audit driven culture of the NHS.

Annotating each test with which requirements it addresses enables a tighter coupling between requirements and tests.

RDD fits into the evolution of Agile programming by analogy with one of the justifications for Test Driven Development:

Why are you writing code?
To make a test pass
How do you know when to stop?
When all tests pass
The motivation behind RDD similarly:
Why are you writing that test?
To satisfy requirement ref X
How do you know when to stop?
When all requirements are tested

Another motivation, and tie-in with general Agile principles, is that it ties documentation to the code: the requirements grow when new tests are written as unforeseen requirements surface.

In PyCLU I used annotations, this should be possible in java (I have not written a java annotation before).

The idea is that a dated Requirements Verification Document will be produced by running the tests, that the tests will fail if there is an unmet requirement and the document can be published by Jenkins: here.

Ready to start

Quite a lot of work just to get to the starting line!

Translated PyCLU into java, adding only one test, that the wrong number of arguments throws an exception and that the specified input produces the specified output. This showed the slight differences in the specification: JCLU does not need to output the full name of the unit, does not accept multiple units eg 1 yd 3 in and does not need to add units. Removed unspecified code. This achieves 100% code coverage.

Now to test the requirements!

Reintroduce equals() and hashCode() for testing.

Convert all relevant tests from PyCLU; add tests for unused but needed hashCode() and equals().

Add tests for units not in PyCLU.

Review and submit!

Tuesday 16 July 2013

Adding a library to java library path in Ubuntu

To enable a Java Native Interface (JNI) access to a non-java library, such as BerkeleyDB, then the library needs to be on the library path.

The simple answer given by Oracle and others is to run the command:

java -Djava.library.path=/usr/local/BerkelyDB.5.1/lib MyClass

That is a pain and difficult to ensure happens for all invocations of java.

It would be nice to use Ubuntu's ldconfig, but this is ignored by java! Java will however load from an environment variable LD_LIBRARY_PATH.

A sensible thing to do would be to place this variable in a system wide config, as it is a pain to add to every invocation of java, however due to supposed security issues this environment variable cannot be set in the normal places (/etc/environment, /etc/profile.d/) as it is deliberately nulled if you do set it!.

Sadly the only way is to add the following to the .bashrc of every user which will use your library (root, jenkins, jetty and yourself)

export LD_LIBRARY_PATH=/usr/local/BerkeleyDB.5.1/lib

Wednesday 12 June 2013

Exploring the times tables with Haskell

Ellie couldn't sleep and had noticed that the digits of the 8 times table decline, in the way that the digits of the nine times table always add to nine.

Explore in Haskell

We will need a generator for a times table series

times n = [n * x | x <- [1..]]

gives:

*Main> times 3
[3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93,96,99,102,105,108,111,114,117,120,123,126,129,132,135,138,141,144,147,150,153,156,159,162,165,168,171,174,177,180,183,186,189,192,195,198,201,204,207,210,213,216,219,222,225,228,231,234,237,240,243,246,249,252,255,258,261,264,267,270,273,276,279,282,285,288,291,294,297,300,303,306,309,312,315,318,321,324,327,330,333,336,339,342,345,348,351,354,357,360,363,366,369,372,375,378,381,384,387,390,393,396,399,402,405,408,411,414,417,420,423,426,429,432,435,438,441,444,447,450,453,456,459,462,465,468,471,474,477,480,483,486,489,492,495,498,501,504,507,510,513,516,519,522,525,528,531,534,537,540,543,546,549,552,555,558,561,564,567,570,573,576,579,582,585,588,591,594,597,600,603,606,609,612,615,618,621,624,627,630,633,636,639,642,645,648,651,654,657,660,663,666,669,672,675,678,681,684,687,690,693,696,699,702,705,708,711,714,717,720,723,726,729,732,735,738,741,744,747,750,753,756,759,762,765,768,771,774,777,780,783,786,789,792,795,798,801,804,807,810,813,816,819,822,825,828,831,834,837,840,843,846,849,852,855,858,861,864,867,870,873,876,879,882,885,888,891,894,897,900,903,906,909,912,915,918....

just the first 12

take 12 $ times 3

gives

[3,6,9,12,15,18,21,24,27,30,33,36]

We want each number as a list of its digits:

listdigits n = if n < 10 then [n] else (listdigits (n `div` 10)) ++ (listdigits (n `mod` 10))               

gives:

*Main> listdigits 12345
[1,2,3,4,5]

Add the list together

sumdigits n = sum $ listdigits n                                 

gives:

*Main> sumdigits 12345
15

what we are after is a single number, so recurse

onedigit n = if n < 10 then n else onedigit $ sumdigits n                                                   

gives:

*Main> onedigit 12345
6

now we can repeat for as far though the times table as we want, we expect repeated nines for the nine times table:

do_of n l = map onedigit (take l (times n))                                                                 
*Main> do_of 9 100 
[9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9]                                                                                                            

the answer to Ellie's original hypothesis, the eights do repeatedly decline:

*Main> do_of 8 100 
[8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8,7,6,5,4,3,2,1,9,8]

Next we want to display this and other series...

Setup Haskell Plotting

Plotting is not simple, plotting in Haskell is worse. Setup separated into its own post as it took a while.

Once plotting is setup:

main = do 
    plotListsStyle 
        [Title "Multiplication Series Reduced - For Ellie"] 
        [(PlotStyle LinesPoints $ DefaultStyle x ,  (onedigit_tuple x)) | x <- [2..9]] 

we get:

Customise the line labels:

main = do 
   plotListsStyle 
      [Title "Multiplication Series Reduced - for Ellie", 
       YLabel "Reduction", 
       XLabel "Times"] 
      [ (PlotStyle{plotType=LinesPoints, lineSpec=(CustomStyle [LineTitle $ show x])}, (onedigit_tuple x)) | x <- [2..9]]

The whole program

import Graphics.Gnuplot.Simple

listdigits :: Integer-> [Integer]
listdigits n = if n < 10 then [n] 
                         else (listdigits (n `div` 10)) ++ (listdigits (n `mod` 10))

sumdigits :: Integer -> Integer
sumdigits n = sum $ listdigits n 

onedigit :: Integer -> Integer
onedigit n = if n < 10 then n else onedigit (sumdigits n)

onedigit_tuple :: Integer -> [(Integer, Integer)]
onedigit_tuple n = [(x, onedigit( n*x ) ) | x <- [1..19]]

main = do 
   plotListsStyle 
      [Title "Multiplication Series Reduced - for Ellie", 
       YLabel "Reduction", 
       XLabel "Times", 
       Key $ Just ["outside"]] 
      [(PlotStyle{plotType=LinesPoints, 
                  lineSpec=(CustomStyle [LineTitle $ show x])}, 
        (onedigit_tuple x)) | x <- [2..9]]

Setup Haskell Plotting

I explored the space of Haskell Plotting quite extensively. It is dense, plotting is detailed. None of the examples are simple.

chart-gtk

This has extensive tests. I found the examples and tests pretty complex in themselves. This is probably the right place to dig deeper, but for now I have a limited set of goals in mind.


sudo apt-get install libgtk2.0-dev 
sudo apt-get install libpango1.0-dev 
sudo apt-get install libglib2.0-dev 
sudo apt-get install libcairo2-dev

sudo apt-get install  alex
sudo apt-get install  happy

sudo apt-get install gtk2hs-buildtools
cabal install chart-gtk

easyplot

The clue is in the name: easyplot is not. I could get the complex example to work, but I could not understand or modify it. Some nasty asymmetry in types meant I could not adapt this to my simple use case.


sudo apt-get install gnuplot-x11
cabal install easyplot

plot

At last someone with a clue has posted a little thing that works!


cabal install cabal
cabal install gtk2hs-buildtools
sudo apt-get install libcairo-dev
sudo apt-get install liblapack-dev
sudo apt-get install libgsl0-dev
cabal install hmatrix
cabal install hmatrix-gsl-stats
cabal install plot

The simple example works and can be extended:


timp@eric17:~$ ghci 
GHCi, version 7.6.2: http://www.haskell.org/ghc/  :? for help
Prelude> import Graphics.Gnuplot.Simple
Prelude Graphics.Gnuplot.Simple> plotList [] [(2,10),(3,15),(4,14),(5,19)]

timp@eric17:~$ ghci 
GHCi, version 7.6.2: http://www.haskell.org/ghc/  :? for help
Prelude> import Graphics.Gnuplot.Simple
Prelude Graphics.Gnuplot.Simple> plotList [] [(2,10),(3,15),(4,14),(5,19)]

Sunday 9 June 2013

Project initiation - a recipe

Like with my cooking blogs this starts out in the future, ends up as what actually happened. I hope this is not too confusing. It is intended as a recipe to make life easy next time I come to use it.

This recipe steals from Hackers gonna Hack.

Choose short name

Ensure this is available on heroku.

Choose stack

I challenge your horse/cart analogy: this is going to be a Django project.

Create Repo

The only decision is public or private. It will be a git repo: here.

Create the README

Do it now!

Choose deployment platform

Heroku.

Choose CI

My Jenkins.

Setup dev machine

Follow Ubuntu Setup recipe.

Use a virtualenv, it is best practice and Heroku needs it.


sudo pip install virtualenv

sudo apt-get install sqlite3
sudo apt-get install ruby-dev
sudo apt-get install libsqlite3-dev


sudo apt-get install postgresql-server-dev-all
sudo apt-get install python-dev

There is a nasty issue with deployment to Heroku: ruby and or ruby gems versions. The problem occurs during heroku db:push with a message about date serialization. This is due to a change in the serialisation of dates. If you have a more recent version of ruby than heroku, and/or you have an inconsistent set of ruby gems. This is made worse by being able to both install Heruku from the Ruby repositories and directly. The recipe below uses RVM and worked.


# setup ruby for heroku pg:push
# this can go wrong if ALL versions are not alligned

\curl -L https://get.rvm.io | bash
source /home/timp/.rvm/scripts/rvm
rvm install ruby-1.9.2-p318
rvm use --default ruby-1.9.2-p290
gem install sqlite3
gem install pg
gem install taps
gem install heroku
gem install launchy
gem install addressable
gem install taps

TODO: pgbackups

Setup project

mkdir ucl
git init ucl
cd ucl
git remote add origin https://github.com/timp21337/ucl.git
git pull origin master
virtualenv venv --distribute
echo venv >> .gitignore
git commit -m "Add virtual env" .gitignore

source  ./venv/bin/activate 
pip install django
pip install dj_database_url
pip install django_jenkins
pip install psycopg2 gunicorn 
pip install django-floppyforms

pip freeze > requirements.txt

heroku login
heroku plugins:install git://github.com/ddollar/heroku-config.git
./manage.py collectstatic 
foreman start
git clone git@heroku.com:ucll.git -o heroku

dropdb --user postgres ucll
createdb --user postgres ucll

./manage.py syncdb;
git push heroku master

heroku db:push --confirm ucll postgres://postgres:*@127.0.0.1:5432/ucll

Results in http://ucll.herokuapp.com/.

Tuesday 4 June 2013

Canon MG5200 wireless printer driver for Ubuntu (debian) systems

From this thread I found this pretty odd Canon Australia page which works, however do not think that the deb in the download is the universally understood file extension for a debian package, no it is just some guy's way of indicating that this is targeted at debian based systems. What you actually have to do is inflate and ./install.sh, then all works well.

Sunday 2 June 2013

Dual boot OSX and Ubuntu

This has been one of the least pleasant Linux experiences, but then aren't they all!

As Mike Orr put it:

Linux Air
Disgruntled employees of all the other OS airlines decide to start their own airline. They build the planes, ticket counters, and pave the runways themselves. They charge a small fee to cover the cost of printing the ticket, but you can also download and print the ticket yourself. When you board the plane, you are given a seat, four bolts, a wrench and a copy of the Seat-HOWTO.html. Once settled, the fully adjustable seat is very comfortable, the plane leaves and arrives on time without a single problem, the in-flight meal is wonderful. You try to tell customers of the other airlines about the great trip, but all they can say is, "You had to do what with the seat?"

Many sites cheerfully claim that there is no problem dual booting a mac to run Ubuntu. eg life hacker. I beg to differ!

My mac is a MacBookPro8.2, 2011 build. As shown by:

dmidecode |grep -i macbookpro

A twist is that it has an SSD where its optical drive should be, so I have to install from an external USB CD drive (I tried from a USB memory stick but ran into other problems).

It seems as though there were problems with the previous (12.04) version of the Ubuntu installer such that a problem with the usb cd provoked an issue with the video driver, or some such. I installed rEFIt and fiddled around for an age. I produced black backgrounds, purple backgrounds and even turquoise backgrounds, but no X.

I managed to install a server version of Ubuntu, but could not get ubuntu-desktop to load. By removing quiet and splash from the grub config I could see the X server crash.

Fatal server error:
no screens found

From wikipedia:MacBookPro I found I have dual graphics cards, an Intel card and an AMD Radeon HD 6490M.

From recovery mode boot I was able to see the real culprit:

fb: conflicting fb hw usage inteldrmfb vs EFI VGA - removing generic driver

I reinstalled from the ubuntu mini iso the latest (13.01) Ubuntu. Got the same conflict error. I installed the latest rEFId using a .deb. Still the same error, but now understand it is a conflict between the driver installed by rEFId and the Radeon driver.

From this thread I was able to find an incantation which worked!

  outb 0x728 1 
  outb 0x710 2 
  outb 0x740 2 
  #Power down ATI 
  outb 0x750 0
  linux /boot/vmlinuz-3.5.4-030504-generic root=/dev/sda4 video=efifb i915.modeset=1 i915.lvds_channel_mode=2 i915.lvds_use_ssc=0

This can be put into /etc/grub.d/00_header and then

grub-mkconfig -o /boot/grub/grub.cfg

Thursday 30 May 2013

Conditional inclusion in Python LaTeX template

I have been using a very simple templating technique in python

page = template % dotted

dotted is a dictionary whose keys are a dotted path to every property in an object eg child.carer.email

To conditionally include text in LaTeX you can use the etoolbox package:

\usepackage{etoolbox}

\newcommand\toggleTrue{\toggletrue}
\newcommand\toggleFalse{\togglefalse}

\newtoggle{elephants}
\toggleFalse{elephants}
\iftoggle{elephants}{
elephants
}{
not elephants
}

A boolean object property, say member.isAdult can now be used to control inclusion:

\usepackage{etoolbox}

\newcommand\toggleTrue{\toggletrue}
\newcommand\toggleFalse{\togglefalse}

\newtoggle{isAdult}
\toggle%(member.isAdult)s{isAdult}
\iftoggle{isAdult}{
payment date
}{
date of birth 
}

Wednesday 15 May 2013

Flower planter tip

The thing with a large planter is that once full of soil it is very heavy.

Fill with sealed plastic containers.

Then fill with soil.

Thursday 2 May 2013

Reverting to squeeze from wheezy

On my xen vm I tried to upgrade to wheezy, this is not possible if the host dom0 is still on etch. The kernel you are actually running is supplied by dom0 (why?) so you will now have a broken system. In particular ldconfig will segfault with kernel too old. apt-get install -f fails with:

The following packages have unmet dependencies:
 cpp : Depends: cpp-4.7 (>= 4.7.2-1~) but it is not installable
 g++-4.4 : Depends: gcc-4.4 (= 4.4.5-8) but 4.4.7-2 is to be installed
 gcc : Depends: gcc-4.7 (>= 4.7.2-1~) but it is not installable
 gcc-4.4 : Depends: gcc-4.4-base (= 4.4.7-2) but 4.4.5-8 is to be installed
           Depends: cpp-4.4 (= 4.4.7-2) but 4.4.5-8 is to be installed
           Depends: libgomp1 (>= 4.4.7-2) but it is not going to be installed
           Recommends: libc6-dev (>= 2.13-5) but 2.11.3-4 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

Part way home it failed with:

libgcc1 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed 
libstdc++6 : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed

and similar.

To return to a clean squeeze install is not easy. First edit /etc/apt/sources back to squeeze.

Now edit all of the following to comment out ldconfig:
/var/lib/dpkg/info/libstdc++6.postinst 
/var/lib/dpkg/info/libstdc++6.postrm   
/var/lib/dpkg/info/libgcc1.postrm
/var/lib/dpkg/info/libgcc1.postinst
# Automatically added by dh_makeshlibs                                          
#if [ "$1" = "configure" ]; then
#        ldconfig
#fi

Edit /etc/apt/apt-conf

APT::Default-Release "squeeze";

Now pin the packages which have been incompatibly upgraded (note Package: * does not work:

Package: libc-bin libc6-dev locales gcc gcc-4.4 cpp libgomp1 libgcc1 libstdc++6\
 build-essential
Pin: release n=wheezy
Pin-Priority: -10

Package: libc-bin libc6-dev locales gcc gcc-4.4 cpp libgomp1 libgcc1 libstdc++6\
 build-essential
Pin: release n=squeeze
Pin-Priority: 900

Clear apt's cache

apt-get clean

Finally you can

apt-get install -f

And uncomment the ldconfig lines and use * in /etc/apt/preferences

Package: *
Pin: release n=wheezy
Pin-Priority: -10

Package: *
Pin: release n=squeeze
Pin-Priority: 900

Now we are back to a system where apt-get works, but sadly still running an etch kernel!

Saturday 27 April 2013

Serialise a Django model to a dictionary following links

To enable a simple templating system I wanted a dictionary with keys as dotted paths in the object tree eg child.carer.user.first_name, however Django does not offer the ability to follow foreign keys out of the box.

def dottedDict(model, name, dict):
  for f in model._meta.fields:
    if type(f) in [models.fields.related.ForeignKey,
                   models.fields.related.OneToOneField]:
      referred = getattr(model, f.name)
      if referred is not None:
        dottedDict(referred, 
                   '%s.%s' % (name, f.name), dict)
    else:
      dict['%s.%s' % (name, f.name)] = 
          model.__dict__[f.name]
  return dict

This enables an object to be rendered (substituting empty strings for nulls):

def registrationFormLatex(self):
  template = file('Form.tex.template', 'r').read()
  dotted = dottedDict(self, 'child', {})
  done = False
  while not done:
    try:
      page = template % dotted
      done = True
    except KeyError, e:
      dotted[e.message] = ''
  return page

Tuesday 9 April 2013

Angelica in the News

Article from The News, Saturday May 11 1991.
pdf

Wednesday 27 March 2013

Javascript calculator

First hit for javascript calculator is so sweet:






From JavaScript Kit

Upgrading xen server from Etch to Wheezy

One of the issues with Xen, which has since been addressed by PyGRUB, is that the kernel used by a VM is controlled by the dom0, not by the vm. So, like me, you can cheerfully dist-upgrade within a vm, to no effect.

Ideally we want to got from Etch to Lenny to Squeeze.

http://www.debian.org/releases/squeeze/amd64/release-notes/ch-information.en.html#xen-upgrades

As a half way house lets try to get an unused vm to work with a squeeze kernel.

I have an unused vm! Update /etc/apt/sources.list to use squeeze. Oh, the debian keyring has changed:


apt-get install debian-archive-keyring

Now try to get a copy of the kernel image we need, as per above release notes.


apt-get install xen-linux-system-2.6-xen-amd64

Fat chance, as the vm is running an old kernel.

On the vm install xen-linux-system-2.6.26-2-xen-amd64, then copy to dom0


cd /boot
scp root@dom.context-computing.co.uk:/boot/vmlinuz-2.6.26-2-xen-amd64 .
scp root@dom.context-computing.co.uk:/boot/initrd.img-2.6.26-2-xen-amd64 .
xm shutdown dom.context-computing.co.uk
cd /etc/xen
mv dom.context-computing.co.uk.cfg dom.context-computing.co.uk.cfg-18
mv dom.context-computing.co.uk.cfg-26 dom.context-computing.co.uk.cfg
xm create dom.context-computing.co.uk.cfg

Whilst the config works (no Error 22), ie we have a xen enabled kernel, the machine crashes, and does so early: no output on console.

Revert to a lenny system running an etch kernel.

Tuesday 19 March 2013

Cleaning a work computer when you leave

When you leave a post what is a reasonable level to disinfect your computer to, so that there is little risk of your information leaking when your work machine is no longer under your control?

When I left my previous post I booted from a usb and shredded the whole disk, however this time I want to leave the Ubuntu installation intact.

Tasks in Order

✔ Create some new users: we are going to wipe the current administrative user.

✔ Unlink the machine from Dropbox, or files will either reappear, or get deleted on Dropbox.

✔ Pass a copy of the password safe to another member of the team.

✔ In Chrome disconnect your account.

✔ Change user password.

✔ Shred home directory: we are not trying to resist a determined attack, just trying to make sure that trivial use of google things is no longer possible.

find -type f -print0  |xargs -0 shred -u
find -depth -print0 |xargs -0 rmdir 

Thursday 21 February 2013

Compress tomcat logs from cron rather than logrotate

The problem with Tomcat is that it is OS agnostic: those who do not understand unix are doomed to reinvent it.

Specifically Tomcat has its own substandard version of logrotate, which creates a new log file every day. Some times the archive is dated sometimes the live log is dated.

#! /bin/sh
#
# TPP 2013-02-21
#
# Tomcat apps use a variety of loggers, mostly log4j. 
# These rotate, conflicting with logrotate the unix log rotation system.
#
# Some files eg catalina.out 
# are rotated to a backup containing a date eg catalina.2013-01-06.log
# which can then be compressed with bz2 to catalina.2013-01-06.log.bz2
#

cd /var/log/tomcat6

# 2013-02-21
date=`date --rfc-3339=date`
year=`date +%Y`
timestamp=`date`

echo "${timestamp} Running ${0}"

for f in $(find catalina* |grep -v bz2 |grep -v '$date' |grep $year)
do 
 echo "bzip2 $f"
 bzip2 $f
done

# However others are active whilst containing a date 
# so we will find all and not compress the most recent

for l in 'localhost*' 'opt-db*' 'opt*' 'host-manager*' 'manager*'
do 
 previous="not_this_one"
 for f in $(find $l |grep -v bz2 |grep $year|sort)
 do
  if [ "${previous}" != "not_this_one" ]
  then
    echo "bzip2 ${previous}"
    bzip2 $previous
  fi
  previous=$f
 done
done


exit 0

Added /etc/cron.d/compress_crontab_logs

23 3 * * * root /etc/cron.daily/compress_tomcat_logs.sh

Tested on a fairly ancient SuSE and debian.

Monday 18 February 2013

In testing prefer lacunae over irreproducibility

When testing you may be tempted to use random inputs. This temptation will occur when you have a large problem space. The wishful thinking here is that the randomness will discover an error where analysis and inspection fail. The tragedy of this approach is that if it were to be successful there is every chance that you could not reproduce the test conditions.

The anxiety which randomness is brought in to quell is that there may be lacunae in non-random generated test data, where two variables vary in lock-step and so never expose a legitimate combination. This anxiety is a genuine one, and is not necessarily exposed by coverage tools. The sticking plaster of randomness is however not the way to address it.

Saturday 16 February 2013

VPN fails to restart after sleep

Thanks to this thread I discover an incantation:
sudo launchctl stop com.apple.racoon
sudo launchctl start com.apple.racoon

Monday 4 February 2013

Create an additional OpenClinica instance

Following on from the situation we were in in previous post we can create a .war file

Using an Ubuntu 12.10 desktop we are going to create an OpenClinica 3.1 instance on a existing SuSe install. This time called researcher.war.


cd /usr/local/oc/distribution/OpenClinica
emacs WEB-INF/classes/datainfo.properties
jar -cvf ../researcher.war *

The properties to change are:

dbHost=db.host.name
#slightly different than default configuration
filePath=${catalina.home}/openclinica.data/${WEBAPP.lower}/
sysURL=https://machinename/${WEBAPP}/MainMenu
log.dir=/var/log/tomcat6/${WEBAPP}

Copy war to remote host and login.


mkdir  /usr/share/tomcat6/openclinica.data/
mkdir  /usr/share/tomcat6/openclinica.data/researcher
chown -R tomcat:tomcat  /usr/share/tomcat6/openclinica.data/

We are using a remote db server. Login to db server, here we have Postgresql 8.4.7 installed.


sudo su postgres
psql
  CREATE ROLE clinica LOGIN ENCRYPTED PASSWORD 'clinica' SUPERUSER NOINHERIT NOCREATEDB NOCREATEROLE;
  CREATE DATABASE researcher WITH ENCODING='UTF8' OWNER=clinica";
  ALTER USER clinica WITH PASSWORD 'XXXX';

Back on the application server


cd /etc/apache2/vhosts.d/
emacs ours.conf

Add our new webapp

        JkMount /researcher/* worker1


/etc/init.d/tomcat6 stop
cp researcher.war /usr/share/tomcat6/webapps/
/etc/init.d/tomcat6 start

This should give you a clean system at https://machinename/researcher/MainMenu with a username of root and a one time password of 12345678.

Postgresql equivalent to show tables, show databases, describe table

Copied from Postgresql: show tables, show databases, show columns, describe table.

-- mysql: SHOW TABLES
\d 
SELECT table_name FROM information_schema.tables WHERE table_schema = 'public';

-- mysql: SHOW DATABASES
\l
SELECT datname FROM pg_database;

-- mysql: SHOW COLUMNS
\d <table>
SELECT column_name FROM information_schema.columns WHERE table_name ='table';

-- mysql: DESCRIBE TABLE
\d+ <table>
SELECT column_name FROM information_schema.columns WHERE table_name ='table';

Also to see the available roles:


SELECT rolname FROM pg_roles;

Friday 1 February 2013

Installation of OpenClinica on Ubuntu 12.10

Installing a new version of Open Clinica on an Ubuntu 12.10 desktop installation.

The hope is to then install this on a SuSE machine which already has a number of Open Clinica instances installed.

This involves creating a .war file with an instance specific name, we will use pets.war.

Register at openclinica.com. Download .zip

Looks like the installation notes are based on something other than the one true distribution; so taking some notes.

You cannot install the default Postgresql. It won't work.

Ensure you have completely removed all traces of Postgresql 9! The following should show no results:


dpkg -l  |grep postgres

Follow the installation of postgresql instructions at OpenClinica.


sudo chown -R postgres:postgres  /opt/PostgreSQL/8.4/

Unpack and copy to expected location:


sudo mv OpenClinica-3.1.3 /usr/local/
sudo ln -s OpenClinica-3.1.3 oc
cd /usr/local/oc

Create database user and database in Postgres:


sudo su postgres
/opt/PostgreSQL/8.4/bin/psql -U postgres -c "CREATE ROLE clinica LOGIN ENCRYPTED PASSWORD 'clinica' SUPERUSER NOINHERIT NOCREATEDB NOCREATEROLE"
/opt/PostgreSQL/8.4/bin/psql -U postgres -c "CREATE DATABASE pets WITH ENCODING='UTF8' OWNER=clinica"

Generate password with Password gorilla, but do not include punctuation.

Change password:


sudo su postgres
/opt/PostgreSQL/8.4/bin/psql -U postgres 
  ALTER USER clinica WITH PASSWORD 'XXXX';

Undocumented
Create a data directory with the webapp name.


sudo mkdir  /usr/share/tomcat7/pets.data
sudo chown tomcat7:tomcat7  /usr/share/tomcat7/pets.data

The way it is recommended you deploy is pretty awful, to paraphrase:

Unzip the .war copy to tomcat webapps directory, edit in place.

This may work on a Wednesday, but is not the right way®.

So what we will do is unpack the .war, stash in git. then build a new .war.

Ideally this would be by using the Maven War Overlay Plugin, but we might get it to work by creating the war manually.


cd /usr/local/oc
git init
emacs distribution/OpenClinica/WEB-INF/classes/datainfo.properties 

You must modify the following:

dbPass=XXXX
adminEmail=admin@example.com
but you probably also need to setup mail sending.

When you have edited these properties, create a .war file, which we will call pets.war


cd /usr/local/oc/distribution/OpenClinica/
jar -cvf ../pets.war *

Deploy:


sudo /etc/init.d/tomcat7 stop
sudo cp pets.war /var/lib/tomcat7/webapps/
sudo /etc/init.d/tomcat7 start

At this point you will need to allow yourself to manage Tomcat7, as it comes secured. Add the following to /etc/tomcat7/tomcat-users.xml.

<role rolename="manager-gui" />
<role rolename="manager-status" />
<role rolename="manager-script" />
<role rolename="manager-jmx" />

<role rolename="admin-gui" />
<role rolename="admin-script" />

<user username="admin" password="admin" roles="manager-gui, manager-status, admin-gui"/>

Unfortunately there is a problem (from localhost.date.log):


Feb 01, 2013 4:16:04 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.akaza.openclinica.dao.core.OCContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/akaza/openclinica/applicationContext-core-db.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set migration/2.5/changeLogCreateTables.xml::1235684743487-0::pgawade (generated):
     Reason: liquibase.exception.JDBCException: Error executing SQL DROP LANGUAGE IF EXISTS plpgsql;
   CREATE PROCEDURAL LANGUAGE plpgsql;
   SET check_function_bodies = false;
   SET client_min_messages = error;
   SET default_tablespace = '';
   SET default_with_oids = true;
   SET search_path = public, pg_catalog;
   COMMENT ON SCHEMA public IS 'Standard public schema';:
          Caused By: Error executing SQL DROP LANGUAGE IF EXISTS plpgsql;
   CREATE PROCEDURAL LANGUAGE plpgsql;
   SET check_function_bodies = false;
   SET client_min_messages = error;
   SET default_tablespace = '';
   SET default_with_oids = true;
   SET search_path = public, pg_catalog;
   COMMENT ON SCHEMA public IS 'Standard public schema';:
          Caused By: ERROR: cannot drop language plpgsql because extension plpgsql requires it
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:281)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:563)
 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
 at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
 at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
 at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
 at org.akaza.openclinica.dao.core.OCContextLoaderListener.contextInitialized(OCContextLoaderListener.java:38)
 at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
 at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)
 at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

Apparently this can be fixed:


sudo /etc/init.d/tomcat7 stop
sudo su postgres
psql -U postgres
  DROP EXTENSION plpgsq;
  \q
exit
sudo /etc/init.d/tomcat7 start

At this point I discover that OpenClinica does not work with Postgresql 9.1!! Blogs are difficult to represent a process. I have updated the above so you won't encounter this error.

You may well see the following, which is a success message !


SEVERE: Servlet /pets threw load() exception
javax.servlet.ServletException: missing jspFile
 at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:123)
 at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
 at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
 at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
 at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5027)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
 at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)
 at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

http://localhost:8080/pets/ takes me to a login screen.

You can now login with a username of root and a one time password of 12345678.

Monday 14 January 2013

Ubuntu setup

After a short toy with the idea that you can develop web service software for Ubunbtu on Windows7, using cygwin, VirtualBox, PuTTY etc I stopped being silly and installed Ubuntu. I had intended to dual boot, but it looks as though Ubuntu was not quite as gentle as it might have been, and Windows got in a tizzy, trying to recover disks, so I blew it away. The joy. It is worth revisiting windows occasionally to re-motivate yourself and renew your passion for opensource.

Setup general packages:

sudo apt-get remove unity-lens-shopping
# Also switch off amazon results on settings >> Privacy


sudo apt-get install openssh-server

#actually I copied these from somewhere else
ssh-keygen

sudo apt-get install git
git config --global user.name "Tim Pizey" 
git config --global user.email Tim.Pizey@psy.ox.ac.uk


sudo apt-get install openjdk-7-jdk

sudo apt-get install eclipse

sudo apt-get install emacs23-nox
sudo apt-get install cvs

sudo apt-get install jedit

sudo apt-get install curl

sudo apt-get install texlive-xetex
sudo apt-get install texlive-latex-extra

#scrollbars in eclipse
sudo su -c 'echo export LIBOVERLAY_SCROLLBAR=0 > /etc/X11/Xsession.d/80overlayscrollbars'


Setup Chrome

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update

sudo apt-get install google-chrome-stable

Setup Haskell:

sudo apt-get install ghc
sudo apt-get install libghc6-zlib-dev
sudo apt-get install cabal-install
cabal update
cabal install cabal-install
sudo apt-get install texlive
cabal install lhs2tex

Jenkins Setup

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

Setup BestMin packages:


sudo apt-get install pgadmin3


sudo su postgres

createuser --superuser --pwprompt dev_user

createdb dolab

exit

sudo apt-get install python

# superstition, maybe
sudo apt-get install python-dev

sudo apt-get install python-pip

sudo apt-get install python-postgresl


sudo apt-get install python-django


sudo apt-get install python-psycopg2


sudo apt-get install gnupg
sudo pip install python-gnupg
sudo apt-get install python-crypto

sudo apt-get install python-yaml


mkdir git
cd git
git clone gitosis@truecolours.nhs.uk:dolab.git

cd dolab/

sudo ln -s /home/timp/git/dolab /usr/local/dolab

./manage.py syncdb

./manage.py runserver 0:8000

Tuesday 8 January 2013

Virtualbox networking

Set up an Ubuntu server. call it psy. You should be able to ssh out from the box to the outside world. The problem is getting in.

VBoxManage modifyvm "psy" --natpf1 "guestssh,tcp,,22,,2222"
VBoxManage modifyvm "psy" --natpf1 "guesthttp,tcp,,8000,,8000"
VBoxManage modifyvm "psy" --natpf1 "guestjenkins,tcp,,80,,8080"

[],tcp|udp,[<hostip>],<hostport>,[<guestip>], <guestport>

See also http://www.virtualbox.org/manual/ch06.html. It may be that this is more effective than http://tim-pizey.blogspot.co.uk/2011/08/alfresco-setup.html