Darren's Developer Diary

Umlauts, pound signs and more!

2012-01-24T19:23:00.000-08:00

Until I learnt this tip, whenever I needed some fancy character (£,°,á,ß, etc.) I went hunting for somewhere to copy and paste it from. Not any more!

I assigned my previously useless windows key to be my Compose key. To type the above pound sign I pressed the windows key, then pressed the L key, then pressed the minus key. Scharfes-S (ß) is simply windows key then press the S key twice. To type á (a with an accent) you press ' (shift-7) and a after the window key.

(Note: until I wrote this post I thought you had to hold the Compose key down while pressing the other keys. That works too, but it is easier to treat it as a modal switch. In other words, you press the Compose key to enter Compose Mode, then the next two characters you type are interpreted together. And if the two characters you type have no special meaning then nothing happens. Either way you exit Compose Mode after your two keypresses.)

See here for the full list of all the compose options

Naturally the above is for linux. This page answers how to do it on Windows and here is the same concept on a Mac

Bonus Tip. Ever wanted to write a Japanese post office sign ( 〒 ) ? Enter your Japanese IME and type yuubin (ゆうびん) (then press space). The ～ symbol can be done by typing kara (から).

˙unɟ ɹoɟ ʇsnɾ ˙uʍop ǝpısdn ʇxǝʇ ɹnoʎ suɹnʇ ʇɐɥʇ ǝʇıs ɐ puıɟ :dıʇ snuoq ɹǝɥʇouɐ
(Yes, this is part of Unicode too.)

boost on centos (vs. on ubuntu)

2011-11-16T06:07:00.001-08:00

You'd think porting a C++ program from one 64-bit linux to another would be trivial. But, no. A program developed with no issues on Ubuntu 10.04 was a lot of trouble to get to compile on Centos 5.6.
But get it to compile I did... read on for the secret words you have to utter...

First thing I did was:
yum erase boost boost-devel

This got rid of the very old 1.33 library. I then ran this:

  yum install boost141 boost141-devel boost141-program-options boost141-regex boost141-thread boost141-system

I then had to hack the makefile to add this to my CFLAGS:

  -I/usr/include/boost141/

That got me compiling. But not linking. I changed my LDFLAGs to look like this but still it would not link:

  -L/usr/lib64/boost141 -lboost_regex -lboost_program_options -lboost_system -lboost_thread

I played around with various -L settings, until I had a breakthrough. I noticed the above line complained with:

  /usr/bin/ld: cannot find -lboost_thread

But some different -L settings instead gave me:

/usr/bin/ld: cannot find -lboost_regex

Blink and you miss it, and indeed I had. My above line was linking with most of Boost, just not boost_thread. In other words, I was really close and hadn't realized it. A bit more poking around discovered that Ubuntu calls it "boost_thread", while Centos calls it "boost_thread-mt" (careful, the first one is an underline, the second one is a hyphen). So, the final magic line for Centos was:

  -L/usr/lib64/boost141 -lboost_regex -lboost_program_options -lboost_system -lboost_thread-mt

NOTE: I did not do any post-installation steps after the yum install steps. I saw some people suggesting copying files from one place to another but I did not need to. (Also, -L/usr/lib64 should be sufficient, as every thing is symlinked; maybe even the -L flag is not needed at all... but I'm in the "If it ain't broke." mindset now, so no more experimentation for me!)

Actual costs: rackspace cloud

2011-11-12T17:21:00.001-08:00

A few months back I decided to put a 24/7 script on a Rackspace Cloud instance, instead of the more obvious Amazon EC2 choice. The reason at the time was my needs were low CPU but relatively high bandwidth and diskspace usage and it worked out cheaper.

Now I've had a few invoices in I am relieved to say there was no catch. My past three invoices have been $11.99, $11.99 and $12.20 (USD). This is for a minimal CPU spec (256MB, 1.6% of a quad core CPU, 10GB disk), 1.1 to 1.3 GB/month of outgoing bandwidth each month (there is no charge for incoming bandwidth), and cloud storage rising from 4 to 8GB. 90% of the monthly cost is for the machine, and the cloud storage has risen from $0.63 to $1.15. The bandwidth is not costing much at all.

In contrast on Amazon EC2, the micro instance would cost $15.65 (including $1 for 10GB of EBS storage), while a small instance would cost $62.25/month, of which $0.03 is the bandwidth usage. (The first year of that micro instance would be free if you are a new customer, but I am not.)

So, at the CPU bottom-end, Rackspace is winning on cost. The other feature of Rackspace Cloud that I love is there is an automatic daily backup of the full disk image, and that backup is stored in the cloud storage. (Storing that backup is basically all my $1/month cloud storage costs.)

What do I not like? I keep using up my 10GB disk space. But there seems no way to move to 20GB without doubling the CPU spec and doubling the monthly cost; with Amazon micro I'd just increase the EBS storage space. With an Amazon small instance I'd get 160GB and would not care.

What do I not like about Rackspace and Amazon? It is that you just get a basic linux distro. You have to spend time installing, configuring and maintaining. And the configuration is not trivial; I've kept a log of all I've had to do, and it includes things like moving ssh off of port 22, setting up an iptables firewall, installing a mail server (not a POP server, just enough so I can *send* email alerts), and writing my own low-diskspace email alert script. The latter was done just the other day after my application broke, yet again, because the machine had run out of disk space.

P.S. As I want 24/7, and have not mentioned the need to scale, what about cheaper shared hosting? Well, I couldn't find a VPS, that gives me root access and no restrictions, for under $10/month. It seems Rackspace is winning that fight too?

PHP, Proxies, HTTPS: v2, v3 or v23?!

2011-11-06T18:43:00.000-08:00

From a PHP http client, using HTTPS via a proxy, I started getting a "400 bad request" error from Apache. I knew it could work because it worked last week. The apache error log message was:
Hostname 127.0.0.1 provided via SNI and hostname mytest.local provided via HTTP are different

My first troubleshooting mistake was messing around with server-side settings: I found out what SNI meant, but as far as I could see I wasn't using it. Then, finally, I remembered I could use curl as a test http client, and it was working fine. I removed all my server-side changes and curl was still connecting fine. So now I knew I'd broken something client-side. I added the -v flag to curl to see the exact headers it is sending. We're both sending the same headers.

Finally I remembered I'd changed this line:
stream_socket_enable_crypto($fp,true,STREAM_CRYPTO_METHOD_SSLv23_CLIENT);

The PHP docs give no guidance on which option to choose, but "v23" sounded like it would work with version 2 or version 3, and maybe do all kinds of auto-negotiation behind the scenes. Which had to be a good thing. I'm sure I'd tested after changing, but I must have tested non-HTTPS or without the proxy by mistake. When I changed back to this line, everything worked again:
stream_socket_enable_crypto($fp,true,STREAM_CRYPTO_METHOD_SSLv3_CLIENT);

I hope that helps someone, as google was no use for me (all the hits about the SNI name and HTTP name difference were due to Apache being case-sensitive about the name comparison, which was not the problem here).

By the way if you want to know how to do http connections, with a proxy, using PHP, supporting both HTTP and HTTPS, I've described it here.

"Git" into this good habit

2011-10-28T00:59:00.000-07:00

This time no story giving insight in to my life as a superhero developer; instead straight to the facts. When you write a .gitignore file, precede each entry with a forward slash. So instead of writing:
*.zip
You should write:
/*.zip

It means zip files only get ignored in the same directory as the .gitignore file, and not in each subdirectory. Do this on principle, and optimize later if you find it is an extension you really want ignoring project-wide. (In other words wait for the noise to appear before using .gitignore to suppress noise.)

Especially important for websites: I may have a temporary zip file in the root of the website from a designer, but deeper in the site I may have zip files that users are supposed to download.

Frapi (PHP web service API system)

2011-10-22T19:12:00.000-07:00

I read about Frapi in PHP Architect (May 2011), and spent a couple of hours trying it out. It is quite interesting, but I don't think I will be using it. It is a full web-interface for making the API. This is what makes it cool, but also its biggest disadvantage. There is a lot of code involved, meaning there is a lot to learn if you need to change it and lots of places for bugs and security exploits to crop up.

It comes with a documentation generator, which could be really useful. This feature is still incomplete (for instance there are no links to it yet, see here, PDF generation didn't work properly), but it looks okay.

There is one specific limitation: I could not create an action with an optional parameter, at least not using the router. E.g. if my action is called "ddd", then I can call /ddd (no parameter). But I cannot call "/ddd/77". (I can give an optional parameter with /ddd?id=77). Or I can define a router as "/ddd/:id" so that I can call "/ddd/77". But in that case id is now required and I cannot use "/ddd".

Another disadvantage is no built-in support for oAuth. (I did find an oauth extension for Frapi but did not try it as the integration seems quite rough still.)

Incidentally if you were looking for a full application built on Zend Framework this may make the perfect study case. Apparently only the admin interface uses ZF, and the actual web services do not; but as far as I can tell they are closely tied and you need to keep both together even on your production servers.

Overall, because making a web service in PHP is not that hard, the advantages are slight and not enough to outweigh the disadvantages (large codebase, inflexible structure, etc.).

R: what to do when the jitter gets too much

2011-10-20T04:57:00.000-07:00

The subject sounds like this will be a discussion of an over-full bladder... instead, brace yourself for some serious functional programming.

So, I had this little bit of R to generate a "rough" exponent curve (as test data):

  x=1:40

  y=exp( jitter( x, factor=30 ) ^ 0.5 )

The problem was it was sometimes giving NA values - too much jitter!
I started off with this fix:

  y[is.na(y)]=exp( x^0.5 )

I.e. wherever it was an NA use the un-jittered value.

Unfortunately it was often complaining with "number of items to replace is not a multiple of replacement length". In this case this was not a warning to ignore, because it revealed I was doing something very wrong. exp(x) is a vector of 40 items. y[is.na(y)] is a vector of however many entries in y are NA. The dangerous thing is if there is just one NA in y, but it is the 5th element, it will be set to exp(1)^0.5, instead of exp(5)^0.5.

Let's cut straight to the solution:
y[is.na(y)]=exp(x[is.na(y)]^0.5)

x[is.na(y)] means the values out of x where the same-sized y-vector has a NA in that position. So, for the above example where just the 5th element in y is NA, then x[is.na(y)] will be "5".

Simple when you know how. (?!)

Renaming files is easy!

2011-10-11T22:22:00.000-07:00

What do you mean? Of course renaming is easy! From the GUI just click it and type the new name. From the linux commandline just type mv oldname newname.

But what about when you have 200 *.JPEG files and you need to rename them to be *.jpg? In the past I've written throwaway PHP scripts to do just this. Well, it turns out linux has a command called rename, that can use regexes. To rename those jpegs:


  rename 's/JPEG/jpg/' *.JPEG

What about a few thousand log files, and I want to remove the datestamp portion from the filename (because I'm going to keep each day's logs in their own subdirectory)? How about this:


  mkdir 20110203
  mv *.20110203.log 20110203/
  rename 's/.20110203//' 20110203/*

(See here for how to then use a bash loop to loop through all the datestamps you have to deal with.)

Add a temporary static IP address

2011-09-17T20:29:00.000-07:00

At home, with wired ethernet, my (Ubuntu) notebook has a few static IP addresses that I use for developing websites. Out of the house, I use wicd, so I have a dynamic IP address, and those static IPs don't exist. wicd configuration is too complex for me to understand, so I just accept this, but it caught me short the other day when I needed to have both an internet connection and to be able to work on a website running on my notebook.

I failed then, but I'm ready for next time. To temporarily add a static IP address you simply do (as root):

ifconfig eth0:3 10.10.10.10 netmask 255.255.0.0

I'm choosing "eth0:3" for the interface; it can be any unused number after the colon, and you never need to care what this is. netmask can really be anything for our purposes. The 10.10.10.10 is the IP address I've given it. Test with this:

ping 10.10.10.10

To set up a quick virtual host create a file under /etc/apache2/conf.d called 10.10.10.10.conf (any filename is fine) with these contents:

<virtualhost 10.10.10.10:80="">
    DocumentRoot "/var/www/somewhere"
    ServerName 10.10.10.10
</virtualhost>

Tidyup

To remove just the interface that you added above, use this command:


ip addr del 10.10.10.10/32 dev eth0:3

Or, to restore the network to boot defaults (useful if you have done lots of changes) you can do:


ifdown -a
ifup -a

Either way to then remove the apache config: delete the 10.10.10.10.conf file you created and restart apache.

node.js: Good Tutorial

2011-09-12T06:08:00.000-07:00

Chatting with a friend at the weekend, node.js came up; I'd only vaguely heard of it, but apparently it is what all the kewl kids are into. Server-side javascript, right up my street.

I took a very quick look yesterday, and it sounded interesting: especially for speed-critical websites. So, today I took a deeper look. First up you should know the official website has a documentation link that only goes to the API documentation. In real terms that means node.js is officially undocumented. Then from the Wiki I found a link to a free e-book called Mastering Node. I won't give the link as it is (currently) over-priced: poorly organized and unfinished.

I was getting a bit demoralized when a search found http://www.nodebeginner.org/. Now this is more like it. In fact this is an excellent tutorial, that goes right from raw beginner to a reasonably complex mini-application. (I read it in HTML format, but as an ebook it is 60 pages, so you can get an idea of how involved it is going to get.)

I followed along, and it was fairly easy, though I did have a lot of trouble outputting the POST-ed data. It always said "undefined" in my browser. I could do a console.log() of the variable just before and just after writing it to the browser and it was set correctly. I cleared the cache repeatedly, and tried a second browser. Annoyingly I didn't solve that problem in the end.

The main point of this blog post is to recommend the above tutorial/ebook if you're interested in getting a feel for node.js, but What Do I Think About node.js?

Hhhmmm. First and foremost, it should only be used by expert programmers. It is a bit like C++ in that it is going to be easy to shoot yourself in the foot. Asynchronous programming is hard. But if you use a synchronous programming-style you will lock up the whole server. I'm thinking in terms of the web server example application here (which is the use-case I had in mind for it). Asynchronous programming is hard. Yes, I know I already said that but I don't think you thought through what that means in the Real World. What will happen is programmers will take a short-cut: they'll use little bits of synchronous code for jobs they know are so quick that no-one will notice. (Even the above tutorial does this: fs.renameSync) The problem is any job involving any I/O device (like a hard disk file system or a TCP/IP socket) will take 10 times longer to finish than average, about 1% of the time. (I made that statistic up, but the principle is true, so stay with me...)

What does that mean? When that happens it will lock the whole web server up, and all the other requests will block. Take the extreme case of doing a sync action that results in a time-out because the resource has gone offline. If the time-out is 30 seconds, the whole web server is down for 30 seconds. All of your customers are getting time-outs. Every image comes up broken.

Another nice thing about the Node Beginner tutorial is the links to deeper information... and nested in one of the comments is a link to a paper comparing threads and events: Why Events Are A Bad Idea Well, their conclusion is in the title, but if you look at their charts the important thing to learn is that well-written event handling code and well-written threading code are basically as quick as each other. (You won't ever reach the right-side of the charts in a real website on a single server; their example is just serving a static image. So the differences are only of interest to academics.)

But, although dealing with threads is hard, asynchronous programming is even harder, IMHO.

Now, if node.js had a web server module where it maintained a thread pool and each new request got its own thread, then I could program in a synchronous style in my thread, happy in the knowledge I won't break the web server, and also happy I'll be able to meet my deadlines...

Using twitter oauth from commandline

2011-09-02T02:10:00.000-07:00

The twitteroauth library is the most recommended PHP library for using oauth for API access to Twitter. It also supports the commandline approach (which can work completely behind a firewall, no need for a web server to host a callback page), but it is not very well documented.
(Note: when I say behind a firewall you do still need web access, because you need to login to twitter to get a PIN code; but this is much less demanding than needing to set up a web server on a public IP address.)

Anyway, after I'd worked it out, I put my sample code up on github. The three files to look at are oob1.php, oob2.php and oob3.php. Here is how you use them:

Step One:
Edit config.php, to set:

      define('OAUTH_CALLBACK', 'oob');

(If not already done, go to https://dev.twitter.com/apps, create and configure the app, and add the consumer key and secret to config.php.)

Step Two:

   php oob1.php

Step Three:
Visit the URL it tells you to, and approve the application

Step Four:

   php oob2.php 1234567

(where 1234567 is the PIN number you got at the end of step three)

Step Five:

   php oob3.php account/verify_credentials

(this command will show your account; see the oob3.php source code for other supported commands, and some shortcuts.)

Let me know if you have any problems with, or questions about, these files.
If you find bugs, or want to encourage its inclusion in the main twitteroauth library, you can comment on the github pull request.

Debugging Regexes

2011-08-09T23:54:00.000-07:00

Cue: dramatic music. There I was under pressure, enemy fire going off in all directions, and my unit test had started complaining. The test regex was 552 characters long, the text being tested was almost as long, and each run of the unit test takes 30 seconds. Talk about finding a needle in a haystack. James Bond only had to choose between cutting the red or the blue wire. He had it easy.

But I lived to tell the tale. Playing the Bond Girl in this scenario was http://www.regextester.com/ (I actually used version 2 which, though alpha, worked fine).

It still wasn't smooth sailing. The above site assumes the regex is surrounded by /.../ but mine wasn't. So, first I had my unit test output the regex, then I escaped it correctly for use with /.../ then pasted it into the Regex Tester. I also pasted in the text to test. It should match; it doesn't. So I put the cursor at the end of my regex and deleted a few characters at a time. After deleting about two-thirds (!!) of it, finally the text turned red and I had a match. I could see exactly where the match stopped and realize what was missing in my regex. I fixed the regex (simultaneously in RegexMatcher and in my unit test script) and repeated. I had to delete back to almost the same point. It took half a dozen passes before the whole regex matched.

The code looks to be open source javascript. So maybe I will hack on it, to automate the above process (my better Bond Girl, if you like): I would give the regex, the target text, say I expect a match, and it will find the longest regex that matches and show me how much of the target text got matched. (Ah, it uses ajax requests to back-end PHP for the preg and ereg versions, and that code is not available; but at least I could do this for javascript regexes.)

Enough with poking around inside today's Bond Girl. Down the cocktail, jacket on, back to the field...

svn, ssh, /bin/false, git, /etc, etc.

2011-08-01T18:32:00.000-07:00

I just spent almost two hours troubleshooting an svn server. It was set up to allow ssh access from a few users over ssh as described in this helpful blog post.

Today I realized none of the svn clients could run svn update: "svn: Network connection closed unexpectedly"

Part of the challenge was that I'd not run "svn update" in 1-2 months, so I had a big time frame for breakage. But all I could remember changing recently was commenting out some lines in my iptables firewall, so I spent quite a while staring at that. And I'd updated some packages and rebooted.

I spent a lot of time looking at ssh verbose output. To do this use

export SVN_SSH="ssh -vv"

(-v for quite verbose, -vv for more, -vvv for even more, etc.) No error messages: it connects, runs the svnserve command, passes up some environmental variables, sends some data, and then simply loses the connection.

Then I wondered if the svn user had somehow lost permission for something important. It has /bin/false as its shell, so I gave it /bin/bash, used su to become root, then "su - svn". Ran the svnserve command. It is fine. Looked at the repository. It is fine. Ran svnadmin to check for locks. None.

Giving up, I started crafting an email to post somewhere asking for help. I went back to get the exact error message svn update was giving me... and it worked. Yep, the other client works now too. My first guess was running svnadmin had quietly fixed something. But then, on a hunch, I changed svn's shell back to /bin/false. Yep, that broke it. It seems the svn user has to have a valid login shell.

Okay, problem solved. But I'm sure that svn has always had /bin/false, and now I wanted to know if that is true, and if and when it changed. It is too late now, but ready for next time, I decided to put all of /etc into a git repository (no need for a central repository, so git is far better choice than svn for this). The git commands (all run as root) are trivial:

cd /etc
git init
git add .
git commit -m "Initial files" -a

I found this page of someone who has done something similar, and he suggested sending the git status report from a daily cron, so I did that too.

It is so easy, and disk space so plentiful, that I think I will do it on all my linux machines.

UPDATE: /etc has some files that get edited a lot, so to reduce noise this is my .gitignore file so far:

cups/printers.conf*
cups/subscriptions.conf*
emacs/site-start*
mtab

(Use "git rm --cached cups/printers.conf*" (for instance) to stop git tracking the files if they've already been added.)

Rcpp: embedding C++ in R scripts

2011-07-31T01:19:00.000-07:00

Rcpp is an R package to aid in making R extensions written in C++. It also includes a side-project for embedding R code in a C++ program (including being able to use all the R packages), but most amazing of all is the integration with R's inline package. Let me show you a minimal (but complete) script:

library('inline')
src='Rcpp::List output(1);output[0]="Hello World";return output;'
fun=cxxfunction(signature(),src,plugin="Rcpp")
fun()

Yes, it is the classic Hello World script, the output looks like:

[[1]]
[1] "Hello World"

What do I call this amazing? The contents of src is the body of a C++ function. The cxxfunction takes your C++ code, sends it to your C++ compiler, compiles it with the necessary headers, and finally embeds it in your R session as an R extension. It just works, there is no catch (*).

(I could have used a character object instead of a list. The Rcpp::List is just like an R list, meaning each element can be of a different type. If you've always wanted something like this in C++, also take a look at boost::any.)

The cxxfunction() call takes 1.9s to run on my machine, so the overhead to start-up the compiler and run the compilation process is not too bad.

Okay, not a motivating example, but the Rcpp project is not just technically amazing, it is also well documented. You can find some more interesting examples in the documentation, and if this has got your interest you will enjoy this video (1.5hrs) of a talk the Rcpp main authors did at Google.

By the way, if anyone knows of a PHP project for embedding C++ like this, please let me know!

*: You need to install the Rcpp and inline packages, and you need R version 2.12.0 or later. If you are using Ubuntu and your default version of R is too old, there are good instructions here on getting the latest R version as a deb package.

R: extract column from xts object as a vector

2011-07-17T18:55:00.000-07:00

A quickie for the R language... you will look at the answer and think it is hardly worth writing a blog post over. Well it turns out that everyone has thought the same... which is why it took me over half an hour of failed searching and trial and error to work it out.

I've an XTS object, x, and I've added a column which is the sum of the other columns:

  x$sum=apply(x,1,sum)

print(x$sum) shows me the sum for each row, along with the datestamp of that row. I.e. x$sum is still an XTS object. Normally this is good, but I wanted it as a simple vector, without the dates associated. Here is how you do that:

  as.vector(x$sum)

Why did I want that? Simply because summary(x$sum) uses up 7 lines, half of it being noise about the datestamps; summary(as.vector(x$sum)) is 2 lines, all signal, no noise.

Customize less: less annoying

2011-07-14T17:26:00.000-07:00

Open ~/.bashrc (or ~/.bash_profile) and add this line at the end:

LESS=-FRX;export LESS

Don't ask why, just trust me. ('cos actually I don't know what it means either...)

This was the magic incantation to tell "git log" to wrap commit messages (instead of just chopping off everything extra).

But it has had the delightful side-effect of man now works properly. About 6 or 7 years ago I used to be able to type "man whatever", press q, and the advice would stay on screen. Then linux (all distros as far as I could tell) changed to hiding the man page as soon as you press q. Frequently I'd have to have two terminal windows open, simply so I could keep the man page open while I type my command. Yes, you're ahead of me; the above LESS command fixes this. If I'd known it would be so easy I'd have done this years ago!!

Even better, the help system in the R language was working just like man pages (and it was even more annoying there), and that has been fixed too.

It is a miracle, I tell you, a miracle! July 13th should go down as "Saint LESS=-FRX" day; in 100 years time it will be a public holiday and children will be taught about it in schools. They might even be taught what it means...

R: foreach parallelization

2011-07-04T01:58:00.000-07:00

I was experimenting with the foreach and doMC packages in R, which make your code multi-threaded. It actually took a bit of refactoring to use foreach(), as there seems to be no shared memory between the threads - they each took their own copy of the in-scope variables. So I had to return my results as a one-row data frame from each iteration, and combine them when the foreach loop had finished.

Here are my results; the user CPU column was giving me strange numbers, so I'm just going to list total time spent (wall clock time). I have three different loops, so here are the base timings (running the foreach loop in sequential mode, without doMC loaded) for each:

4.33         7.72         16.14

My CPU has 4 cores, but the OS sees it as 8 virtual cores. Here are my results for 1, 2, 4 and 8 threads:

1   4.36         7.90         16.20
  2   2.50 [2.18]  4.30 [3.95]   8.80 [8.10]  [8-15% slower]
  4   1.46 [1.09]  2.50 [1.98]   5.06 [4.05]  [25-35% slower]
  8   1.32 [0.55]  2.30 [0.99]   4.30 [2.02]  [110-140% slower]

All times are in seconds, and this loop represents most of the time spent in my script, so while the results are a long way from linear, they represent a useful speed-up. The numbers in square brackets show the speeds if I had got linear improvement.

By the way, my foreach loop had 200 to 250 iterations. The above results tell me that when each foreach loop iteration does more work we get better efficiency. This is fairly course parallelization, which suggests to me that there is lots of room for improvement in the doMC() code.
UPDATE: When running with top, and cores set to 4, I notice 4 new instances of R appear each time it hits the foreach loop, and then they disappear after. I.e. it appears to be using processes, not threads, and creating them on-demand! No wonder my results indicate so much overhead!

Apache: use both PHP module and PHP-CGI

2011-07-03T17:29:00.000-07:00

I had a need the other day to configure apache so it uses the PHP Apache module for all directories except one, where I wanted it to use the cgi version of PHP. The apache module is more efficient, but runs as part of the apache process. I wanted one URL to run in its own process. (I was troubleshooting and had a hunch this might help; luckily it did in this case!) Instructions for both Linux and Windows follow.

On Ubuntu I needed some preparation: 1. install the php5-cgi package (different from php5-cli, which is what we use when using php from the commandline); 2. enable the actions module.
On Windows I was using xampp, which follows an Everything-But-The-Kitchen-Sink philosophy, and all I needed was already there.

Here is the code I added (to my VirtualHost). First for Ubuntu:

ScriptAlias /usephpcgi /usr/bin
Action  application/x-httpd-php-cgi  /usephpcgi/php5-cgi

And then for Windows XAMPP:

ScriptAlias /usephpcgi C:/xampp/php/
Action  application/x-httpd-php-cgi  /usephpcgi/php-cgi.exe

(That is copy and paste from working configurations, so I think the trailing slash must be optional!)

Then for both Apache and Windows:

AddType application/x-httpd-php-cgi .phpcgi

Now, I have to admit my dirty secret: I cheated. Instead of enabling php-cgi for all *.php in one directory, I left *.php going to the Apache PHP module, and I created the *.phpcgi extension to use the cgi binary. Initially this was simply because I managed to get it working that way; but on reflection I realized I preferred it: I can switch a script between using php module and php-cgi just by changing the extension; also I can use php-cgi anywhere in my VirtualHost. If that does not sound so useful I should explain the script in question is already hidden between an Alias, something like this, so no public-facing URLs need to change:

AliasMatch ^/admin/(.*?)/(.*)$ /path/to/admin.phpcgi

What about my original plan to configure it for a directory, without changing the file extensions? I had trouble with this, and gave up on this; but Ben kindly left a comment, so now I know how to do it. First, you still need the ScriptAlias and Action directives shown above. Then it is simply this.

<Directory /var/www/path/to>
    <FilesMatch "\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php-cgi
    </FilesMatch>
</Directory>

As Ben explains (see comment#1 below) the reason we need the FilesMatch inside the Directory is because mod-php is setting a global FilesMatch directive; and that takes priority over our attempts to use AddType or AddHandler for a directory.

R: removing columns in bulk (xts, matrix, data frame, etc.)

2011-06-21T20:18:00.000-07:00

I've an R object, z, with a number of intermediate-working columns I want to delete. In this case it is a quantmod object, but the solution I will show applies for xts and zoo objects, and also for matrices (which is what xts, etc. are under the surface). It also applies to data frames.

(By the way, I use = for assignment, not <-, in the below.) Here is the long-winded approach; setting a column to NULL removes it:

  z$close1=NULL
  z$close2=NULL
  z$close3=NULL
  z$close4=NULL
  z$close5=NULL
  z$open1=NULL
  z$open2=NULL
  z$open3=NULL
  z$open4=NULL
  z$open5=NULL

However using a string for the column name won't work. In other words, these don't work:

  z['open1']=NULL   #This one does work for data frames though.
  z[['open1']]=NULL

The solution I eventually worked out is:

  z=z[, setdiff(colnames(z),c('close1','close2','close3','close4','close5','open1','open2','open3','open4','open5')) ]

colnames(z) gets the current list of columns.
I then use setdiff() to subtract from that the list of columns I want to remove.
The z[,...] syntax means take a copy with just these columns (keeping all rows).

If you don't understand my motivation, I should first say I had more like 30 fields, not just the 10 shown in the example above. But also I like doing things programmatically, as it can avoid introducing typos. For instance, the above solution could be rewritten as:

  #Remove the openN and closeN columns
  z=z[, setdiff(colnames(z),c(paste('close',1:5,sep=''),paste('open',1:5,sep='') )) ]

PHP PDO: so hard to debug

2011-06-16T19:48:00.000-07:00

I wrote a simple PDO helper function to update fields in a certain database table. The fields are given as the key/value pairs in $d, and my function looked like this:

$q='UPDATE MyTable SET lastupdate=:lastupdate';
foreach($d as $key=>$value)$q.=', '.$key.'=:'.$key;
$q.=' WHERE username=:username';

$statement=$dbh->prepare($q);
foreach($d as $key=>$value)$statement->bindParam(':'.$key,$value);
$statement->bindParam(':lastupdate',date('Y-m-d H:i:s'));
$statement->bindParam(':username',$username);

It all looks reasonable doesn't it? Create the SQL, then assign the values to. But it didn't work. My $d array looked like:

array( 'status'=>'expired', 'mode'=>'' )

Instead of getting set to expired, the status field ended up blanked out. Yet lastupdate and username got set. This had me scratching my head for ages.
PDO has a debug function that is next to useless: it tells you the parameters, but not the values you've assigned to them. Incredibly annoying.

Have you spotted my bug yet?

Here's the answer. Though all the examples in the PHP documentation use bindParam(), the function to assign a value is bindValue(). You should always use bindValue(), unless you actually need the advanced functionality that bindParam() gives you. What advanced functionality you wonder? Instead of assigning the value immediately, it attaches a reference, and uses the final value of that reference variable. You're ahead of me: in my foreach loop the $value variable changes on each iteration.

If PDO had a decent debug function I'd have discovered that in half the time. Oh well, now I know!

svn to git: central repository setup

2011-05-26T00:05:00.000-07:00

The best thing about git is you can just start working; no messing around setting up the central repository, working out secure access, etc. But the learning curve for git, coming from svn, is much bigger than say from cvs to svn. The thing I'd not got my head around was what do you do when you need that central repository? For instance, when you have a 2nd developer! I think I've got it now, so here is my guide on it.

Let's assume it is a website ("mysite"), and we have three machines involved:
devel: where most development is done
store: the central repository server
www: the web server

The website is on devel currently, under git, and we want to check it out to www.
If not a website, but just two developers, then in this example "www" is the machine of that second developer.

For simplicity we'll assume all developers have ssh access to the "store" machine. And that all our git repositories will be kept under /var/git/

1. On store:
cd /var/git/
mkdir mysite.git
cd mysite.git
git init --bare

(see update#1 below, if your git does not support --bare)

2. On devel (from the root of the website)
git remote add origin darren@store:/var/git/mysite.git
git push origin master
(the "origin master" part is just needed the first time)

3. On www,
cd /var/www/
git clone darren@store:/var/git/mysite.git my.example.com
(this creates /var/www/my.example.com and checks out the current version of the site)
(you need to configure your web server to not serve the .git directory, as well as any other files end users should not see)

4. If we make a change on devel and want to update www.
First on devel, do the commits, then:
git push
Then on www:
git pull

5. If we make a change on www, and want to update devel.
First on www, do the commits, then:
git push
Then on devel do:
git pull origin master

Not quite symmetrical is it? To fix this, so in future you just need to do "git pull" from the devel machine too, edit .git/config. You'll see these lines:

[remote "origin"]
url = darren@store:/var/git/mysite.git
fetch = +refs/heads/*:refs/remotes/origin/*

Don't touch them but, just above them, add this block:

[branch "master"]
remote = origin
merge = refs/heads/master

(unless you are doing something clever, you can use that verbatim). Now "git pull" works without having to specify "origin master" any more.

So, "git commit ...;git push" is like "svn commit ...". And "git pull" is like "svn checkout".

UPDATE #1: Older versions of git don't have the --bare function. Do these steps (on "store" machine) instead:

cd /var/git/
mkdir mysite.git
cd mysite.git
git init
mv .git/* .
rmdir .git

UPDATE #2: What to do if you already have some files on www, with some differences from what is in git? E.g. I had a test and live version of a website, with no source control; the test version had a number of extra files. I scp-ed the live version to devel, created a git repository. Then I set it up and push-ed it to "store". Then on my test site, I had to do these steps:

#Temporarily move current test files out of the way
mv my.example.com{,.old}
#Get the git version
git clone darren@store:/var/git/mysite.git my.example.com
#Merge in the test site
cd my.example.com
cp -a ../my.example.com.old/* .
#See what is different
git status
#For all changed files, revert to the git (live server) version
git reset --hard

By the end of all this, git status will just list (as untracked) the files that were on test but were not on live.

PHP, PDO, SQLite, mysterious lock problem

2011-05-24T21:28:00.000-07:00

Let's start with the conclusion:
If doing a prepare() or query() with PDO and sqlite, and then you want to do something else with sqlite in the same PHP function then unset the first PDOStatement, before trying to do that something else.

As an example here is my code, to get a unique ID ($dbh is a PDO connection to an sqlite):

function get_next_id($dbh){
$q='SELECT next FROM MyNextId';
$obj=$dbh->query($q); //Throws on error
$d=$obj->fetch(PDO::FETCH_NUM);
$next_id=$d[0];

$q='UPDATE MyNextId SET next=next+1';
$row_count=$dbh->exec($q); //Throws on error
if($row_count==0)throw new Exception("Failed to execute ($q)\n");

return $next_id;
}

It works on Ubuntu 10.04, with sqlite 3.6.22, but fails on Centos 5.6, with sqlite 3.3.6, with this message:
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 6 database table is locked'

I went through the whole changelog from 3.3.6 to 3.6.22, but got no clues (though I am now impressed with how active and organized the sqlite development is). But finally I tracked down this article on someone getting similar errors.

And that was it. I could have used $obj->closeCursor(), but deleting $obj is just as good:

function get_next_id($dbh){
$q='SELECT next FROM MyNextId';
$obj=$dbh->query($q); //Throws on error
$d=$obj->fetch(PDO::FETCH_NUM);
$next_id=$d[0];

unset($obj);

$q='UPDATE MyNextId SET next=next+1';
$row_count=$dbh->exec($q); //Throws on error
if($row_count==0)throw new Exception("Failed to execute ($q)\n");

return $next_id;
}

If you are doing just one PDO action per function then there is no need, because exiting the function will automatically do the unset.

(I don't know why this is a problem on sqlite 3.3.6 but not sqlite 3.6.22... in fact, I suspect it may be due a difference in the PDO or PHP version or configuration instead. Apologies for the loose end!)

oauth for php (and Ubuntu)

2011-05-22T03:10:00.000-07:00

There is a "standard" PHP OAuth library, documented in the manual, and which is installed via pecl. There is no package shortcut under Ubuntu; you still have to use pecl
sudo pecl install oauth

If you get a complaint about no "phpize", install the "php5-dev" ubuntu package. And then if you get an error in "php_pcre.h" when compiling, then you need to install the "libpcre3-dev" ubuntu package.

Finally, you need to enable it. Still as root:
cd /etc/php5/conf.d
echo "extension=oauth.so" >oauth.ini

(This creates a config file just for oauth; you could also simply put the extension line in php.ini.)

Finally, "php -m" should list "OAuth", and you can create a OAuth object in your php scripts.

Ubuntu package manager lists "liboauth-php"; the minimal information, and the lack of mention of pecl, should have given me the clue that it is something different.

Also different is this: http://code.google.com/p/oauth-php/

One man's feature request...

2011-05-16T19:26:00.000-07:00

...is another man's bug.
Since Thunderbird 3, if you selected part of a mail then hit reply, only your selected part gets quoted. Incredibly annoying for those of us who think out loud with our mouse.

But it can be switched off. Edit|Preferences, Advanced tab, Config Editor. Then in the search box type:
mailnews.reply_quoting_selection
Change it from true to false (double-click that line is enough).

It turns out this was actually a Thunderbird feature request, dating from 2000! https://bugzilla.mozilla.org/show_bug.cgi?id=23394

ZendForm: inserting static text

2011-02-07T00:49:00.000-08:00

I'm not a fan of Zend Form, or Zend Framework generally: too much structure, too much work to do anything slightly unusual. Trying to add static text to your form epitomizes this. Note, you have to work through the $form object, as in the view page this is all there is:

  <?=$form ?>

If you are using php 5.3, there is a quite reasonable solution:

  $form->addElement('text','whatever_id',array(
    'decorators'=>array(array('Callback',array('callback'=>
    function(){return "<h3>Whatever</h3>";}))),
    ));

Yes, that is as minimal as I can get it, and yes, you must give a 'whatever_id' that is unique on your form, but you have complete control over what is output, and nothing else gets output.

If you are using php 5.2, there are two approaches. The first way is equivalent to the above, just slightly more verbose:

  $form->addElement('text','whatever_id',array(
    'decorators'=>array(array('Callback',array('callback'=>
      create_function('','return "<h3>Whatever</h3>";')
      ))),
    ));

The second way is a different approach:

  $form->addElement('text','whatever_id',array(
     'label'=>E('<h3>Whatever</h3>'),
     'helper'=>'formNote',
     'decorators'=>array(array('Label',array('escape'=>false))),
      ));

Yes, four lines, and every line matters. Here is what gets output:
<label class="optional" for="whatever_id"></label><h3>Whatever</h3>

I cannot stop the <label> tag appearing; If you can, let me know. (It does not appear to the end user, just in the HTML source, so this is not a serious issue.)

(2011-03-02: Edited to show the create_function() approach in php 5.2)

Using Amazon EC2 for Shodan Go Bet

2011-01-28T16:14:00.000-08:00

I've put up a short article on how Amazon EC2 was used for the Shodan Go Bet event held at the end of December 2010:
http://dcook.org/gobet/using_amazon_ec2.html

It is not about computer go, but instead will be of interest if you have wanted to see a concrete example of how EC2 is used, just how fast the fastest choice is, exactly how much it costs, how the Amazon EC2 Windows instances work (I used remote desktop running on linux!), etc.

Buffalo Air Station setup on existing home LAN

2011-01-18T20:45:00.000-08:00

I found this surprisingly difficult; though it turns out the steps involved are quite easy...once you know how.

For this article I'll assume your existing LAN is 10.0.0.0/8 with 10.0.0.1 as the default gateway. I'm going to give the wireless LAN router 10.1.2.3. Substitute 10.0.0.1 and 10.1.2.3 below for something suitable for your LAN (and change the 255.0.0.0 network mask accordingly).

These instructions assume factory reset status, with the "router" switch (on the case) set to "On", rather than "Off" or "Auto". The AirStation is connected to the LAN hub using the Internet socket (the blue one).

1. Connect a computer to the Air Station directly (with ethernet cable, to one of the four LAN sockets), and set your computer to use DHCP, so that it gets an IP address that can see the Air Station! (Most likely your computer will receive 192.168.11.2.)
Note: The computer you connect with does not need to have a wireless card in it; it does have to have wired ethernet though.
Tip: Don't use your main computer for this step; that way you will be able to still use your main computer for: a) googling when you hit problems; b) ping tests (see step 5 below).

2. Connect by browser to 192.168.11.1
Use root as the username, with blank password.

3. Internet/LAN | Internet
IP Manual: 10.1.2.3
255.0.0.0
Extended:
Default GW: 10.0.0.1
DNS: (whatever DNS server you use: this is what will be handed out to wireless clients that connect using DHCP)

4. Wait for it to restart. (I also enabled pings at this point too.)

5. Then cycle the power; this appears to be essential.
Now you should be able to ping 10.1.2.3 from outside, and also ping from the connected computer to 10.0.0.1. And if you can do that then you can also connect to internet! You are done, time for a nice cuppa.

6. Test from a wireless device to make sure you are required to type in the key.

7. Set a root password.

scp with multiple targets: ssh-add

2010-12-07T16:45:00.000-08:00

I sometimes have the need to upload the same file to two or more locations on the target server. In fact I even have a script to help me, which goes something like this:

scp file1 file2 remote01:/path/to/somewhere/
 scp file1 file2 remote01:/path/to/another/

The problem is I need to type the password for remote01 twice. I had never managed to find some clever scp syntax to allow specifying two destinations, and a post on the TLUG mailing list confirmed that. But what I did learn from TLUG is that there is something called ssh-agent that can store passphrases for key pairs; this plugged a gap in my knowledge. There are three ways to login in via ssh/scp:

Give your password
Make a keypair with no passphrase
Make a keypair with a passphrase

The second way is used to allow scp from cron jobs, to automate copying files between two machines. I'd never really got the point of the third way: if you have to type a passphrase why not just use the first method, and save messing around making a key pair. (Well, yes, there is better security: a log-in then requires both something I have and something I know; you have to turn off PasswordAuthentication on your ssh server for this to have any meaning though. Thanks to Kalin for this comment.)

But it turns out there is this program called ssh-agent that remembers passphrases for you. And I found it is already running in the background on Ubuntu.

Enough chat, let's look at the solution. First, I created a one-off keypair for this script, on my machine, using something like this:

cd ~/.ssh
  ssh-keygen -t dsa -C me@example.com -f key_for_remote01
  chmod 600 key_for_remote01*
  scp -p key_for_remote01.pub remote01:~

When generating the keypair give a reasonably secure passphrase (you will have to type it in each time, and it is only of use to people in possession of key_for_remote1, so no need for a 20-random-character monster; I believe it is perfectly fine for it to be the same as your normal ssh login password for remote01).

Then log in to remote01 and append key_for_remote01.pub to ~/.ssh/authorized_keys. If that file does not exist then you can just rename key_for_remote01.pub to authorized_keys and move it into ~/.ssh/

(By the way, there is no need to put your private half of the keypair in ~/.ssh/ but that seems as good as place as anywhere else.)

Now, I modified my script as follows:

ssh-add -t 120 ~/.ssh/key_for_remote01
 scp -i ~/.ssh/key_for_remote01 file1 file2 remote01:/path/to/somewhere/
 scp -i ~/.ssh/key_for_remote01 file1 file2 remote01:/path/to/another/
 ssh-add -d ~/.ssh/key_for_remote01

What happens is the ssh-add line will ask you for your passphrase. The two scp lines then work automatically. Finally the ssh-add -d stops it caching your passphrase (forcing you to type your passphrase each time you run this script).

The -t 120 parameter says the passphrase will expire after two minutes. This is just in case the batch file doesn't complete and so does not get chance to run ssh-add -d.

Note: you can use that same key pair for other machines. Basically anywhere you put the *.pub half of the key pair will let you login. And you can login from anywhere you have the private half of the key pair.

Note: the timeout/deletion code above is deliberate for this application, but you don't have to do it this way. By allowing it to cache it permanently you would only be prompted for your passphrase once, and then all future ssh and scp logins would be automatic. They will be cleared when you log-out of gnome (on ubuntu, at least) or shutdown your machine.

Note: If you don't want to specify -i each time you use ssh/scp then you can add an entry to your ~/.ssh/config file, like this:

Host remote01
        Hostname 10.1.2.3
        Port 22
        IdentityFile ~/.ssh/key_for_remote01

Note: I used -C with my real email address. This is put in the public key, and I wanted the administrator of remote01 to know who put the key there. Without -C it defaulted to "myusername@myhost". The administrator of remote01 knows nothing about my machine names so this seemed unreasonable and I decided to use -C. But the advantage of that default is it seems ssh-agent knows about that name and will prompt automatically the first time you try to use ssh/scp, which means there is no need to run ssh-add first. I have not worked out yet if ssh-agent can be told to know about my email address too.

Troubleshooting php and xinetd with strace

2010-11-28T18:42:00.000-08:00

In a previous entry I mentioned I was trying to track down a difference in a php script between two machines. I've now got closer, close enough in fact to fix the problem but not to understand it. As a troubleshooter, the big discovery for me was how useful strace can be.

First, a PHP code snippet:

while(1){
stream_set_blocking(STDIN,true);
$s=trim(fgets(STDIN));
if($GLOBALS['log_fp'])fwrite($GLOBALS['log_fp'],$s."\n");
$params=explode(" ",$s);
$command=array_shift($params);
...
}

The first two lines say listen on stdin for a command, and wait forever for it to arrive. When it does we log it and then split it up into a command and parameters.

This script is used over xinetd. Xinetd listens on a port for us, and when it gets something it feeds it to stdin; php script output is then fed back over the socket to the client.

Here is the setup:

machine A: ubuntu 8.04, 32-bit, php 5.2
machine B: ubuntu 10.04, 64-bit, php 5.3

Both machines run an indentical version of xinetd: "xinetd Version 2.3.14 libwrap loadavg".

Machine A runs fine, machine B sometimes sits there. Analyzing this some more, by connecting over telnet to machine B:

I send a command
It replies straightaway
After 60 seconds or so it gives an error message
If I do nothing this error message appears again every 60 seconds.

On machine A, or if I run the php script directly on machine B, it behaves like this:

I send a command
It replies straightaway
It sits there forever waiting for input

Out of desperation I ran strace (using -p you can attach it to a running process; use ps -a | grep php to find the PID), and I was pleasantly surprised to see it was not too verbose.

Over xinetd on machine B (the problem configuration) the interesting snippet is:

...
write(1, "mydata\n", 7)                 = 7
fcntl(0, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(0, F_SETFL, O_RDWR)               = 0
poll([{fd=0, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 0 (Timeout)
write(3, "\n", 1)                       = 1
write(5, "\n", 1) 
...

On machine A over xinetd it instead looks like this:

...
write(1, "mydata\n", 7)                 = 7
fcntl64(0, F_GETFL)                     = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl64(0, F_SETFL, O_RDWR|O_LARGEFILE) = 0
read(0,

(Yes nothing comes after the "0," until I send some input.)

And direct access to the php script on machine B it practically the same:

...
write(1, "mydata\n", 7)                 = 7
fcntl(0, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl(0, F_SETFL, O_RDWR|O_LARGEFILE)   = 0
read(0,

So, the cause is clear: fgets() blocks for only 60 seconds instead of blocking forever, but only on this machine and only over xinetd. And the bug in my script then becomes clear: I'm not prepared to handle blank input!

Here is my fix:

while(1){
stream_set_blocking(STDIN,true);
$s=trim(fgets(STDIN));
if($s=='')continue;
if($GLOBALS['log_fp'])fwrite($GLOBALS['log_fp'],$s."\n");
$params=explode(" ",$s);
$command=array_shift($params);
...
}

As an ironic postscript, the real problem was elsewhere (a mismatch in another program version and the configuration being used for it!), and I also used strace to track that mistake down. But, what was so important about fixing the above was that it stopped distracting me with an error message every 60 seconds on machine B, when the configuration error was actually on machine A!

php 5.3, ticks, pcntl_signal, pcntl_signal_dispatch

2010-11-27T18:22:00.000-08:00

I'm trying to track down a problem with a script that works on php 5.2 but behaves strangely on php 5.3 (there are lots of differences between the environments, and I suspect php version will actually turn out to be completely unrelated). php 5.3 introduced pcntl_signal_dispatch() which processes outstanding signals and I've been investigating if that could somehow explain the behaviour differences I see.

The confusing part is that I've seen people saying that the old way of "declare(ticks=1)" is now deprecated in 5.3, and you must use pcntl_signal_dispatch(). This seemed very silly as you'd have to litter your code with calls to pcntl_signal_dispatch(), as well as have very different code for php 5.2 and 5.3.

If you're confused, like I was, here is what you need to know:
1. declare(ticks=1) still works: no deprecated message (with E_ALL|E_STRICT error reporting);

2. My script, using ticks, behaves identically under 5.2 and 5.3 when I send it a ctrl-C or a kill signal;

3. The docs don't mention it being deprecated; I realized everywhere saying this was user-contributed comments or blogs!

There is a performance aspect with using declare(ticks=1). I believe it is minor, but I think pcntl_signal_dispatch() has been introduced so you can use it instead of ticks if you want to take fine-control over when signals get considered.

bash tips: meld on program output

2010-11-25T19:57:00.000-08:00

Here is the challenge: I want to run diff on two large log files, but I'm only interested in the entries at a certain time in each log file.

This used to require four commands:

grep "ABC" a.txt >tmp1.txt
  grep "ABC" b.txt >tmp2.txt
  diff tmp1.txt tmp2.txt
  rm tmp1.txt tmp2.txt

(Imagine "ABC" is a datestamp, but it could be any other way to filter your log file.)

Thanks to the gurus on TLUG's mailing list I can now do this as a one-liner:

diff <(grep ABC a.txt) <(grep ABC b.txt)

It works perfectly for meld (a wonderful visual diff program) too!Here is another way to use it, to compare the output of a program on two different machines (here I'm comparing the php configuration):

diff <(php -i) <(ssh someserver 'php -i')

We're using the form of ssh that runs a program on the remote server.The command in the brackets can get quite complex. Here is an example where I needed to compare datestamps in two csv files, but the first field was an id number, arbitrary, and therefore different for all records.

diff <(egrep -o '[0-9]{8} [0-9]{2}:[0-9]{2}:[0-9]{2}' dir1/abc.csv)
  <(egrep -o '[0-9]{8} [0-9]{2}:[0-9]{2}:[0-9]{2}' dir2/abc.csv)

The -o flag to egrep tells it to only output the matching part, not the whole line. This next version shows the complete rest of the line starting with my datestamp field; i.e. this version just excludes the csv field(s) before the datestamp field:

diff <(egrep -o '[0-9]{8} [0-9]{2}:[0-9]{2}:[0-9]{2}.+' dir1/abc.csv)
  <(egrep -o '[0-9]{8} [0-9]{2}:[0-9]{2}:[0-9]{2}.+' dir2/abc.csv)

Bash loops

2010-10-18T19:54:00.000-07:00

Faced with having to write 160 commands by hand (to split 5000 files into subdirectories, based on the first 3 digits of their 6 digit filenames) I quickly learnt bash loops instead:

for i in `seq 102 189`;
  do
    mkdir $i
    mv $i???.sgf $i/
  done

I.e. it does commands like this:

mkdir 102
  mv 102???.sgf 102/
  mkdir 103
  mv 103???.sgf 103/
  ...

UPDATE: In a comment, traxplayer kindly explained how I would make, for instance a filename like 120abc.txt; obviously writing $iabc.txt won't work. The trick is to write $i as ${i}. E.g.

mv ${i}abc.sgf $i/

I just needed to use it, and it worked! This bash script tries to find the last reference to each of XXX06..XXX20 in two logfiles.

rm lastorder.log
touch lastorder.log

for i in `seq 6 9`;
    do
        grep -h XXX0${i} order_verbose.log.old order_verbose.log | tail -1 >> lastorder.log
    done

for i in `seq 10 20`;
    do
        grep -h XXX${i} order_verbose.log.old order_verbose.log | tail -1 >> lastorder.log
    done

By the way, traxplayer also said he prefers to use $(...) instead of `...` for readability:

  for i in $(seq 102 189);

When bash history no longer works

2010-09-13T20:09:00.000-07:00

I've been pulling my hair out: I have all the HISTORY variables set correctly, but history never survives to the next session. I thought it must be an Ubuntu 10.04 bug, as only that machine, but couldn't track down anyone reporting the same problem.

The breakthrough came when I stopped typing "history" to view the history, and decided to "cat ~/.bash_history". I got permission denied... and discovered the file was owned by root. Suddenly it all made sense.

(By the way, I've a bunch of blog articles about setting up a new Ubuntu dual boot, RAIDed, encrypted machine; coming soon to an RSS feed near you...)

Putting parentheses into complex doctrine sql queries

2010-08-22T18:34:00.000-07:00

Consider this query:

SELECT * FROM mytable WHERE
( status IN ('A','B') OR access IN ('X','Y') )
AND created_at > '2010-01-01';

Doctrine offers no way to do this! The following code makes a query without the parentheses, meaning it will return all records with status of A or B, whatever their creation date.

$q=Doctrine_Query::create() ->from('mytable');
$q->whereIn('status',$statusList);
$q->orWhereIn('access',$accessList);
$q->andWhere('created>?','2010-01-01');

If your IN lists are of fixed length you could do it like this (untested):

$q=Doctrine_Query::create() ->from('mytable');
$q->where('( status IN (?,?) OR access in (?,?) )',array('A','B','X','Y'));
$q->andWhere('created>?','2010-01-01');

But a far better solution is suggested on Scott Daniel's blog

I changed his code in two ways. First I put it straight into Doctrine/Query.php, which saves me having to change the class name in my source code. I'm comfortable doing this as I have the Doctrine source in SVN, meaning I won't lose my change when updating Doctrine. Secondly I used an explicit reference to save having to reassign the variable. I doubt there is any difference in speed, but this way looks clearer to me. So the full code I put at the end of Query.php looks like this:

/**
* Custom addition to Doctrine, to allow wrapping a set of OR clauses
* in parentheses, so that they can be combined with AND clauses.
*
* @see http://danielfamily.com/techblog/?p=37
* I modified it slightly to use an explicit reference.
*
* @return Doctrine_Query this object
*/
public function whereParenWrap()
{
$where = &$this->_dqlParts['where'];
if (count($where) > 0) {
    array_unshift($where, '(');
    array_push($where, ')');
    }
return $this;
}

To use it simply add a call to whereParenWrap() just after your OR clauses. For instance, here is how my original example is modified:

$q=Doctrine_Query::create() ->from('mytable');
$q->whereIn('status',$statusList);
$q->orWhereIn('access',$accessList);
$q->whereParenWrap();
$q->andWhere('created>?','2010-01-01');

In my actual code this required me to shuffle things around so the OR clauses were defined first, then the straightforward AND clauses came last. But that was no hardship.

However there is one flaw in this approach: if you need to have two such sets of OR clauses, which are themselves joined by an OR or AND. I thought a more generic approach might work: openParenthesis() and closeParenthesis() functions which you call exactly where you want them. But openParenthesis goes wrong (it ends up making "WHERE ((AND status IN..."; i.e. the "(" looks like an existing clause to Doctrine). I'm sure this could be made to work, but it will be more intrusive. So I'm going to be pragmatic and worry about this should I ever actually need it.

Doctrine many-to-many relationships with onDelete

2010-08-03T00:57:00.000-07:00

In a previous article I explained why I now try to put an explicit onDelete on every relation. The problem comes with many-to-many relationships. If you put onDelete: CASCADE in the relationship it gets ignored and the entries survive in your refclass table. I found some page on this but none that really answers the question fully:

http://stackoverflow.com/questions/1488171/in-symfony-doctrines-schema-yml-where-should-i-put-ondelete-cascade-for-a-many

http://groups.google.com/group/doctrine-user/browse_thread/thread/2f924ee3b8cd2689?pli=1

http://trac.doctrine-project.org/ticket/1123

The common theme is if you want an onDelete on a many-to-many relationship then you have to put it on the reference table. But not a single example anywhere. So, let's start with the "friends" nest example in the manual.

Here is what does not work (no error, simply leaves the records behind in FriendReference)

User:
# ...
relations:
    # ...
    Friends:
      class: User
      local: user1
      foreign: user2
      refClass: FriendReference
      equal: true
      onDelete: CASCADE

FriendReference:
columns:
    user1:
      type: integer
      primary: true
    user2:
      type: integer
      primary: true

This one won't even compile (well, I didn't think it would):

User:
# ...
relations:
    # ...
    Friends:
      class: User
      local: user1
      foreign: user2
      refClass: FriendReference
      equal: true

FriendReference:
columns:
    user1:
      type: integer
      primary: true
    user2:
      type: integer
      primary: true

relations:

onDelete: CASCADE

So, how do we change it so it will work? I'll update this if and when I work it out; if you know please let me know.

Be explicit about OnDelete in doctrine schemas

2010-08-03T00:31:00.000-07:00

In doctrine, I tried $obj->delete(); and got an SQL exception from another table. But you're supposed to have deleted that too, that's the whole point of me telling you about all the relations between the tables, screamed I.

So I went through my schema inserting "onDelete: CASCADE" in every relation. But as I did it realized sometimes I wanted "onDelete: SET NULL", and sometimes I wanted "onDelete: RESTRICT". This is a helpful article on the onDelete choices (though for null you actually have to write "SET NULL" not just "NULL". There is also another choice: "OnDelete: SET DEFAULT" (e.g. when a user is deleted, you may want his comments to become owned by the admin user, rather than become orphaned.)

So, now my advice is to explicitly put onDelete on every relation; treat it as required. I also find I need a line of documentation to explain every time I choose something other than cascade. It is good to be forced to think of these things.

All is smooth until you come to many-to-many relationships, especially self-referential (nest) relationships, such as two users being friends (symmetrical) or one user blocking another (one-way). I will cover it in a separate article.

Wine broke, or PHP, or xinetd??

2010-07-25T23:01:00.000-07:00

Sometimes it can be really educational to sit on the shoulder of brilliant developers and watch how they troubleshoot and debug. So, here is your chance to sit on my shoulder,... and snigger as failure reduces me to tears.
[2010-11-18 UPDATE: a mere 5 months after originally posting, I think I have the answer; see the bottom of this article.]

A program worked fine Saturday. This is on my Ubuntu 8.04 (hardy heron) machine.

Sunday I updated firefox and ghostscript only. I then rebooted, which means kernel 2.6.24-28 finally became active (I updated it July 11th, but hadn't had chance to reboot until yesterday).

Today that program doesn't run. It is one particular wine program: a go program. Another wine program (also a go program) runs fine. And here is the real killer: if I start that problem wine program from gogui (gogui.sf.net) it works. Exactly the same commandline, but it works from gogui, and doesn't work when started from my php script. That PHP script was last changed last Wednesday.

I rebooted into the previous kernel (2.6.24-27), and the problem is exactly the same.

Have you ever done those logic problems where you get a list of clues and have to work out who did what? Applied to the above we discover the only logical explanation is that... reality is warped. It can't be: wine, the go program, php, my php script or the kernel.

Here is some more evidence. The first time I run it after a reboot the program (called valhallgtp.exe) fails to start with a stack dump and backtrace. Here are the first few and last few lines of that:
=>1 0x7bc3b23c __regs_RtlRaiseException+0x4c() in ntdll (0x014ded68)
2 0x7bc76de3 in ntdll (+0x66de3) (0x014df0cc)
3 0x7bc3a936 RtlRaiseException+0x6() in ntdll (0x014df144)
4 0x00415833 in valhallgtp (+0x15833) (0x014df2ac)
5 0x004159dd in valhallgtp (+0x159dd) (0x014dfa24)
... (6..20 are all in valhallgtp)
21 0x0049b02c in valhallgtp (+0x9b02c) (0x014dff08)
22 0x7b8773a7 in kernel32 (+0x573a7) (0x014dffe8)

I also see: "err:seh:raise_exception Exception frame is not in stack limits => unable to dispatch exception."

The 2nd and subsequent times I try to start it I get:
Failed to start wine ValhallGTP.exe ...
(this is an error from my php script; i.e. proc_open() is failing.)

The CWD (current working directory) is correct, the parameters are correct; both are exactly what gogui is using. And running it from a bash shell is fine too.

The fingers are pointing at PHP. But no PHP upgrades either yesterday or in the July 11th batch. And that same PHP is still successfully starting 4 other programs, including another one that uses wine. (BTW, go to synaptic package manager, file menu, history; this is where you can see exactly what Ubuntu updated and when.)

Disk space is fine ("df"), memory is fine ("cat /proc/meminfo"). Machine load is low ("w"). Running it as root has same problem.

When I ran from the -28 kernel I also got errors about no X DISPLAY. And then when I started the other wine program it opened but had no display. And running wine config was the same. I've not seen that again. Let's pretend it never happened and concentrate on what we can reproduce.
(UPDATE: it happened again (this time with the -27 kernel). I had the gtpmfgo.exe running fine under wine; then starting mfgo.exe it came up but with no display. Close and retry showed it was repeatable. After closing gtpmfgo.exe it worked, and then I could open gtpmfgo.exe fine. Note: not repeatable; I haven't managed to get it to happen again.)

Okay, where to next? What I don't understand is how come I cannot reproduce the problem I get when I start it the first time. That seems like a small enough challenge to be achievable: how to get it to crash consistently.

...time passes...

Aha! It seems it is crashing in the same way; it was just crashing and dying before I got chance to read the crash message. So the first time after a reboot it must take longer than a second to start up (perhaps wine initialization).

...aha, aha! Narrowed down further. To explain this I need to give more background: I have a script called frontend.php that calls backend.php over sockets. backend.php uses stdin/stdout and uses xinetd to turn it into a socket server. When I run backend.php directly it works!

Could the updates/reboot have affected xinetd somehow? ...probably not. The xinetd files are all dated Dec 2007 except my own config file which is June 30th. And, the other 4 programs are started in exactly the same way and work fine.

Starting backend.php in exactly the same way as xinetd does works fine. (I've tried it both from bash and from sh.)

...time passes...

I seem to have a solution. (I had started working on a xinetd replacement php script, but that doesn't seem needed now.) My solution was to set xinetd to listen on another port, and use that port for just the problem program. This is working. I cannot explain it however: I had already tried connecting to only that program, so xinetd wouldn't have been doing anything different to now.

UPDATE: 7 weeks later and I do another reboot and it broke in exactly the same way!!
And again the only solution I could find was to give xinetd a new port for just this particular problem.
Weird, very, very weird, ...

2010-11-18 UPDATE: It was bothering me again, but on a different machine, and the trick with setting up xinetd on a different port didn't work. But adding this code to the top of backend.php seems to have solved it:
if(trim(getenv('DISPLAY'))=='')putenv('DISPLAY=:0.0');

The problem program has a GUI you see. I switch the GUI off with a commandline option ("hidewindows") but I think it actually creates windows and just hides them. And yes, none of the other programs uses a GUI... sigh, the clues were there.

Curiously the "hidewindows" option no longer works! That is probably going to annoy me, but not as annoying as not working at all!

Zend Form: display group and custom decorator

2010-06-24T19:10:00.000-07:00

I keep meaning to write a proper review of Zend Framework, but I am waiting to finish a big project that uses it. It is running late, and I think ZF can take some of the blame. So don't hold your breath waiting for a positive review.

Today's topic: I want to have part of my form hidden initially, and instead show a button saying: "toggle more questions".

So, we make it a display group (where ex1 and ex2 are the form elements to hide):

$form->addDisplayGroup(array('ex1','ex2'),'extras');

By default this wraps it in a fieldset, with no place I could see to slip in my CSS, javascript and the toggle link. What I need is a decorator, I said to myself.

Oh, gasp, does Zend make this difficult or what! In another part of this project custom decorators had been used for a minor layout change. 7 classes in 3 directories, almost all of it boilerplate. The real work was being done in CSS; those 7 classes were just to give a way to name the items as far as I could tell.

The problem is the Zend Framework Philosophy of making things complex. Did you realize there is no addDecorator($myclass) function! You have to keep decorators in a special directory and then tell Zend where to get them. Then addDecorator('part_of_my_class_name').

ZF's saving grace is that it is open source. So I poked around, and here is my solution. First, I'll define an alias for readability (optional):

$displayGroup=$form->getDisplayGroup('extras');

Remove all the default stuff I don't need (this step is optional too):

$displayGroup->removeDecorator('Fieldset');
  $displayGroup->removeDecorator('HtmlTag');
  $displayGroup->removeDecorator('DtDdWrapper');

Now insert my decorator (these 4 lines are in lieu of addDecorator($myclass)):

$decorators=$displayGroup->getDecorators();
  $decorators['MyTest']=new MyTestDecorator();
  $displayGroup->clearDecorators();
  $displayGroup->addDecorators($decorators);

(I.e. get the current decorators, add mine, then replace the existing decorators with the new set.)

Finally we get to the meat. All you need is to define a render() function that takes a string (the existing content) and returns that string, optionally modified. Here is the minimal version that does nothing.

class MyTestDecorator extends Zend_Form_Decorator_Abstract
{
  public function render($content){ return $content; }
}

And here is the full version: it hides the elements in the group and uses JQuery to show/hide it. The CSS is inline.

class MyTestDecorator extends Zend_Form_Decorator_Abstract
{
  public function render($content)
  {
    $js="$('#extra_questions').toggle();return false;";
    return '<a href="" onclick="'.$js.'">Toggle Tags Visibility</a>'.
      '<div id="extra_questions" style="display: none;">'.$content.'</div>';
  }
}

Yep, it's that simple. Oooh, the architects of ZF must be turning red with rage ;-)

UPDATE: The best article I've found so far on Zend Form Decorators. As many of the comments say, the length of the article is also a very good argument against using Zend Form. And it still didn't answer my questions, so it really needed to be 2-3 times as long. But if you need to format a form, it is a far more useful resource than the utterly inadequate Zend Form manual.

fclib 0.4.20 release

2010-06-22T22:00:00.000-07:00

I just put up another fclib release. Fclib is an ad hoc collection of php libraries, started about 10 years ago; the i18n-related functions are perhaps of most interest to people (with functions for Japanese, Chinese and Arabic language processnig).

This new release has some minor bug fixes, a new utf8 function (for truncating a string), and a new file, modify_images.inc, that is a high-level interface to use gd functions to make doing thumbnails, cropping, resizing and basic drawing edits. It can operate on one or a batch of images.

(The previous release was 11 months ago, described here)

C++ socket library: asio

2010-06-21T20:53:00.000-07:00

It is when it comes to writing a simple socket client in C++ that I really feel how much PHP has spoilt me.

I previously have used SmartNetwork, part of the SmartWin library ( http://smartwin.sourceforge.net/ ) but it is fatally flawed: when the remote server dies there is no error reporting, so it happily carries on sending data to oblivion.

Therefore when an application I am currently working on needed to write to a socket I decided to try out a new library. I looked at 4 or 5 libraries, and after that first pass rejected all of them. Mainly based on their lack of clear documentation; sometimes based on their license. Faced with a choice of zero, I lowed my criteria, and chose to try boost::asio (also available in a non-boost version). Boost is a major project, already installed on my machines, portable for at least Linux/Windows, and the libraries are peer-reviewed by some very clever people. Boost libraries are also usually flexible (to the point of making them hard to use), and definitely able to give me the error reporting I need.

Here is the one line review: I've been banging my head against ASIO for 2-3 weeks, still do not have working code, but every time I consider running away I decide to stick with it.

Or in other words, it is terrible, but the alternatives are worse.

It is terrible in three ways: no high-level functions, it is hard to use properly and it is basically undocumented. Yeah, yeah, there is API documentation for all the functions, but it doesn't say when to use which function. And, yeah, yeah, there are a dozen or so tutorials. But they are all toy examples, and don't explain why the code has been written the seemingly-complex way it has.

Let me explain a bit more. Asio offers sync operations, but they are no use for any real-world code. For something you intend to use in production you will be forced to use the async functions. That means you'll need to use boost::threads, boost::bind (for the callbacks), boost::smart_pointer (as you have to wait for all callbacks to finish before you can delete an object, and it turns out your async callbacks can get called even after you've closed the socket) and understand how async programs work. All of that is hard.

I keep thinking I've got the code working, then I discover a timing problem that causes a crash only once in 15 runs, or it works on Linux but crashes on Windows, or is fine connecting to localhost but crashes when connecting to a remote server (due to different timing).

Going back to the first of my reasons for describing it as terrible, an example of the lack of high-level assistance is that there is no timeout parameter on any async operations. To do an async_connect that I want to give up after 3 seconds I have to write code for both the connect and the timer, which means two callbacks, and coordinating those two callbacks. Time-outs are part of the low-level BSD sockets but the asio code is doing something that deliberately cripples that, so trying to use them won't work either.

But I don't like to moan without being constructive. So I've been working on two things. First a high-level class called SocketFeeder (and SocketFeederSet) that you can just call write() on and not have to worry about all these callbacks. Second, a tutorial explaining how SocketFeeder has been written, and the thinking behind the design decisions.

The class is for a client's production environment but has been written on my own time, so I'll be able to release it as open source. I'll edit this blog to link to it when it is finally ready; if you are keen to see it and the tutorial then leave a comment or send me a mail. Constructive nagging usually works! (And if you want to sponsor me, or are a magazine that pays for articles, let me know; it will get released eventually, but financial incentive means I can give it priority.)

doctrine: delete object but leave it in database

2010-06-15T03:06:00.000-07:00

I've a website where I make a page from a doctrine object which is taken from a database. Nothing special there. For adding the data to the database I have a form that goes to a preview page then a confirm button that actually writes it. For the preview page I share the same display logic, so I construct a doctrine object and simply don't save it. For editing, I do the same: I load the doctrine object from the database, make the changes specified on the form, and then either show the preview page or save to the database (depending on if preview or confirm was clicked).

The problem came with editing a one-to-many relation. As a concrete example let's say the user is registering multiple email addresses. I was doing this:
   $User->Emails->delete();

Then doing this for each address the user gave:
   $User->Emails[]=new Email(...);

It nicely handles when the user has removed an email address, changed one, or added an extra one. But when I realized the flaw I slapped my forehead so hard I gave myself whiplash. Do you see it? Have a moment...

Yep, if they click cancel (or leave before clicking confirm) they're expecting nothing has changed, whereas in fact all the email addresses they'd previously input have vanished.

The problem is that delete() happens immediately, rather than waiting for me to call save(). I hunted high and low for a way to stop that. My final solution was:
   if(confirmed)$User->Emails->delete();
   else $User->unlink('Emails');

I.e. unlink() appears to be the delete-that-does-not-touch-database I was searching for. (Of course, don't do something silly like go and call save() now, or the user will lose their email addresses and you'll have some orphaned records in your Emails table.)

Incidentally this did not work:
$user->clearRelated('Email');

I'd hoped it would, after reading the description in the manual. But in fact it did nothing at all.

jquery: dcookorg_annotator V0.3 released

2010-06-11T02:37:00.000-07:00

I know, I'm behind in my jquery plugin announcements, so I'm going to do three in one.

First up is annotator, for annotating an image (or anything):
http://dcook.org/software/jquery/annotator/

You can have any number of annotations, can drag them anywhere, and can resize them. In the default mode each annotation has a text box appear underneath it for adding a comment.

New in version 0.3, and shown in the screenshot to the left, are some hooks for attaching a form to each annotation, so you can create a custom form for each one (or any other idea you have)!

MIT-license open-source, and tested in all of IE6, IE7, IE8, Safari, Firefox 3 and Firefox 3.5.

Next up is selector_aspect, which is for selecting part of an image while maintaining a fixed aspect ratio. E.g. useful for cropping an image.
http://dcook.org/software/jquery/selector_aspect/

A simple straightforward plug-in, with not many options.

Third is get_percentage_position, which is used to get the size of one div (or any DOM position) in terms of another div (or any DOM object), and also to get the relative position in the same terms. Not very glamorous, but useful in conjunction with the selector_aspect plugin, for instance. It is available here:
http://dcook.org/software/jquery/get_percentage_position/

Finally, a reminder that my first jquery plugin, to run a magnifier over an image is introduced here:
http://darrendev.blogspot.com/2010/04/jquery-plugin-image-magnifier.html

and available here:
http://dcook.org/software/jquery/magnifier/

and all my jquery plugins are being kept here:
http://dcook.org/software/jquery/

C++: incomplete type and cannot be defined

2010-06-06T17:31:00.000-07:00

A very confusing error just now:
error: aggregate ‘std::ofstream out’ has incomplete type and cannot be defined

from simple code:
std::ofstream out;

And the similar one:
error: variable ‘std::ofstream out’ has initialiser but incomplete type

from this code:
std::ofstream out(fname.c_str(),std::ios_base::app);

Using this didn't help:
#include <iostream>

Using this also didn't help:
#include <ofstream>

That was confusing me most, but now I see I've been getting "error: ofstream: No such file or directory" and I was missing it in the noise of other warnings and errors.

The solution was simple:
#include <fstream>

Yes, a very simple solution, but google wasn't helping. If it had been saying "ofstream is not a member of std" I'd have known I was missing a header file; a strange error message has you looking in other places. (I guess another std header file is doing a forward declaration for ofstream, which is why we get "incomplete type".)

Mumble, grumble, back to work.

Rotating in jquery (firefox problems)

2010-05-31T20:43:00.000-07:00

In my entry on rotating images in jquery/javascript I mentioned I'd not had the problem that the unofficial patch was supposed to fix. Well, now I've seen it even using that patch. The problem is this: if you try to rotate an image (in Firefox, at least) that hasn't fully loaded it all goes wrong.

When rotating in Firefox/Safari you are making a copy of the image; the jquery-rotate patch is to make sure your copy has been properly initialized before doing anything with it. My problem was similar: I was trying to rotate an image that hadn't loaded. (Curiously it didn't happen on a 106Kb image, but did consistently on a 69Kb image; the smaller image was in landscape, while the larger image was portrait, but I have no idea if that is related.)

My rotate call is in a function called init(). I first tried calling init() from the image's onLoad(), which worked most of the time. I also tried a more complex solution that involved not calling init() until both the image's onLoad() and JQuery's $(function(){...}); (what I personally like to call onDomLoaded) had both run. But still it happened on certain images, and I now feel that that level of complexity is not needed (also, onDomLoaded always seems to run before the image's onLoad triggers).

So, my solution is this:

  <img src="test.jpg" id="img"
  onLoad="window.setTimeout('init()',40)">

Using a time-out of 25ms was not enough, but 40ms seems reliable. The downside is that the image flashes on screen briefly in the original orientation before rotate can kick in. We can fix that by having it invisible initially:

  <img src="test.jpg" id="img" style="visibility:hidden"
  onLoad="window.setTimeout('init()',40)">

  <script>
  function init(){
    $('#img').rotate(90);
    $('#img').css('visibility','visible');
    }
  </script>

Rotating in jquery (and IE8 problems)

2010-05-24T18:30:00.000-07:00

Rotating an image in jquery is harder than you might think. Harder than I had imagined it would be, at least. The first problem is that it is not part of either JQuery or JQueryUI, so you are out there in the wilderness where the 3rd party plugins live. And, trust me, it can get wild out there.

I first tried jqueryrotate. It is quite heavy, at around 10KB, but the main reason I abandoned it is that it didn't work properly for me (sorry, I cannot even remember why now).

Next I tried jquery-rotate and I soon got this working in Firefox 3. My code also worked in Safari and Firefox 3.6 first time. IE6/7/8 were the problem. Under the hood all these plugins use DXImageTransform.Microsoft.Matrix for the IE browsers (which works back to IE5 apparently), and Canvas for all other browsers (which works from at least Firefox 3). They even allow rotation of any angle, though I only needed to rotate in steps of 90 degrees.

(By the way jquery-rotate seems to be unmaintained, so I'm using this unofficial patch even though my testing didn't see the problems it fixes.)

Here is my code:

  $('#img').rotate(rotation,true);

Then:

  if(rotation==90 || rotation==270){
    $('#img').width(300);
    $('#img').height(225);
    }
  else{
    $('#img').width(300);
    $('#img').height(400);
    }

(To keep that code sample clearer I've hard-coded the sizes, to assume an image that has a 3:4 aspect ratio in its original position). The point of this code is to scale the image down to fit in the layout. This was where the first cross-browser problem appears. When Firefox/Safari rotate it they are creating a new object of the new dimensions. As far as I can tell when IE rotate it they are creating an optical illusion. It appears to have been rotated but when you read its width/height you get the numbers for the original position; and the same when you try to set them. (As an aside I tried a number of ideas to get around this underlying issue, but all ended in failure, so I guess it is just the way IE works.)

Tinaysh... Tenacish... Tenaciousness is my middle name, even if it is hard to spell. This code does the job:

  if(rotation==90 || rotation==270){
    if(document.all && !window.opera){  //IE-specific
        $('#img').width(225);
        $('#img').height(300);
        }
    else{
        $('#img').width(300);
        $('#img').height(225);
        }
    }
  else{
    $('#img').width(300);
    $('#img').height(400);
    }

OK, images rotate nicely in all browsers, on to the Next Problem.

I attach a draggable and resizable div (see the demos at http://dcook.org/software/jquery/magnifier/ to get an idea) to the image. I don't want it to leave the image so I use this:
mydiv.draggable({containment:img});

When rotation is 0 or 180 everything is fine, but rotations of 90 and 270 go wrong in IE6/IE7/IE8; it seems the different coordinate system confuse it and I can move mydiv outside the image. (Incidentally jquery's resizable containment is broken, so I had to hand-code that; my resizable containment code is not affected by this problem!)

But things get worse. In IE8 *only*, the rotated image overlaps the following page items; IE6/IE7 correctly push the page down when a landscape image gets rotated to become a portrait image. I could have lived with containment not working, but this one is a showstopper.

We can put the image in a named div, then alter the width/height of that div each time we rotate. So the code ends up as this:

  if(rotation==90 || rotation==270){
    if(document.all && !window.opera){  //IE-specific
        $('#img').width(225);
        $('#img').height(300);
        $('#img_outer').width(300);
        $('#img_outer').height(225);
        }
    else{
        $('#img').width(300);
        $('#img').height(225);
        }
    }
  else{
    $('#img').width(300);
    $('#img').height(400);
    if(document.all && !window.opera){  //IE-specific
        $('#img_outer').width(300);
        $('#img_outer').height(400);
        }
    }

You can also fix the other problem by using #img_outer as the containment div, instead of #img. That fixes all problems in IE6/IE7 (though be careful with margins, padding and borders; i.e. make sure #img_outer is exactly the same size as #img).

IE8 is still less than ideal. Using #img_outer stops it overlapping with the following content, but it puts hard white space in the area where the image would be if height/width were reversed (and the draggable div can still be dragged into that area). For a landscape image that has been rotated that means whitespace to the right of the image, messing up multi-column table layouts. For a portrait image that has been rotated that means a lump of whitespace between the bottom of the image and the start of the next page content. Using overlap:hidden did not help.

This is very novel: a bug in IE8 that isn't in any other browser, not even IE6! I've not solved this, and welcome advice. Yes, okay, I could use absolute positioning to put a "Download Firefox" icon in that white area, but that isn't quite the advice I'm looking for...

Linux keyboard shortcuts

2010-04-29T17:31:00.000-07:00

Have you ever gone to hit ctrl-tab (to switch tabs) or ctrl-w (to close current tab) in Firefox and suddenly all your Firefox windows (even the one playing radio on another desktop) disappear? (I'm using Ubuntu and Gnome, but I get the impression this problem affects all Linux distros and all window managers.)

It must be old age making my fingers clumsier but I didn't even know ctrl-q did that until a few weeks ago. Yet I've done it a couple of times by accident recently, and when I did it today I decided Something Has To Be Done.

The solution is not just joyously simple, but I also learnt another cool function while I was there. First the solution: System|Preferences|Keyboard Shortcuts. Assign ctrl-q to do something; then Firefox never gets to see it. I assigned it to the calculator app which is nicely harmless.

I learnt this trick here, which also mentions that the same idea works in XCFE (go to keyboard panel). I'm betting KDE has something similar.

And the cool function? Looking through the other keyboard shortcuts I saw Alt+Print takes a screenshot of just the current window! You have no idea how many times I've clicked Print, then opened up Gimp to crop the screenshot to show just the window of interest. I'm now alternating between feeling very foolish and feeling very empowered.

jquery, click here to crash IE8

2010-04-22T02:06:00.000-07:00

IE6/7/8 had me pulling me hair out again, but I've just found the problem!

Here is the stripped-down code that shows the problem. We have a few divs, one that is active (and can be resized and dragged). The others are inactive but can be clicked to turn them into the active item.

var currentItem=null;

function makeActive(item){
if(currentItem){
    currentitem.css({borderWidth:1,zIndex:998})
        .draggable('disable')
        .resizable('disable')
        .click(function(){makeActive($(this));});
        ;
    }
item.css({borderWidth:3,zIndex:999})
    .draggable('enable')
    .resizable('enable')
    ;
currentItem=item;
}

Clicking back and forth between two items was fine in firefox, but IE8 would lock up (IE6/IE7 were the same). I kept stripping it down until it clicked (pun intended, sorry!). Yes, on each loop the click handler is added again. Each click handler is calling this function recursively and I think that is what locking up IE8.

The solution is simply to change the .click line to look like this:
.one('click',function(){makeActive($(this));});

That doesn't just make IE8 happy, it is also more clearly describes the click handler we want. In fact now I understand the problem I'm surprised firefox was not crashing too.

P.S. resizable('disable') does not work in JQuery 1.7; it is apparently fixed in JQuery 1.8 though.

jquery plugin: image magnifier

2010-04-19T21:36:00.000-07:00

I've just released my first fully-fledged and useful jquery plugin:

http://dcook.org/software/jquery/magnifier/

It allows you to magnify an image and examine just one part of it.
Handles on the edge of the "magnifying glass" allow resizing it, which alters the degree of magnification.
The plugin is fully documented, with numerous usage examples.
It runs on all major browsers and operating systems.
Naturally it is open source (MIT).

find, grep and tr

2010-04-15T02:29:00.000-07:00

A practical example of three useful unix commands, and how they can work together.

I thought I had found another chance to use sed (see Right Sed Fred, I'm too sexy to search and replace), but in the end did not use it; I wanted to remove newline characters but sed works line by line so this is not possible. (There is a way to do this with sed apparently, but it looked awfully hard to understand.) So I used tr instead.

Here is my final command:
find /path/to/myfiles/ -mtime -125 -name '*txt' -print | grep -v xxx | tr '\012' ' '

The first part:
find /path/to/myfiles/ -mtime -125 -name '*txt' -print

means search all the *.txt files under the given path, that have been created or modified in the past 125 days. It outputs one filename per line.

However it include some files I didn't want. Luckily they all had the same filename component ('xxx' in my example above), so were trivial to identify. I used "grep -v" which means exclude anything that matches.

At this stage I had the list of filenames, but one per line. I wanted to use them as the parameters to a batch file, so needed them all on one line. I couldn't get sed to work, but stumbled on a way to use the "tr" tool. It takes two parameters: the character to replace, and what to replace it by. '\012' is octal for the unix linefeed character. So
tr '\012' ' '

means replace LF with a space. I slapped this on to the previous command and got just want I wanted.

(For unix beginners, the | is called the pipe character, and it means take the output of the command before it and give it as the input to the command after it. Many unix commands are designed with this kind of piping behaviour in mind.)

Jquery cheat sheets

2010-04-08T22:28:00.000-07:00

A cheat sheet, printed out, can be invaluable. Here are a few I've looked at:

My choice:
http://www.javascripttoolbox.com/jquery/cheatsheet/

Nice and compact, going into good details. It comes in colour, but the wonderful thing is it also comes in Excel format, so I could edit the colours.
It doesn't show the new in 1.3 functions, and doesn't point out the deprecated ones ($.browser and $.boxmodel); I annotated that myself using the "Too Colourful" one below.

Too new:
http://labs.impulsestudios.ca/jquery-cheat-sheet

This one is nice, fits on one page, already monochrome (though using a light grey for optional parameters, which is hard to read). Does not show the parameters like the my first choice above. It is for jquery 1.4, showing what is new for 1.4; but I also wanted to know what to avoid if I wanted to write a plugin that would work back to 1.2.

Too Colourful:
http://www.artzstudio.com/files/jquery-rules/jquery_1.3_cheatsheet_v1.pdf
2 pages worth, shows what is new in jquery 1.3, and what is deprecated. Uses colour and looked terrible as a b/w print-out. Also, it lists each version of similar functions (such as event handlers), which is why it needs two pages instead of one.

The Dangers Of GroupBy

2010-03-23T18:20:00.000-07:00

I had a complex join (three tables) with a group by and a where cause with three conditions. It seemed to be working, until I added more data; I eventually narrowed problem down to just a single table. Consider this query:


SELECT w.user_id, w.some_date, w.id
FROM mytable w
GROUP BY w.user_id

id is the primary key. That query returns all the data:


user_id   some_date   id
   1      2010-02-01   1
   5      2010-02-02   2
   1      NULL         3
   1      2010-03-22   4
   1      2010-03-24   5

I'm only interested in the record with the latest date for each user, so I add a MAX on some_date and GROUP BY user_id:


SELECT w.user_id, MAX(w.some_date), w.id
FROM mytable w
GROUP BY w.user_id

That gives:


user_id   MAX( w.some_date )   id
   1       2010-03-24           1
   5       2010-02-02           2

Hang on! The dates are right, but 2010-03-24 is from id=5, not id=1. The some_date column has been max-ed in isolation.

This issue is explained in an excellent two-part article: part1 part2

In fact it is so excellent I didn't fully understand it, and am going to re-read it three times a day until I do. I really need to understand it because, as the author shows, it is very easy to get correct answers from a query on your test data, pass all your unit tests, and deliver something that goes wrong when one more record is added.

My query is simpler than the one in his example, but after many tries I cannot find a solution. The "join(...)as d" syntax does not seem to work in MySQL or I am not using it correctly. It seems what I'm trying to do is very basic, so I do not understand why I cannot find more advice on the subject. I'm open to suggestions!

(For the moment I'm going to give up, do a simpler query and have PHP process the results to get the data I actually want.)

Image Protection in Zend Framework

2010-03-07T21:55:00.000-08:00

Sometimes you want to only show certain images (or other media) to certain users (e.g. those that are logged in, or those that have paid for it). This quick tutorial will show how to write the controller for Zend Framework for serving images. This is for ZF 1.10, but as far as I can tell it is not using anything new or unusual.

This tutorial does not cover the validation part of the code.

We start with a controller that does not use views, layout and all that stuff:

class ImgController extends Zend_Controller_Action
{
public function init()
{
$this->_helper->viewRenderer->setNoRender(true);
$this->_helper->layout()->disableLayout();
}
}

Now add an action that will serve all images:

public function imgAction(){
$type=$this->getRequest()->getParam(1);
$fname=$this->getRequest()->getParam(2);
$ext=$this->getRequest()->getParam(3);

echo "type=$type, fname=$fname, ext=$ext";
}

public function badAction(){
$this->getResponse()->setHeader('Content-Type','image/jpeg');
}

The idea is that URLs such as http://127.0.0.1/img/folder/abc.jpg will end up at the imgAction() function, and $type will get set to "folder", $fname to "abc" and $ext to "jpg". To do that we need to set up what is called a router, which we will do next. The badAction() will handle any problems; this crude code will send back a 0 byte jpeg.

To create a router, jump to your bootstrap file and create a function something like this:

public function _initCustomRouting(){
$frontController=Zend_Controller_Front::getInstance();
$router=$frontController->getRouter();
$router->addRoute(  //To catch any that are not formatted correctly
'imgHandlingBad',
new Zend_Controller_Router_Route('img/*',
array('controller'=>'img', 'action'=>'bad')
)
);
$router->addRoute(  //Formatted as /url/type/fname.ext
'imgHandling',
new Zend_Controller_Router_Route_Regex('img/(.+)/(.+)\.(.+)',
array('controller'=>'img', 'action'=>'img')
)
);
}

Note that order: imgHandlingBad must come before imgHandling. I found this introduction (7 minute screencast) to using Zend_Controller_Router_Route very useful; then some trial and error and study of the ZF source code.

Now you should be able to test http://127.0.0.1/img/folder/abc.jpg and see our debug comment. But without a folder (e.g. http://127.0.0.1/img/abc.jpg) it will get handled by the badAction(), as will those without an extension: http://127.0.0.1/img/folder/abcjpg

Now the final step is to feed back some images, so in imgAction() replace the echo line with this code:

$path=$basePath.$type;
$fullPath=$path.$fname.'.'.$ext;
if(!file_exists($fullPath))return $this->badAction();

switch($ext){
case 'jpg':$mime='image/jpeg';break;
case 'png':$mime='image/png';break;
case 'gif':$mime='image/gif';break;
default:return $this->badAction();
}
$this->getResponse()->setHeader('Content-Type',$mime);

readfile($fullPath);

It decides the filename, decides the mime-type based on extension, and then readfile passes on the binary data.

In the above code we use the $type directly to decide the image path. You could instead map type to different directories. E.g.

switch($type){
case 'www':$path='/var/www/main_images/';break;
case 'www/articles':$path='/var/www/html/special/articles/';break;
case 'family':$path='/home/user/images/family/';break;
default:return $this->badAction();
}

Another way you might use $type is if certain users can only see certain types of images. You could do different validation checks in each case statement above.

Got any suggestions to make this code better? Let me know!

P.S. One special note about this technique. In the typical MVC Zend Framework setup, if you create public/img/ then apache will serve images from there; only if the image is missing in that location will Apache ask the Zend Framework to serve it. You might use this to your advantage to speed up delivery of certain common images. But it also opens the way to accidentally serving images that are supposed to be protected.

Creating A Doctrine Custom Behaviour, Part 2

2010-03-02T18:21:00.000-08:00

In part 1 we made a simple custom behaviour and then gave it some options we could customize in our schema. The second half of the power of Doctrine's behaviours though is being able to set values. First, uncomment this line in the last example in part 1:


    $this->addListener(new DarrenTestableListener($this->_options));

You should get told class DarrenTestableListener does not exist, so create DarrenTestableListener.php in the same directory as DarrenTestable.php (though, as mentioned before, it can go anywhere in your models directory tree), and fill it with this code:


class DarrenTestableListener extends Doctrine_Record_Listener
{
    protected $_options = array();

    public function __construct(array $options)
    {
        $this->_options = $options;
    }

    /** Called when a new record is created */
    public function preInsert(Doctrine_Event $event)
    {
        $name = $event->getInvoker()->getTable()->getFieldName($this->_options['name']);
        $modified = $event->getInvoker()->getModified();
        if ( ! isset($modified[$name])) {
            $event->getInvoker()->$name = "C";
        }
    }

    /** Called when an existing record is updated  */
    public function preUpdate(Doctrine_Event $event)
    {
        $name = $event->getInvoker()->getTable()->getFieldName($this->_options['name']);
        $modified = $event->getInvoker()->getModified();
        if ( ! isset($modified[$name])) {
            $event->getInvoker()->$name .= 'U';
        }
    }

    /** Handle dql update queries */
    public function preDqlUpdate(Doctrine_Event $event)
    {
        $params = $event->getParams();
        $name = $event->getInvoker()->getTable()->getFieldName($this->_options['name']);
        $field = $params['alias'] . '.' . $name;
        $query = $event->getQuery();
        if ( ! $query->contains($field)) {
            $query->set($field, '?', 'U');
        }
    }

}

It has default options (and a constructor to set the options), then one function to handle INSERTs and two functions to handle UPDATEs. (There are also hooks for DELETE, save and validation.) All three functions follow a similar pattern:

Get the real fieldname
See if we need to do anything
If so, set the field

What we do is set the field to "C" when it is created. Then each time it is updated we append a "U". E.g. after three updates it will look like "CUUU". (TODO: I'm not so familiar with DQL, and I am not using it, so the DQL version replaces the existing data instead of appending. If you can supply the proper code for it let me know!)

Incidentally if you set a default in the options that will get precedence over your code (that is what the getModified() call is for, I assume). E.g. if I change my YAML schema to:


ActsAs:
    DarrenTestable:
        name: being_silly
        options:
            default: Hello!

Then after a couple of updates being_silly will contains "Hello!UU".

Creating A Doctrine Custom Behaviour, Part 1

2010-03-02T17:41:00.001-08:00

Doctrine is a wonderful ORM library, but the manual is..., well, let's just say I think it was written by the developers, and out of obligation :-)

Behaviours are a motivating feature for using Doctrine, so here is a two-part tutorial on how to add your own minimal behaviour. The first part will show how to add a field to a table. The second part will show how to have its contents set automatically. For a more in-depth example see here or study the source code of the core behaviours.

The first thing you need to know is to put your behaviour classes in your models directory, or a sub-directory of your choice. You don't need a directory structure that matches the "XXX_YYY_" prefixes of the class name. This is assuming you are using Doctrine's auto-loading of table classes; if not you can put them anywhere you like of course. I'm putting them in models/behaviors/ (yes, using American spelling!)

Make a file called "DarrenTestable.php" with these contents:


class DarrenTestable extends Doctrine_Template{
    public function setTableDefinition(){
        $this->hasColumn('darren_testing','string');
    }
}

See, I told you it was minimal! To add it to an existing class, add this to your YAML schema:


ActsAs:
    DarrenTestable

Recreate the class from the schema and you should see a field called "darren_testing". On MySQL the type is "text" (i.e. this is what "string" maps to if you are using MySQL).

Even that very small example is doing something useful: if this behaviour is used in 10 classes and we need to change the name or data type (or set a default, add an index, etc.) we only need to edit it one place.

While still staying within the scope of a minimal example, we can expand it in two directions. We can have our code automatically set the field value (see part 2 of this article), and we can allow users to customize it. Here is how we add some options, with defaults:


class DarrenTestable extends Doctrine_Template
{
    protected $_options = array(
        'name'=>'darren_testing',
        'type'=>'string',
        'options'=>array(),
        );

    public function setTableDefinition()
    {
        $this->hasColumn($this->_options['name'],
            $this->_options['type'], null, $this->_options['options']);
        //$this->addListener(new DarrenTestableListener($this->_options));
    }
}

I have added the call to the listener we'll use in part 2, but commented it out for the moment. The options contains a sub-array called options which is passed on to Doctrine_Table::setColumn() and can be used to set defaults and so on.

Usage is exactly like above if we want the defaults. If we want to change the name, and require it to be set (i.e. NOT NULL) you would do:


ActsAs:
    DarrenTestable:
        name: being_silly
        options:
            notnull: true

When a unix dot means a whole different world...

2010-02-27T15:49:00.000-08:00

Subtitle: new ways to shoot yourself in the foot.

A week ago I was changing ownership on a web site subdirectory:
chown -R darren:www *

There were some files/directories that start with a dot, such as .htaccess, so I then did:
chown -R darren:www .*

If you just screamed out "Nooooo!!!!" you are allowed to polish your Unix Guru badge and wear it with pride. Personally I just sat there wondering why it was taking so long to return and why my disk was suddenly sounding so busy.

That's right. ".." refers to the parent directory. It had gone up a directory, then was using -R to recurse into every subdirectory on my web site, destroying previous ownership settings with abandon.

I realized after a few moments and killed it, but the damage was done, and I'm still discovering small parts of web sites that got broken. But a question for the gurus: how should I change all files and directories in a tree including those that start with a dot? Did I need to use some complicated find and chown combination command?

Google Messing Up I18n

2010-02-16T17:36:00.001-08:00

People often seem to think Google can do no wrong; an image helped no doubt by being surrounded by companies such as Microsoft and Apple. But, even with all those computer science PhD's it snapped up, it sometimes just does not get it.

Once a month or so, Google resets back to showing me the search results in Japanese, and only searching Japanese pages. My browser is set to say I prefer English pages, but it ignores that and uses my IP address. Japanese people use a Japanese browser that will send a header saying they want to see Japanese. Unless they actually want to see English. Why is Google ignoring this?

Perhaps Google needs fewer PhD's and more people who know what this header means:
Accept-Language en-gb,en;q=0.5

Google (!) tells me Google need to look at section 14.4 of RFC2616.

But, the user-unfriendliness goes deeper. Here is how I get out of Japanese mode.
Click: 検索オプション
Find: 言語検索対象にする言語
and scroll down to find and select 英語

No! No, no, no! It just goes to show, even though I read Japanese I still messed up. Here is the correct way:

In the top-right of the search results click "設定 ▼" then 検索設定
In the line that says 表示言語の設定 scroll down to select 英語.
Then find the button in top-right that says 保存。When it pops up something that looks like an error message (but is actually a success message), press OK.

Hopefully I'm good for another month.

But the point of my rant is the only piece of English on any of those pages was "google". Not a single helpful line such as "switch this page to English". Perhaps they should use the Accept-Language header to decide what language the user would most like to see a "switch this page to XXX" link in?

Well, rant over. Google have been doing this for 10 years, so I don't expect they're going to change any time soon.

Facebook Making Me Redundant

2010-02-04T20:50:00.000-08:00

Back in 2007 I wrote an article on how to port your PHP applications into C++ (in the Oct 2007 issue of PHP Architect), showing the speed-up and memory-usage benefits. It's a nice idea, though I've only done it in a production project once. The programmer man-hours is very rarely worth it.

Well, Facebook have gone and automated the process, calling it Hip-Hop. Over 90% of Facebook's (400 billion PHP-based page views every month) traffic is now running on their Hip-Hop system, i.e. on PHP pages that have been compiled into C++.

This is very interesting to me as I have recently been writing less and less in C++, more and more in PHP. Even a tree searcher for computer go I have been doing in PHP, as an experiment to see how it goes. On reflection that particular experiment is probably a failure - I need to set bits in a 32-bit or 64-bit integer and PHP puts a layer of abstraction in the way.

The main php script is very slow (its been running non-stop for 5-6 weeks so far, with months left to run) but ironically all that time is being spent in calling other applications (written in C++!) to do the hard calculations. However I've another script that analyzes the data, needing to hold it all in memory at once, and that is showing signs of cracking - I'm already having to set my memory_limit to 1.5G, and that will probably hit 4G when the other script has chomped through all my data.

Hip Hop might be my saviour.

P.S. PHP Architect magazine also think this is going to be a very important technology.

Zend Framework

2010-01-16T19:29:00.000-08:00

I've been hearing a lot about Zend Framework over the past year or two, it seemed to have become the de facto standard framework (something else I've been hearing a lot about) and I felt a need to get up to speed on it.

I put up my review of my review of Easy PHP Websites with the Zend Framework. I bought the book a few months back, as it seemed the best of the choices at the time. Quick summary of the review: Okay book, not impressed by Zend Framework, last third of the book is useful and interesting.

I'll talk about not being impressed by the DB part of Zend below, but I feel the main other aspect that bothered me was the way data is given to and then used in views. In the controller I have to write:
$this->view->something=$mymodel->get_something();

Then in the view it is used as:
Our something is <?php echo $this->something;?> today!

All that $this-ing and view-ing makes the code verbose, hiding the real thing I'm trying to say with my code. In particular the view code is ugly - ZF has not helped at all compared to writing plain PHP.

Why do I care? Try getting a designer to edit a view that looks like that. They may be comfortable with HTML but will go pale and faint at the sight of a dollar sign let alone all the surrounding junk. I can tell you from experience they will refuse to even edit a file that contains text that looks like that, let alone type it themselves.

I was also not impressed with how Zend Form does input validation. It seems to be just as much work as doing it in plain PHP. If I add a dependency I want value for money!

Back to databases. I recently wrote about being impressed with Doctrine ORM. Zend DB didn't impress me at all; I already use Pear::DB to abstract away the database, and all Zend adds is having to use PHP instead of SQL to write common queries, while still having to use SQL to write the less common ones.

A web search found I wasn't alone in this opinion, but what was interesting was I found people using Doctrine within Zend Framework, as a replacement for Zend DB.

To quote an article by Ruben Vermeersch: "While Zend_Db isn't a bad technology, it is still quite low-level and close to the underlying database. Using Doctrine, you can manipulate your data like objects, without worrying about the database too much." The article is pro-Zend and pro-Doctrine, and shows how to use the two together. Good clear article.

Here is another example of using Zend and Doctrine together. It is just example code, not a teaching article, but interesting as it also uses ajax (using jquery).

Here is an article comparing Symfony (using Doctrine) with Zend, with a pro-Symfony slant. Also check out the comments, especially the reply by Matthew Weier O'Phinney describing what is coming up in Zend (including how it may have more formal integration with Doctrine at some point, and hopefully auto-admin pages and auto-forms-from-schema).

By the way, the afore-mentioned Ruben Vermeersch article suggests these articles to get up to speed on Zend:
http://framework.zend.com/docs/quickstart
http://akrabat.com/zend-framework-tutorial/

I'd like to stress again that my views of Zend Framework are currently only based on studying it; I'll get back to you once I've tried it out on a full-scale real-world project.

ASP, Error-Handling and Microsoft's Latest Catchphrase

2010-01-14T19:41:00.000-08:00

Microsoft's Latest Catchphrase: It may be crap, but at least it is ready installed crap!

What inspired today's rant? Using ASP, of course. The question I had was to how to handle SQL errors. (This article is more than just rant; I will also tell you how to do a couple of essential tasks that ASP makes hard.)

I've been using ASP for a few months, for one particular sub-project. We wanted to minimize the installed components, and as one sub-system already needed IIS, and the required scripting was fairly simple, we decided to go with that instead of installing WAMP (Windows Apache/MySQL/PHP).

I'm using ADODB COM object for the database connectivity to Microsoft SQL Server. I've used the same COM object from PHP before, and errors get thrown as exceptions. I wrap them in try/catch blocks and everything is good. A bit of searching discovered how it works in ASP/VBScript. First exceptions are thrown just the same. Second, there is no try/catch functionality.

Read those two key facts again, and have a good long think about them.

Some more searching discovered things are not quite that bad, and you can handle errors. The first thing you must do is precede your function with "on error resume next". That tells it to ignore errors. Then immediately after any action that might have had an error you do:

if err.Number <> 0 then
'Handle errors here
end if

See here and here.

By the way, to turn off "resume next" mode, you give the almost mystical "On Error Goto 0". And the other thing you need to know is that resume next mode only affects the current function; if you call another function and that function doesn't explicitly "on error resume next" then any error there will terminate the script. Which I suppose is better than the confusion resulting if the flag was global.

As my next piece of evidence let me show you how to do the equivalent of PHP's "s=file_get_contents(fname);" in ASP (due to the afore-mentioned difficulties in error-reporting it returns a blank string if the file does not exist):

Function file_get_contents(fname)
fname=Server.MapPath(fname)
mode=1 'Read-only
set FSO = server.createobject("Scripting.FileSystemObject")
if(FSO.FileExists(fname))then
set fp= FSO.OpenTextFile( fname, mode)
file_get_contents=fp.ReadAll
fp.Close
set fp= nothing
else
file_get_contents=""
end if
set FSO = nothing
End Function

Oh, what fun! But I try to be fair, and every language has its good and bad points, so I tried to think up all the good features of ASP. After all, I have actually been writing useful, working code in ASP. The bad points flashed into my mind at every turn, but in the end I did manage to find one good point...

It is already installed, if you are already using IIS.

Which brings us back to that catchphrase that Microsoft is hurriedly trade-marking in 120 countries worldwide: It may be crap, but at least it is ready installed crap!

(By the way, speaking of ready-installed crap, did you know IE6 still has 15-20% market share? And IE6+IE7+IE8 together have 67% share? It was news to me - I thought the browser wars were over and everyone used Firefox, or at least Safari!)

Portable PDF reader

2010-01-05T23:39:00.000-08:00

More and more books, especially tech books, are only available as PDF. Another advantage is I can get the PDF delivered immediately rather than the "2-4 weeks" amazon.jp likes to tell me. But I don't like sitting at my computer to read - I want to be able to sit in an armchair, on the train, in a cafe, etc. to read (see also my moan about php Architect magazine going PDF only)

I've done some searching, and some asking, and I suspect the device I want does not yet exist, but I'm open to suggestions. I don't want the heavy bulk of a notebook. The Kindle (or similar) seem hard-wired for content being sold by Amazon (or whoever), which is no good at all (Update: PDF supported since version 2.3, and content can be uploaded over USB cable; see wikipedia article)

I'm very interested in how easy it is to read a PDF formatted for A4 on an iPhone (or iPod Touch), as that would have lots of other advantages (e.g. portable video player, as well as portable email and web browsing in the case of the iPhone).

UPDATE: list of e-book readers
Apparently the Consumer Electronics Show in Las Vegas this month will see some new e-book readers announced. So, sitting on my hands seems best at the moment.

UPDATE:
Video demo of AjiReader viewing a PDF on the iphone (apparently the app costs $1)

Another video demo showing PDF, DOC, XLS, PPT but the comments seem to be saying you need a jail-broken phone for this?

UPDATE: (8 months later!)
The new Kindle looks good. I decided to stop Waiting For The Next Big Thing, and went ahead and pre-ordered one. It does PDFs, it does Japanese, everyone I've spoken to says e-ink is great; but, above all, at $139 it is now affordable.

I also ordered the case with built-in light; it was 1/3rd of the total cost, so I dithered a lot about that. But while I think the case is over-priced it also looks very cool.

Some people said the 6" screen is still too small to read A4 PDFs in comfort. If I find it unusable for that (my key need) I'm hoping I might be able to resell it at only a small loss. After all, ordering from Japan is a pain.

I'll put up a review in about a month. Watch This Space :-)

Doctrine ORM

2010-01-04T00:22:00.000-08:00

I mentioned before, when talking about my Step-By-Step Regex article in php|a magazine, that the same issue (August 2009) had some other interesting articles.

One is about Adminer, an alternative to phpMyAdmin which is designed to be very compact and exists in just one file (so it is easy to upload temporarily to a production server for instance); the article is mainly about how Adminer was written, which was interesting.

But the article - and library - I most want to talk about is Doctrine ORM. The main problem this library is solving is this: when you add a field to your database you have to make the change in at least two places, your PHP script and your database. That sucks. The article describes itself as a tour around Doctrine's more advanced features, but I found I didn't need any other introduction, and it sold the features in a very clear and logical way.

The first thing I liked was you can describe your database schema in any of three ways: SQL, PHP, or the compact YAML files, and utilities are provided to convert between all three. By SQL I mean an existing database, that you could have been creating interactively from phpMyAdmin, or it could be a legacy database that has been around for years and now needs to be connected to a php script.

What only the Yaml and PHP schema formats can specify are what Doctrine calls behaviours. By adding the Timestampable behaviour it will add created_at and updated_at fields to the database and keep them up to date for you. The Sluggable behaviour is for making URLs and is demonstrated in the php|a article, but others that caught my eye are SoftDelete (records just get the deleted_at field set when you delete, rather than physically being removed), and Versionable (each time a record is changed the previous record is saved, and can be reverted to). NestedSet helps build trees, I18n is for holding translations (see below), Searchable does indexing, Geographical can find nearby places.

Incidentally I covered I18n in a previous article and Doctrine is using type 1, but without the language table to store collation. There is no consideration of different collations, and no concept of default language. It is sufficient for most purposes though.

Looking at some behaviours not in the Doctrine core distribution, Taggable looks interesting, allowing you to add blog-style tags. Locatable apparently ties in with Google Maps to get latitude and longitude automatically given an address (it could then be used, for instance, with the Geographical behaviour to automatically find all people within 10km of you based on just their address). EventLoggable logs all actions (select, update, delete, etc.) to a disk file. Blameable records (in the table) who last made any change to a record.

Doctrine's behaviours feature is powerful, portable, the existing ones are customizable (all the above seem to have sensible defaults but also can have their behaviour fine-tuned) and new ones look easy to write.

The third big feature of Doctrine that I like is the way data is retrieved on demand, and how that ties in with table relations (see listing 6 in the php|a article). And if this becomes inefficient you can optimize without having to change much code (listing 7). I'm not a big fan of using PHP to write SQL queries. Generally to do anything interesting you end up dropping down to writing SQL anyway. But I was impressed with the example in the article, and you only need to start explicitly describing joins if you have profiled and found you need to optimize.

(By the way, it goes without saying that Doctrine ORM is database-neutral, not tying you into MySQL or any other database.)

The final cool Doctrine feature is migrations. When you make a change to the schema in either your YAML file or your PHP file you run a script and it will make a migration file. This can then be run to move a database between versions (backwards or forwards). I admit I currently do this with some hand-crafted SQL scripts, which are one way only, and painful. And sometimes I'm not even that organized.

I've not used Doctrine ORM for a real project, but I expect I will try it out very soon.

Right Sed Fred, I'm too sexy to search and replace

2009-12-19T17:18:00.000-08:00

Last week I hand-edited 22 XML files to change one attribute in each. I had only ten minutes, and I knew that solution could be done in that time.

Today I had exactly the same task, but with less time pressure, I went hunting. Here is the solution I used:
sed -i 's/old/new/g' *.xml

-i means replace inline. I had struggled with sed years ago and thought it was a horrible monster that made emacs look user-friendly in comparison. But that is so simple. Perhaps I was hurt before by trying to do something that couldn't be described as a simple regex?

Actually I vaguely remember my need at that time was to modify all html files in a directory tree, which sed cannot do. But with find it can:
find . -iname \*.html -execdir sed -i 's/<html>/<html mytag="test">/g' {} +

That inserts an attribute in the html tag of all *.html files in current directory and its subdirectories. (Shamelessly stolen from comments on this page here, and then I did a quick test to confirm I hadn't introduce a typo.)

Cool. One step closer to unix guruness. (sed is also available in cygwin, which is where I was actually doing the edit that started this article.)

UPDATE: for an example where tr is more useful than sed see
find, grep and tr.

UPDATE: Here is an expansion of the find+sed example above. I wanted to alter three types of extensions: html, php, phtml. And I only wanted to alter those files in just certain subdirectories. The -regex parameter of find seemed to do the job:

find . -regex './$dir1\|dir2/subdir1\|dir3\|dir4$/.+\.$html\|php\|phtml$' -execdir sed -i 's/ABC/XYZ/g' {} +

That is the command exactly as you run it at the bash command prompt. Notice that, in the regex, not just (), but also | need to be prefixed with a single backslash.

php Architect: no more print version

2009-12-01T02:49:00.000-08:00

The subject says it all: the publishers of the excellent php|a magazine have suddenly stopped the print edition. It has gone PDF only.
(Click php|a on the right to see my other blog articles about php|a magazine.)

This is such a shame: I would willingly pay much more for a print version than a PDF version. In fact I was doing exactly that. Then a year ago they turned everyone to be Print+PDF subscribers, cut the price dramatically, and gave me free extra 12 issues to make up for the price cut. In fact that only works out as 6 or so free issues, but as they were free I can hardly complain.

It is just a shame they fiddled with their business model, as obviously the price cut made the print magazine too expensive.

Why do I prefer the print version? It can be read on the train, read in the mountains, is light and portable. It can be kept on my bookshelf, pulled off and opened on my desk while I'm coding. As an author, a print magazine is also much more impressive. It is something physical I can show a potential client, something I can show my Mum.

Escaping CSV in C++

2009-11-23T17:56:00.000-08:00

There are two escaping rules for each field in a comma-separated value row:
1. Change each double quote to two double quotes.
2. Surround with double quotes if the field contains a comma or double quote.

These are the rules used by Excel and all other software that deals with CSV data.

As an example, if my fields are:
hello world
a,b,c
"CSV" is popular
""

Then it becomes:
hello world,"a,b,c,","""CSV"" is popular",""""""

C++ has a justified reputation as a hard language for text manipulation. Boost has libraries to make it a little easier, but I didn't want to add Boost as a dependency for a project I was working on. Fortunately std::string's replace() function turned out to be more powerful than I had realized:


void output_csv(std::ostream &out,std::string s){
if(s.find('"')!=std::string::npos){  //Escape double-quotes
    std::string::size_type pos=0;
    while(1){
        pos=s.find('"',pos);
        if(pos==std::string::npos)break;
        s.replace(pos,1,"\"\"");
        pos+=2; //Need to skip over those two quotes, to avoid an infinite loop!
        }
    out<<'"'<<s<<'"';
    }
else if(s.find(',')!=std::string::npos){ //Need to surround with "..."
    out<<'"'<<s<<'"';
    }
else out<<s;   //No escaping needed

If you like compact code then the while loop can be rewritten:


void output_csv(std::ostream &out,std::string s){
if(s.find('"')!=std::string::npos){  //Escape double-quotes
    for(std::string::size_type n=0;(n=s.find('"',n))!=std::string::npos;n+=2)s.replace(n,1,"\"\"");
    out<<'"'<<s<<'"';
    }
else if(s.find(',')!=std::string::npos)out<<'"'<<s<<'"';
else out<<s;
}

P.S. If you need to do the same in PHP, PHP 5.1 finally introduced fputcsv for it. The comments on that page show how to do it in older versions of PHP; my fclib library also contains functions for it.

samba and strange permissions

2009-11-16T18:54:00.001-08:00

On my linux server I run a samba share, which is used by both Linux and Windows clients on the LAN. I moved it from an old FedoraCore machine to Ubuntu a few months ago, and ever since have been getting strange permissions: text files kept becoming executable (but only for user, not group or other).

It took me this long to realize what was going on, and when I tried to track it down about a week ago I concluded it was SciTE being strange on just samba partitions. I.e. I'd edit a file with rw-r--r-- permissions, save it, and it would end up with "rwxr--r--". Every time. But not on normal partitions, and gedit didn't do it on the samba partition. I noticed today that files created by a PHP script also got those weird permissions, and the penny dropped: gedit must be explicitly setting permissions when it saves a file. Scite wasn't the cause of the problem at all.

So, I went hunting again. I referred to Mount samba shares with utf8 encoding using cifs a lot, but in fact it didn't give me the answer I wanted: the instructions there gave me the same problem. (It did show me how to set my samba partition to mount from /etc/fstab, replacing my crude entry in rc.local though.)

Hunting through the troubleshooting section I found the "nounix" flag and tried that. Initially it made things worse, giving all files rwxrwxrwx permissions. Then I changed from "file_mode=0777,dir_mode=0777" to "file_mode=0644,dir_mode=0755" (which was what I had originally), and that combined with nounix works! All text files are rw-r--r-- before and after saving. Oh, the other change I added relative to the above page was including "uid=darren,gid=darren". Otherwise files were owned by "nobody:root" and I didn't have permission to edit anything (even with the suggested 0777 settings).

My guess is that my old FedoraCore samba server didn't have the unix extensions, Ubuntu 8 does, and somehow those unix extensions are misconfigured in Ubuntu. My samba server configuration is all defaults however... Anyway, it now works the way it has for the previous few years, so I'm happy.

UPDATE: I just realized this has also fixed another irritation - delete (that moves to the Trash folder) hasn't been working on that partition, but now it does. Trash folder vs. direct delete was only a minor factor; what was really annoying was every time I pressed delete it then popped up a dialog box requiring me to confirm it.

How much is that password worth?

2009-11-04T21:20:00.000-08:00

These people have published details about how they used Amazon EC2 to crack passwords:
http://news.electricalchemy.net/2009/10/cracking-passwords-in-cloud.html

Personally, I skipped all the details and went straight to the interesting conclusions page:
http://news.electricalchemy.net/2009/10/password-cracking-in-cloud-part-5.html

It tells you how much it will cost someone (in EC2 charges) to crack your passwords, based on their lengths and the number of characters you use.

I used to think 8 characters was a good password. Seems it is worth about $3, or $45 if I've mixed in some numbers. Gulp. And all this is assuming there are no dictionary words in there. Double gulp.

Regex Article in php|a

2009-10-25T17:38:00.000-07:00

The August edition of PHP Architect magazine contained my introductory article on regexes. It has a PHP slant but mainly it is about regexes. Well, most of it is introductory but it contains a bonus section at the end using advanced regexes in SQL to repair a database in situ.

The print magazine arrived by air mail a couple of weeks ago and I finally got to read it. I was pleased (and relieved) that it read well. If you have read it I'd love to get some constructive criticism - especially if you are uncomfortable with regexes.

It was also my first print article to use colour, and I think that worked well too.

One correction in the complicated "Repairing With Regexes" section at the end of the article. The text says "So \1 has to be written \\\\1 (that is four backslashes)." That seemed a strange thing to write, so I took a look at the unit test source code (included with the PDF download version of php|a magazine) I had written. Four backslashes is correct when in PHP, but when in SQL it should be "\\1" not "\1".
(I just checked my emails from when we were proof-reading, and one of the edits had removed all those backslashes; I caught that at the time but ironically I got it wrong when putting them back in.)

The other minor correction I only realized after Arne Blanket's column in the same magazine issue! I had written: "...an IP address after that at sign, such as darren@10.0.0.1, which, while unusual, is technically valid". In fact "darren@[10.0.0.1]" is the technically valid form. Which is doubly annoying because my article's regex would reject those square brackets and I hadn't explicitly pointed that out. (No harm done, though, as this email form is highly discouraged.)

Arne's column also points out that top-level domains are no longer just two or three characters. So my '\.[a-zA-Z]{2,3}$' suggestion should really have been '\.[a-zA-Z]{2,}$'. Luckily, that suggestion was just in a list of other ideas, not part of the article's main regex.

(By the way I enjoyed, and will blog about real soon, some other articles in this particular issue; if you are not a subscriber, and your work involves data and PHP, then this is the back issue you should get!)

Control firefox from PHP?

2009-10-14T18:51:00.000-07:00

Internet Explorer can be controlled from a COM object interface, and therefore from PHP (i.e. any scripting language that has COM support).

But is there a way to get script control of firefox? Ideally I'm looking for a platform-independent solution, and something I can use from PHP. Google is not helping (PHP's dominance as a server-side technology overwhelms the client-side related hits).

Here is my dream PHP script:

$firefox=new FirefoxInstance();
$firefox->set_url("http://dcook.org/work/");
$firefox->wait_until_fully_loaded();
$links=$firefox->get_links();
$found=false;
foreach($links as $id=>$info){
if($info['text']=="MLSN"){$firefox->click_link($id);$found=true;break;}
}
if(!$found)echo "MLSN link is missing...\n";

Or:

$form=$firefox->get_form("login");
$form->set("username","guest");
$form->set("password","guest");
$form->submit();

Et cetera. I.e. I'm talking about operating firefox the same way a user does; I know I can grab the raw HTML, parse it, etc. all from PHP, but that doesn't test a web site the same way clicking links in a browser does. Especially web pages with javascript, iframes, AJAX, etc.

(I'd heard of XPCOM but, if I've understood it correctly, it is a library to build firefox and its extensions, not something to control firefox? It also has no PHP bindings.)

BTW, going back to controlling IE from the COM interface, I don't suppose anyone has seen a detailed tutorial on how to use it to fill out and submit forms? I only ever see simple examples of how to set the URL, but I believe full control should be possible.
Dec 16th 2009 UPDATE: This came to the top of my to-do list so I read the MSDN docs on the Internet Explorer COM object, and now it is my understanding that I cannot manipulate and submit forms via the COM object. None of the example usage even hinted at doing this.

PHP, reliability, ASP and remote desktops

2009-09-24T17:24:00.000-07:00

I've been working in ASP on an IIS server recently. Not my environment of choice, but the scripts I need to write are relatively simple, so I decided it was easier to go with it than ask for PHP to be installed (which then requires apache). This is quite a helpful page:
http://www.design215.com/toolbox/asp.php

We'll see how it goes. In many of the systems I work on, windows or linux, I usually use commandline PHP as the glue. I'm always amazed at how rock solid it is. Especially on Windows it is always the most stable element. E.g. I've a 24/7 PHP script that deals with COM objects. The COM objects crash regularly. But I just catch the exception in PHP, delete the COM object, create another one and off we go. Needing to use 3rd party libraries only available in C are the only reason I write C++ recently, but they cause me no end of trouble (mysterious crashes deep in windows system calls for instance). Windows itself needs to be rebooted every 3 or 4 weeks or it starts to go all wobbly on us, and my 24/7 PHP scripts simply run for those 4 weeks without a single problem or restart.

Of course saying PHP is more stable than C would be stupid; PHP is itself a C app. The difference is that PHP is a well-crafted piece of code, better than the COM objects, 3rd party C libraries and Microsoft OSes I'm comparing it to.

Going back to IIS, I don't have it installed locally, so have been developing on a remote server. "rdesktop" wins this month's "application I wish I'd discovered months ago" award. Previously I would have to boot up my windows notebook just to use remote desktop. rdesktop gives me this functionality on linux, works perfectly (touch wood) and copy and paste works too.

Windows SEH in C++

2009-09-03T02:14:00.000-07:00

A windows app was mysteriously dying, roughly once a day; no clues in any of the various log files.
So I enabled Windows SEH:
_set_se_translator(win32_exception_handler);

(See MSDN article about _set_se_translator)

And here is my implementation of that function:

void win32_exception_handler(unsigned int code, EXCEPTION_POINTERS *ep){
throw new win32_exception(code,ep);
}

And the class (cut down to basics) is:

class win32_exception{
std::string info;
public:
win32_exception(unsigned int code,EXCEPTION_POINTERS *ep){
//Describe it in info
}
const char *what()const{return info.c_str();}
};

Then, I have this code (which was already here before I enabled SEH):

try{
e=main_loop(stuff);
catch(std::exception &e){
ErrorLog()<<"Got a std::exception thrown by main_loop:"<<e.what()<<" (will treat as fatal)\n";
return -9;
}
catch(win32_exception &e){
ErrorLog()<<"Caught win32_exception thrown by main_loop:"<<e.what()<<" (will treat as fatal)\n";
return -9;
}
catch(...){
ErrorLog()<<"Got an unexpected exception thrown by main_loop (will treat as fatal)\n";
return -9;
}

Now, what I don't get is today the program died with this message:
Got an unexpected exception thrown by main_loop (will treat as fatal)

I'm staring and staring at the code and don't get why the win32_exception code didn't catch and instead some other exception type got caught. Enabling SEH was the only change, all such exceptions should go through my function, and that only throws win32_exception instances.

The other mystery to me is what is the type is of this "unexpected" exception.

(I'm hoping to come back and add to this blog entry when I work out what is going on!)

php6 - maybe not so painful?

2009-08-23T04:27:00.000-07:00

I long ago learned that if you want to get development work done then always use the distro's own version of PHP, Apache, etc, rather than compiling your own version. Otherwise every upgrade due to a security or bug fix requires you to first realize the need (costs time keeping up with the study) and then do something about it (costs time). Time is money; not spending the time is insecure. So I outsource all of that to the experts at my distro or ISP.

Consequences are that my code avoids using the latest feature, and tries to work on as wide a set of versions as possible. PHP 5.0.0 was released in July 2004 yet at the start of this year two of my clients were still using php 4. One of them was still using php 4.2. (They've both upgraded this year; one because they moved to a new ISP, the other because we were troubleshooting code and running out of ideas, and wondered if moving to php 5 might magically fix it.)

Which is a long-winded introduction to say that PHP 6 will be out before long, and I'm very unexcited. It brings native unicode; but the mbstring extension already did the job fine. It removes bad stuff that was already deprecated, but I didn't use it anyway.

This article made me concerned it may turn out to be a negative experience overall, rather than a neutral one:
http://www.linux-mag.com/id/7433/2/

It says PHP4-style classes are disappearing. Yet they work fine in PHP 5, and E_STRICT has not been complaining if I've not explicitly written "public". Of even more concern is php 5.3 apparently brings in this change too?

However, it turns out this was just a poor Linux Magazine article. What is being removed in php 6 is "ZE1 compatability mode" (see the php manual description); it had defaulted to off, and only affected the way objects were copied/compared. The "php 4" example Linux Magazine show is valid php 5 code, and (as far as I can tell) will be valid php 6 code too.

So that was a red herring. But I'm also bothered how (string) works for binary strings in php 5, but will need to be changed to (binary) for php 6 - that is a very annoying change for anyone storing binary data in strings (disabling unicode is not an option if you also need to store UTF-8 in other strings)!

Storing website translations in SQL

2009-08-20T10:14:00.000-07:00

In php|a magazine, April 2009, there was an article called "Storing Multilingual Records in the MySQL Database". As the title says, it has some mysql-specific elements, but the concepts are quite general. The author introduced and compared three alternatives; as I have used a fourth way I thought I'd write about it here.

The situation was a product database, where the name, url and description have to be localized. But some entries will be untranslated and should fall back to using the default language entry.

1. Translations in a separate table

Here is the database schema:

product: product_id, group_id, price.
product_translation: product_id (links to product), language_id (links to language), name, url, description.
language: language_id, collation.

This requires an SQL join which is hard for mysql to execute efficiently. Fulltext indexes are also a problem, as the name and description fields contain text from different languages. The only good point of this approach is that it is easy to add a new language.

2. Data Copy

Here the group_id and price fields have been moved into the product_translation table. That makes the SQL queries a bit cleaner, as it saves one join, but doesn't really solve the other problems. And the data redundancy is an accident waiting to happen (I'm having a physical reaction just thinking about it).

3. Translation Directly In The Database

This is like 2 above, but each language gets its own field. For instance if we have three languages, English (en), German (de) and Japanese (ja) then it looks like:

product: product_id, language_id (links to language), name_en, url_en, description_en, name_de, url_de, description_de, name_ja, url_ja, description_ja, price, group_id.

The advantage this brings is using the default language is easy; you can either just always select the default language field, or use mysql's IFNULL function. E.g. either:
SELECT name_de,name_en WHERE product_id=123;
(and then check in your PHP script to see if name_de is blank, and if so use name_en instead)
Or:
SELECT IFNULL(name_de,name_en) as name WHERE product_id=123;

The other advantage is FULLTEXT indices work well now, as only one language is kept in each text column. Disadvantages are the space-wasting if actual translations are sparse, the work required to add a new language, and possibly hitting the mysql maximum row size (64K).

4. One table per language

This is the approach I've used in MLSN, among others:

product: product_id, group_id, price.
product_translation_en: product_id, name, url, description.
product_translation_de: product_id, name, url, description.
product_translation_ja: product_id, name, url, description.

Fulltext indices work well, joins are relatively easy, no data duplication. Adding a language feels cleaner than solution three above, as it doesn't require modifying an existing table, just adding a new one. If the site takes off then different languages can be split to different servers easily. And the language table can disappear as collation will now be part of the field definition when the table is created. (I think the same can be said of solution three, but the php|a article kept the language table in its schema - was that an oversight, or am I missing something?)

The dark side? Getting the default language requires two queries. Or would something like this (untested) SQL work?
SELECT ISNULL(de.name,en.name) FROM product_translation_de de,product_translation_en en WHERE de.product_id=123 AND en.product_id=123

Thoughts?

The 4th option works well for me, but I'd be interested to here arguments against it. Perhaps you use something different again?

New fclib and dcflash releases

2009-07-27T20:39:00.000-07:00

A few days ago I updated my dcflash project on SourceForge. This is an open source (MIT license) actionscript library, with a data visualization emphasis:
http://sourceforge.net/projects/dcflash/

In particular there is now an AS3 version. In fact the AS2 and AS3 releases currently have only a couple of classes in common. The AS3 classes are mostly for loading images, movies, and setting up slideshows; none of the chart classes from the AS2 version (or the only partially ported and unreleased AS1 library) are there yet. As I say in the release notes, I'm open to offers to port them to AS3. (A look at http://dcook.org/work/charts/ will show you the kind of things in the AS1 and AS2 libraries.)

And now today I've put up a new release of fclib. This is an open source (MIT license again) PHP library. Fairly ad hoc, but naturally lots of internationalization-related classes. The new 0.4.19 release contains some arabic classes, but other than that it is mostly just tweaks and small improvements:
http://dcook.org/software/fclib/

Date comparisons in C++

2009-07-22T17:03:00.000-07:00

Just been scratching my head over a code snippet that looks like this:

time_t now=...;
time_t last_tick=...;

struct tm *timeinfo=localtime(&now);
...
struct tm *tick_timeinfo=localtime(&last_tick);
...
if(tick_timeinfo->tm_mday==timeinfo->tm_mday &&
tick_timeinfo->tm_mon==timeinfo->tm_mon &&
tick_timeinfo->tm_year==timeinfo->tm_year)return OK;
else return BAD;

I'd realized this was code was always returning OK, even when last_tick was yesterday, a month ago, or even from last year.
Can you see the bug? The above is all you need to know. Answers in a comment. No stamp required.

(To be fair to my bruised ego, the problem can really only be one thing in the way I've presented it above.)

Kudos to the first reply. And I'd be interested to hear how people do date comparisons in C++ more safely.

Screen still going blank! (/proc/mtrr)

2009-07-21T20:40:00.000-07:00

I wrote before how my screen keeps going blank, and how it had seemed to get worse after upgrading to software RAID. It has still been doing it, and the past week or so I've been systematically disabling things to see if I can fix it. It still happens, but less frequently, so some combination of things I've disabled may have helped?

But, it does still happen, so I went googling again. I think I may have found something:
https://bugzilla.redhat.com/show_bug.cgi?id=446620
https://bugs.freedesktop.org/show_bug.cgi?id=15360

It seems Intel video driver and/or the linux kernel cope badly with unusual memory configurations. I moved to software RAID at the same time as I moved from 2G to 6G main memory! Here is my /proc/mtrr:

reg00: base=0xd0000000 (3328MB), size= 256MB: uncachable, count=1
reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
reg02: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg03: base=0x100000000 (4096MB), size=2048MB: write-back, count=1
reg04: base=0x180000000 (6144MB), size= 512MB: write-back, count=1
reg05: base=0x1a0000000 (6656MB), size= 256MB: write-back, count=1
reg06: base=0xcf700000 (3319MB), size= 1MB: uncachable, count=1

This looks like one of the problem ones from the above bug report. Unfortunately I'm at a bit of a loss what to do with it now. My understanding of the problem is something like main memory and video memory overlap, so when some program uses that memory it kills my video and X dies. My ctrl-alt-F1, breath in, breath out, ctrl-alt-F7 "fix" must reset X?? (Hhhmm, how come I never see problems with any programs having their memory overwritten by the graphics card though?)

Here are my dmesg entries either side of the only mtrr complaint:
[ 142.691737] [drm] Initialized drm 1.1.0 20060810
[ 142.700818] ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 17
[ 142.700827] PCI: Setting latency timer of device 0000:00:02.0 to 64
[ 142.700880] [drm] Initialized i915 1.6.0 20060119 on minor 0
[ 142.715171] mtrr: type mismatch for d0000000,10000000 old: write-back new: write-combining
[ 143.311587] set status page addr 0x00033000

These are the best explanations I've found so far:
http://www.rage3d.com/board/showthread.php?t=33821469
http://www.rage3d.com/board/showthread.php?threadid=33736241

Both the solutions in the 2nd link talk about modifying the video=... parameter given to the kernel. But I don't have one of them. I just tried throwing it on the end of the kernel commands but the /proc/mtrr output is unchanged, so I don't think it had any effect:
kernel /vmlinuz-2.6.24-24-server root=/dev/md3 ro quiet splash nomttr

I've just tried changing Advanced|Chipset|Northbridge|Memory Remap from Enabled to Disabled, in my bios. Main memory has dropped from 6G to about 5.3G, and /proc/mtrr has changed to:
reg00: base=0xd0000000 (3328MB), size= 256MB: uncachable, count=1
reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
reg02: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg03: base=0x100000000 (4096MB), size=2048MB: write-back, count=1
reg04: base=0xcf700000 (3319MB), size= 1MB: uncachable, count=1
reg05: base=0xcf800000 (3320MB), size= 8MB: uncachable, count=1

Well, it looks just as dubious to my untrained eye (for starters it still seems to think there is 6G of memory), but at least it is different. Let's see what happens... no screen still goes blank.

What about DVMT mode? http://www.techarp.com/showfreebog.aspx?lang=0&bogno=322
I currently am in "DVMT" mode, with 256M. I've plenty of memory, so let's try fixed mode, with 256M. ...it made no difference to the /proc/mtrr output, and the problem still happens.

(There is also an "ASMT resolution" option in the bios, which is "enabled". Google isn't helping me much here, but some explanation is here and it doesn't seem to be to do with video memory: http://www.avsforum.com/avs-vb/showthread.php?t=938473&page=12)

(I've been holding off on posting this for the past week, in the hope I'd resolve the problem. But unfortunately I haven't yet. I guess fiddling with /proc/mtrr directly is needed, but I don't have time to investigate that currently.)

Email on ubuntu (exim)

2009-07-17T19:28:00.000-07:00

When I installed real player a few months back I noticed it had installed exim. I thought that highly suspicious - what was real player planning to do behind my back? Well, I thought I'd try removing it today, and discovered realplay depends on lsb-core (linux standards), and lsb-core depends on exim. realplay was innocent.

The magic command to remove exim is:
dpkg --purge --ignore-depends=lsb-core --ignore-depends=mailx exim4 exim4-base exim4-config exim4-daemon-light

But after doing that I changed my mind. As I discovered I couldn't send any email from commandline. Did somebody just say "Duh!". Well I was hoping I could then configure something to tell it to use SMTP directly to my ISP. (Why do I want to send email from commandline? Well, yes to test programs, but much more importantly that that I realized my system has been sending me warning emails and they've been undeliverable, so for instance the daily mail telling me about my RAID problem has not been reaching me...)

Anyway, lsb-core and mailx were screaming at me that their dependencies were missing, so re-installing exim was just a matter of telling them to re-install. Then the key step I had been missing was this:
sudo dpkg-reconfigure exim4-config

Still not easy, but it becomes much clearer if you know that what exim calls a "smarthost" is what the rest of the world calls "my ISP"! I.e. in my case I told it to use a smarthost for outgoing email, no incoming email, and for the smarthost I just looked in my thunderbird configuration for the SMTP server and gave it that address. It also kept asking me for a default domain and I chose "dcook.org" for everything.

Seems to have done the job!

Oh, the raid problem? It is just complaining I didn't set up swap as RAID. This was deliberate, see my original post on raid setup. At the time I wrote that I didn't understand the issues involved regarding RAID and swap. But if the system is complaining I will set it up as raid to make the message go away.

Alternative http logins from firefox?

2009-07-14T18:37:00.000-07:00

Open source works on the "scratch-an-itch" principle, and so there should be solutions for the itches that developers have. So I must be using the wrong keywords as I just cannot track down the firefox plugin I need.

When I build a web site I need to login as different users. E.g. a normal permissions users, and an admin-level user. I've input both in, firefox is storing both of them, but it only ever offers me one. Ironically way back when Firefox was called the Mozilla Application Suite it did have this - when two or more usernames had been saved it would pop-up a dialog letting you choose. Firefox 1.5 was a two steps forward, one step back kind of release in that respect. (Of course Firefox 2 and Firefox 3 were both "0.6 steps forward, 0.55 step backward" releases, with the kicker that you need to waste time finding replacement plugins, but that is a rant for another time.)

Help! Please? There must be some firefox plugins that does this simple task.

P.S. I'm also keen to have some function that will allow me to selectively remove certain http authentications, but not the blanket "Miscellaneous|Clear Private Data|HTTP Authentication" option of the Web Developer plugin.

The Shodan Go Bet

2009-06-04T00:03:00.000-07:00

I am a natural optimist. A hopeless optimist. I spend hours battling against myself with carefully constructed cynicism, but things still get past my guard. And one day, back in 1997, I made the mistake of putting my money where my mouth is. Let me tell you were it all started...

http://dcook.org/gobet/

(About a bet I made with John Tromp, in 1997, that a computer could beat him at the game of go before the year 2011; the above URL is to publicize this bet and also has places where you can vote with your opinion and leave comments. If you enjoy it please do link to it, blog about it, tell your friends, etc.)

Linux partition advice

2009-05-28T19:38:00.000-07:00

When I first partioned my disk I put /boot on the first partition and gave it 96M, following some advice found online I assume. This is not enough. I suggest giving it 256M. It is still a fraction of your hard disk. And because /boot is special it is impossible, or at least complex, to move some files to other partitions and link to them.

About a week ago ubuntu upgrade started complaining about a kernel upgrade problem. I ignored it for a few days assuming it would sort itself out. But it kept happening. Then when I viewed details of the upgrade I noticed (just briefly before it vanished off-screen) it was saying not enough diskspace on /boot.

Not again! When I tried to upgrade from Ubuntu 7 to Ubuntu 8 lack of space on /boot caused problems then too.

Poking around I also found I still had linux-generic packages installed, even though I'd switched to the linux-server kernel (see 6Gb on 32-bit linux). I deleted all packages that had the word "generic" in their name. After a reboot I still had one "generic" file left in /boot which I then just deleted. There was also a *.bak file for my current kernel. Datestamp was for a week ago, so I deleted that too.

It doesn't look like I've broken anything, and I'm now down to using 40M on /boot with 48M free (and I still have Ubuntu 7's linux-server kernel in there, which I think is now pointless, so I could reduce it even more).

Therefore, you can get away with a mere 96M /boot partition, it just requires more time and effort.

Conversely, I think a /boot partition above a certain size (1G?) causes problem at boot time, which is the whole reason for have a separate /boot partition. But I'm no expert, and that may be old-fashioned advice, and every BIOS on every motherboard made in the last 10 years may in fact be fine.

I dunno, and am too busy with more interesting stuff to study up on it, which is why I'll go for a 256M partition on my next computer.

Japanese NLP mailing list

2009-05-25T19:20:00.000-07:00

A new mailing list for discussing Japanese NLP (natural language processing) in English has been set up, by Jim Breen (of JMDict fame):
http://groups.google.com/group/nlp-japanese?hl=en

There is a lot of software for processing Japanese text which is only documented in Japanese, and even then only minimally documented. So the new list is an ideal place (for those of us more comfortable in English than Japanese) for asking about how to use chasen, cabocha, namazu, etc. Or to describe what you are trying to do and get program and data suggestions. Hopefully people will also post about new software and data releases, related conferences, new academic papers, and so on.

Also, if the above interests you then you will also want to know about this Ubuntu repository for all kinds of NLP software:
http://cl.aist-nara.ac.jp/~eric-n/ubuntu-nlp/dists/hardy/japanese/

Much of the Japanese stuff is UTF-8 ready (as opposed to the EUC-JP that academic Japanese software still likes to default to).

Adobe PDF reader and Japanese fonts

2009-05-21T18:36:00.001-07:00

Another casualty of my recent enforced ubuntu upgrade was Japanese fonts in pdf files. Adobe acroread has also moved from version 8 to version 9. When you meet Japanese in a PDF file it tells you the URL to go to to get asian font pack. Unfortunately that page only has asian fonts for acroread 8 and earlier!

The link should be:
http://www.adobe.com/support/downloads/product.jsp?product=10&platform=unix

(I'm mentioning this as it took a bit of work to discover it.)

Scroll down to the add-ons section, and it seems each language is now its own file, and the files are much bigger. (I don't know the difference between a "Font Pack" and a "Font Packs"; the files I are identical so I chose the latter.)

Unzip the bz2 file with "tar xjf FontPack910_jpn_i486-linux.tar.bz2"
Then move into the JPNKIT directory and type "./INSTALL"

The install process asks:
"Enter the location where you installed the Adobe Reader? /opt"

I didn't install it, Ubuntu did. However it seems Ubuntu is putting it in /opt!
Strange for a package-based distro to put anything there, but I accepted the /opt default and it worked. (This was different in Ubuntu 7, as I remember having to try lots of paths until I guessed the one it was after.)

Incidentally I have already got medibuntu.org as an extra repository, but there is no acroread-fonts or similar package. Perhaps there is some legal issue (though I thought medibuntu.org's raison d'etre was packages with legal issues).

Moving encrypted partition to software RAID

2009-05-09T19:41:00.000-07:00

I moved most of my partitions to software RAID a few weeks ago. But I left /home/darren/ because it was encrypted. However a few days ago I moved it too. Here is how.

The quick overview: it is exactly like moving any other type of existing partition to software RAID, except where you would format /dev/md7, prior to copying the existing data over to it, you would set up crypt on /dev/md7 instead.

Detailed Steps

These instructions assume that you have moved other partitions to software RAID, or are at least familiar with the process (see previous article). All commands are run as root.

If you have not got an existing crypt partition you need to prepare for it:
* Install cryptsetup
* modprobe dm-crypt
* Add dm-crypt to /etc/modules

I will be setting up /dev/md7 for software RAID 1, but with just /dev/sdb7 in the raid array initially, i.e.
mdadm --create /dev/md7 --level=1 --raid-disks=2 missing /dev/sdb7

Next I setup /dev/md7 for crypt with these commands:
cryptsetup luksFormat /dev/md7 -c aes -s 256 -h sha256
cryptsetup luksOpen /dev/md7 somecryptraid
mke2fs -j /dev/mapper/somecryptraid -L somecryptraid

Each command only takes a few seconds to run. You will be prompted for a password. Forget that password and the data on your partition is lost forever; there is no recovery ability. So choose carefully.

The next command sets chkdsk to run every 100 days, however many times you boot. This is just personal preference, and completely optional (the default is to run chkdsk more frequently):
tune2fs -c 0 -i 100 /dev/mapper/somecryptraid

Run "mkdir /mnt/md7" then edit /etc/fstab and add this line:
/dev/mapper/somecryptraid /mnt/md7 ext3 defaults,noatime 0 0

What we are doing here is saying we want our new encrypted software raid partition to be mounted somewhere temporary, so we can copy our existing data over to it.
(the "noatime" flag is optional, and nothing to do with software RAID or crypt)

And edit /etc/crypttab to add this line:
somecryptraid /dev/md7 none luks

This is the command that will cause it to prompt for password on boot time. If you already have an encrypted partition then you are adding the above in addition to your existing entry: you will have two encrypted partitions for the next boot.

Now reboot into single-user mode (the recovery kernel). You should be prompted for the password for your new crypt partition (in addition to any existing crypt partition). Now I move /home/darren to the new crypt partition with:
cd /home/darren/
cd -dpRx . /mnt/md7

This took over an hour to run for me.

Once it finishes edit /etc/fstab to change the /dev/mapper/somecryptraid entry from /mnt/md7 to be /home/darren. And comment out the previous entry for /home/darren. I.e. the new entry looks like:
/dev/mapper/somecryptraid /home/darren ext3 defaults,noatime 0 0

Reboot. When running df you should see a line like:
/dev/mapper/somecryptraid ... ... ... ... /home/darren

You can now remove (or comment out) the old "somecrypt" entry from crypttab. Also use fdisk to change the /dev/sda7 entry from "83 Linux" to "fd Linux RAID autodetect" (use the fdisk "t" command to do this). Reboot again.

Now you should be able to run:
mdadm --add /dev/md7 /dev/sda7

This will take a while to run, as it mirrors the data from /dev/sdb7 to /dev/sda7. "watch cat /proc/mdstat" will show its progress.

"rmdir /mnt/md7" as a tidyup step at the end. You might also want to delete commented out lines in /etc/cryptab and /etc/fstab if you like to keep those files lean and clean.

Troubleshooting

When I ran "mdadm --add /dev/md7 /dev/sda7" I got this error:
"mdadm: add new device failed for /dev/sda7 as 2: No space left on device"

I tracked this down to slightly different sizes as reported by fdisk:
/dev/sda7 16988 23361 51199092 fd Linux RAID autodetect
/dev/sdb7 16988 23361 51199123+ fd Linux RAID autodetect

Even though start and end cylinders are the same, the number of sectors is different! When I change the view ("u" command in fdisk), to show start and end sector, the problem is clearer:
/dev/sda7 272896281 375294464 51199092 fd Linux RAID autodetect
/dev/sdb7 272896218 375294464 51199123+ fd Linux RAID autodetect

In other words /dev/sda7 starts a few sectors later. My fix was to delete /dev/sda7 from the partition table, and then recreate it. Then /dev/sda7 showed the same start/end sector as /dev/sdb7. Weird and spooky, but now the "mdadm --add" command worked (after another reboot of course).

Tip: use fdisk to check your partition sizes are exactly the same before starting! The other way to fix this would have been to make /dev/sdb7 smaller but, by the time I realized, it was too late to do that.

Ubuntu Hardy: Hard(y) Work

2009-04-29T22:50:00.000-07:00

After last week's upgrade on my notebook, I did the forced upgrade from Ubuntu 7 to Ubuntu 8.10 on my main machine yesterday. Numerous breakages.

Easy Fixes
* Skype disappeared. I still had the deb package and installing it again cured that. All my settings were still around.

* SciTE: It overwrote my old settings. Luckily I keep a backup copy and merged in my differences.

* Evolution was back. So I deleted again all packages it would let me delete (some parts are entwined around the heart of either Ubuntu or Gnome).

* The Permit Cookies firefox plugin didn't like firefox 3. You have to go to developer's own site to get a version that will install.

* SCIM (Japanese input) hot key had stopped working. This fixed itself when I fixed something else (my guess is the xorg.conf change for the Alt-F1/F7 problem - see below).

Hard Fixes
* pcmanfm. This is better than nautilus. Full stop. But after the hardy upgrade trying to launch anything fails - it tries to launch, but after 20s or so gives up. This even includes opening the terminal. I tried reinstalling, and tried getting the latest version, but same problems.
Solution: I gave up. My nautilus problems seem to have gone, and I installed nautilus-terminal package which gives me the "open a terminal in this directory" feature that I'd become addicted to. (Cannot get F4 to open terminal though: the closest I can manage is "Alt-F, t")

* Ctrl-Alt-F1/F7 no longer worked. See my screen blanking blog entry for the xorg.conf fix for this.

* Real Player Plugin. Firefox 3 decided to move the goal posts and keep plugins in a new directory; so you have to move the firefox plugins to the correct place (/usr/lib/firefox-addons/plugins/) yourself.
That still wasn't enough. I've been disabling various other plugins, reinstalling real player, and lots of swearing. I finally have it working, but I cannot tell you exactly which steps were needed.
In the end I used the Real Player's deb, which is highly suspicious as it has dependencies such as "exim" (an email client)?!! Many years ago Real got a bad reputation for malware, and I'm wondering if they are still up to their bad tricks?
But even just that deb didn't work. So, I think that the thing that finally worked was deleting the totem-mozilla package. (I'd already disabled all plugins with the word totem in them however, which was no help, so I'm not entirely sure... the real Real Player plugin calls itself "Helix DNA Plugin", and I think I'd also disabled that... which with hindsight was probably the cause of all my problems?)

Other Fixes

A few months back the red icon to shutdown gnome stopped working. Hardy didn't fix that. But I did find the cause today: Preference | Sessions | Enable Power Manager. A tad confusing - you'd think it was something only needed on notebooks. But, anyway, that was the key.

Things That Did Work Okay:
* Samba (aka windows) shares
* Software RAID
* Encrypted Partitions
* Email/Browser/sound/video card/etc.

Summary

Thankfully Ubuntu 8 is a LTS (Long Term Support) release, so I shouldn't need to go through this for another couple of years.
And while I now hate Ubuntu I have been reminded about its most important feature: a huge user base. Which means typing "Ubuntu 8 some problem" into Google will always find someone having the same problem, usually with some kind of solution.

Thanks to all the hundreds of Ubuntu users who answer questions, and blog about their fixes!

Ubuntu Expired On Me!

2009-04-24T02:53:00.000-07:00

So, I boot up my notebook into linux (Ubuntu 7) for first time in a while, maybe 2-3 months, to prepare it for a meeting. I'm a good boy and first go to install the updates I know will be there. It says it cannot find some. Strange.

On a hunch I wonder if Ubuntu 7 has reached end of life. But, no, this page tells me Gutsy Gibbon (7.10) is still current:
https://help.ubuntu.com/community/UpgradeNotes

It is wrong. Ubuntu obviously haven't read their own press releases. It EOL-ed a week ago:
http://www.ubuntu.com/news/ubuntu-7.10-eol

What is annoying is how EOL breaks everything. Packages won't validate. Worse, I cannot install some new software (such as mysql; but I could install lame). This is not simply "we're not going to do updates any more". This is "We've pulled the plug. Should've upgraded earlier. Loser."

Two forehead slaps for Ubuntu in the space of a week. I'll be back to Fedora at this rate!

Anyway, I started the upgrade to 8.04, but after running for 20 minutes it finally told me it was going to take 19hrs to run. Not good timing for my meeting!

By the way, 19hrs was a bad estimate. It takes 10-20 minutes to get it started, then 3-4 hours downloading, then 1+hrs installing. It took 6 hrs in total for me. You can leave it alone for the downloading, but need to be around to answer questions during the install stage.

Screen keeps going blank

2009-04-22T04:25:00.000-07:00

I mentioned this in previous entry (on software RAID). My screen keeps going blank. Ctrl-Alt-F1, breathe in, breathe out, Ctrl-Alt-F7 fixes it. I.e. that must reset the video card or X.

It used to happen once a day or less. Since switching to RAID it is happening once an hour roughly. But 9 times in past hour, which was past my threshold, so I went hunting...

...and came up empty-handed. Xorg.0.log has entries but they match what happens if I do the Ctrl-Alt-F1/F7 sequence. /var/log/acpid also has an entry that I think goes along with Ctrl-Alt-F7. But nothing else. No errors in either. Nothing in messages, syslog et al that could be associated.

/var/log/gdm/:0.log also gets an entry when I do ctrl-alt-F7, but nothing beyond that.

Nothing else in /var/log at all.

I'm stumped. I doubt over-heating, and I doubt CPU load (as all I've been doing the past hour is typing in a text file). w tells me "0.00 0.09 0.07".

Not a very informative blog, sorry. Anyone got any suggestions? I don't want to mess around trying different video cards, monitors, etc.; it is still just a minor irritation.

UPDATE: After updating to Ubuntu 8 I still had this problem, but worse my Ctrl-Alt-F1/F7 fix no longer worked! A bit of googling tracked down a solution:
In /etc/X11/xorg.conf, in "InputDevice" section, I commented out these three lines:


  #Option  "XkbLayout" "jp,jp"
  #Option  "XkbVariant" "latin,"
  #Option  "XkbOptions" "grp:alt_shift_toggle,grp_led:scroll"

and replaced with just:


   Option  "XkbLayout" "jp"

Even better, Ctrl-Alt-F1..F6 now give me a terminal! They never have in Ubuntu, instead they just gave me a blank screen. The above xorg.conf has been there since the start - Ubuntu 8 upgrade didn't alter it.
(Oh, and the above change has not fixed the screen going blank problem, so I'm still no closer to fixing that...)

Update: I've noticed if I move the mouse, especially the mouse wheel, just after the screen goes blank, it comes back. It doesn't always work - not sure if there are two problems here, or if I'm too slow reaching for the mouse sometimes.
Anyway, if that suggests a potential cause for someone let me know!
(By the way, I have screensaver set to blank, after 30 minutes of inactivity. Just to see if that is related I've changed screensaver to some animation, and I'll let you know!)

Software RAID on an existing linux system

2009-04-20T02:31:00.000-07:00

So, as mentioned in a previous article I bought a terabyte hard disk, mainly to impress the ladies. It doesn't seem to have been working too well in that respect. I considered sending it back, claiming it needs more testerone. But I decided to instead set up software RAID 1.

My main guide for this task was this article: http://www.howtoforge.com/software-raid1-grub-boot-debian-etch
Rather than repeat everything I will just comment on what I did differently. Note that despite the name of that article, it applied fine to Ubuntu 7, and I suspect it will apply equally well to any linux distro of the past two or three years.

First, my disks were different size: 320G vs. 1T. So I opened fdisk -l /dev/sda in one terminal, then ran fdisk /dev/sdb in a second terminal and set up the first six partitions (plus one extended partition) to be the same as /dev/sda. I then set up /dev/sdb8 to be one big partition with the rest of the disk, which I mounted as /backup.

In all software RAID guides they number the partitions /dev/md0, /dev/md1, etc. That is confusing so I decided to keep the same numbering schemes as my hard disks; it turns out this is fine. So I used commands like this:
mdadm --create /dev/md3 --level=1 --raid-disks=2 missing /dev/sdb3

compared to this from the HowToForge article:
mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb3

The above article did all partitions in one go. I wanted to take baby steps. So I started by mirroring one partition, choosing a non-critical one, and ignoring all that stuff about grub configuration. It worked well. I then did /var and /, leaving /boot unmirrored and therefore again ignoring all that grub configuration stuff. /var went smoothly but / did not. It kept saying /dev/sda3 was in use: "mdadm: Cannot open /dev/sda3: Device or resource busy" But df didn't list it. Trawling /var/log/* didn't list it. "Eh?" thought I.

Eventually I realized that grub was set to boot from /dev/sda3, not from /. I.e. grub works at the device level not the partition level.

Key Advice: mirroring one partition at a time is fine, but then do "/", "/boot" and all that grub stuff in one go.

Now, something the HowToForge article didn't mention at all was doing file copies in single-user mode. I know just enough unix administration to scare myself and the idea of taking a copy of /var and / while all the daemons were still running made me nervous. So I tried to boot to single-user mode.

Discovery #1: Ubuntu is weird. They don't use run levels like the rest of the linux world. Sure, their numbering scheme may make more sense, but it also causes much surprise.

Discovery #2: "telinit 1" doesn't work. The GUI shuts down and then nothing. I can see the system is still running but nothing on screen. Nothing on Ctrl-Alt-F1 through to Ctrl-Alt-F12. I think ctrl-alt-del caused a reboot.

So, I booted, and choose the "recovery kernel" from the grub menu. Recovery being another name for single-user mode. It asks for the root password. Ah. Ubuntu doesn't use one. I have to continue into a normal GUI boot.

Key Advice: Set a root password. This has nothing to do with software RAID: you never know when you are going to want to boot into a recovery kernel. I think Ubuntu deserves a forehead slap for this design decision? To set a root password, do "sudo passwd": it will first ask for your user password (that is how sudo works), then it will say "Enter new UNIX password:". I chose the same password as my normal user; easy to remember. (If that makes you nervous, remember that this is equal security to being able to run "sudo passwd"; more importantly, it saves me having to write down my root password somewhere.)

Other comments: the "cp -dpRx" command is the most time-consuming step for those partitions when you have a lot of data. The system will be sluggish while doing this (and if you are in single-user mode you cannot be browsing or doing anything else anyway). But also when you do the "mdadm --add /dev/md3 /dev/sda3" to actually create a genuine RAID partition it will be copying a lot of data, and your system will be sluggish for the time this takes (about 10 minutes per 30G). Bear this in mind.

Swap. I've created /dev/md2 as a swap RAID, but haven't used it. So /dev/md2 just contains /dev/sdb2. And "cat /proc/swaps" tells me only /dev/sda2. Apparently if my sda disk dies my swap will disappear and my system might crash. On the other hand, mirroring swap has a performance downside (apparently). I don't understand the trade-offs well, but this is a workstation, not a server and a crash should /dev/sda ever die is acceptable. I've also got more memory than I need so I suspect I could switch swap off completely. Anyway, so far Inertia born from Igorance means I'm doing nothing about this. Expert opinions are welcome.

Everything works. Just two things I've noticed. Sometimes the system doesn't boot. It comes up in an "ash" shell, which is one of those recursive acronyms standing for "Ash is Satanic Hell". So far the only command I've mastered is "reboot", which works well - I've not had two boot failures in a row. After 10-20 reboots I think this boot failure is happening about 20-25% of the time. I've no idea how to troubleshoot it.
UPDATE: this boot problem doesn't seem to be happening since upgrading to Ubuntu 8. Perhaps the upgrade fixed some mistake in my grub.conf? Maybe related is that I no longer have an (hd1,0) section in grub.conf; they are all (hd0,0). I think I'll leave it like that: if "hd0" dies I can edit the grub boot string that once, and then edit grub.conf.

The other thing is my screen sometimes goes blank. The system is fine, it is just like the video card has died. My solution has always been: Ctrl-Alt-F1, pause for half a second, then Ctrl-Alt-F7. I guess this process resets the video card. It used to happen maybe once a day or so. Now it seems to have happening once an hour on average, and is becoming almost irritating enough that I'll have to look into it. The fact that running RAID could affect the display feels like an important clue.

Finally, my home partition is encrypted, using crypt. I have not moved it to software RAID yet, for that reason. I will really soon (as all my data that could really benefit from the security of software RAID is currently the only data not protected by it)! I will run crypt on top of software RAID, rather than the other way around; I'm just not sure the best way to do the file copy.
UPDATE: now done, it was straightforward, see Moving encrypted partition to software RAID.

Excel and PDF from PHP

2009-04-14T23:47:00.000-07:00

Been meaning to post this for ages, but PHP Architect magazine for November 2008 had a useful article on libraries and code for alternatives to HTML.

For PDF html2pdf was introduced. Unlike other PHP PDF libraries (e.g. PDFLib, FPDF) this one has you make your content in HTML then it gets converted. For those of us comfortable with HTML, that means no learning curve.

For Excel, PHPExcel is introduced. This has classes to read and write excel files in a variety of formats. I get this request a lot but so far we have always decided outputting good old CSV is best (simple to make, compact, easy to import into Excel). But if complex layout, or built-in functions, are required this looks worth a look. It can also output into PDF.

Also very interesting is in-memory spreadsheets, with lots of mathematical functions supported. I've not tried it, and neither the article or the website mention explicitly if this will work on Linux (or a Windows machine without Excel installed), but it is described as standalone so it should. It might be an interesting way to port someones spreadsheet to a PHP app!

The article also briefly introduces Google charts, and how to include the produced image in your excel file.

6GB on 32-bit linux

2009-03-30T20:03:00.000-07:00

Well, it all started when a friend said their SMT (statistical machine translation) system was ready to download and install. He then casually mentioned it is a bit of a memory hog, "4Gb minimum, 8Gb preferred".

Wow. I looked at my up-until-then-perfectly-adequate-some-might-even-say-overkill 2 gigabytes and felt like a salesman who had just been told he ought to upgrade his sensible car for a sports car in order to look more successful!

I have dual-channel, and two slots free. Another 4Gb was under 5000 yen at Dospara (Japanese only), where I had bought this computer, so I emailed them to confirm it would work. After a few emails, and a bit of research I found out:
* Windows 32-bit will only go up to 3Gb.
* 64-bit Windows, 64-bit Linux are both fine well beyond 8Gb.
* 32-bit linux will allow up to 4Gb per process, and can use more than 4Gb altogether.

(I wanted to stick with 32-bit linux, as Flash is critical to my work and has no 64-bit version.)

Dospara were rather cautious, saying they don't support linux, but I went for it. When I plugged in the extra 4Gb, the bios correctly recognized 6Gb. Then Ubuntu said I had 3Gb. But that was okay, as I'd been expecting it. I went to the package manager and selected the "linux-server" meta-package, then rebooted.

Drum roll please: "free" reports I have 6Gb available. I'm using 475 Mb, and have 5,745 Mb free. See, I told you I didn't need it. But this is city driving. You wait until I take this beast down the local Formula One track, otherwise known as Difficult AI Problems.

Oh, while I was there I also bought a 1Tb SATA drive. Yes, that is a "T". A whole terabyte in a little, diddy box. It was only 10,000 yen (you could get one for 7,880 yen if you go for Seagate, but a quick bit of research showed lots of unhappy people, so I went for Western Digital which seemed to be the reliability brand).

What do you mean: "I bet he doesn't need that much storage either" ? Just because my current 250Gb drive still has 143G of free space after 18 months, doesn't mean I suddenly won't need more capacity...

And you can bet that when the ladies hear I am part of the Terabyte Generation I'm going to be fighting them off with a stick. Oooh, yeah! I am so sexy.

Periods in regex character classes

2009-03-16T17:53:00.000-07:00

I'm trying to match Japanese numbers with a regex, and this was originally a long article describing what seemed to be a bug when using the following regex:
$regex='/((0|1|2|3|4|5|6|7|8|9|.)+)(万|億|兆)/u';

(a simplified version of my actual code, but this demonstrated the problem equally well)

Now, periods in character classes (a list of alternatives in square brackets) are not special, so don't need escaping. But they do need escaping anywhere else. I knew this, which is why I was scratching my head.

So, naturally, I felt a bit foolish when I realized I was using a list of alternatives in parentheses. It wasn't a character class at all.

But, now, this was even more confusing because I had already been down that street. My code (which in hindsight was correct) had started out like this:
$regex='/((0|1|2|3|4|5|6|7|8|9|\.)+)(万|億|兆)/u';

and that kept saying no match. E.g. for "730.28".

Did I jump out of my bath and shout "Eureka!" when I realized the problem? No. It was more of a forehead slapper. It wasn't "730.28万". It was just "730.28". The code works by trying to match the above regex, then if it fails passes it to another function. I'd been debugging the wrong function. Once I was looking at the right function it was a trivial fix.

Moral of the story: don't be stupid.

(Or more seriously, "Time To Refactor", as the code is now tangled enough that it was too easy for me to get entangled in the wrong place.)

What is an ontology?

2009-03-08T19:20:00.000-07:00

I went to InterOntology 09, at Keio University a week ago. Actually I only attended the first couple of talks and the (free!) banquet; I couldn't make the other talks. Ontology is one of those words I have had trouble grasping, and I attended with no more ambition that understanding what it means.

My ambition was not fulfilled, but I was relieved to know that most attendees were just as confused as me. Okay, relieved is perhaps the wrong word. And by "confused", I don't mean people stood around scratching their heads. I mean people were using it in different ways. Most people who said they were "building ontologies" were actually building databases.

Someone I met there kindly sent me this link:
What are the differences between a vocabulary, a taxonomy, a thesaurus, an ontology, and a meta-model?

This (in conjunction with its first comment) is an excellent article. A bit heavy, but definitely worth the effort.

I feel it backs up the opinion I've been forming that ontology is a word that does not really need to exist. People "building ontologies" are generally data modeling, or taxonomy-building, or semantic-network-building. These terms are more specific, and contain more information about exactly what you are doing.

People using the word ontology may want to emphasize that the grammar allows for validating models using logic. Personally I would rather call that validating the data model, though I do see how a widely used ontology representation language could encourage high quality data validation tools. But they are nowhere near that yet - using SPARQL (pronounced sparkle) appears to be more like writing in assembler than the SQL its name hints at.

On the other hand, if everyone listened to me there would have been no InterOntology conference, and I'd have missed out on a free dinner. Such things need to be taken into consideration. Perhaps I should add "Professional Ontologist" to my business card after all? ;-)

Comparing two arrays in PHP when elements can be in any order

2009-03-05T17:16:00.000-08:00

This comes up so often. Usually I want to compare two CSV strings, where order does not matter.

My usual strategy is to first explode(',',$s) both strings. Then loop through one array, seeing if that element exists in the second array. If it does not then we instantly fail. If it does match then remove that element from the second array. At the end, if no failures and if the second array is empty, then they are the same.

When writing the above code for the umpteenth I wondered for the umpteenth time if there was a better way. I checked the PHP manual and still no "compare_arrays_can_be_in_any_order()" function. (Or, am I missing it - please let me know!)

But maybe I can use a couple of functions?

First idea: array_intersect(). Returns all elements of array1 that are in array2. So:

$intersect1=array_intersect($list1,$list2);
if($intersect1!==$list1)return false; //list2 is a subset of list1.
$intersect2=array_intersect($list2,$list1);
if($intersect2!==$list2)return false; //list1 is a subset of list2.
return true; //They are supersets of each other. I.e. equal.

I've not tried this, as my second idea was better: sort(). The implementation is:

function compare_arrays_can_be_in_any_order($list1,$list2){
sort($list1);
sort($list2);
return ($list1===$list2);
}

Note: this is destructive, as both list1 and list2 are modified. Written as above this is fine as I pass the arrays in by value. But be careful if pasting just that code into another function.

Here is the function to solve my original csv matching problem:

function compare_csv_strings_can_be_in_any_order($csv1,$csv2){
$list1=explode(',',$csv1);
$list2=explode(',',$csv2);
sort($list1);
sort($list2);
return ($list1===$list2);
}

Bear in mind this will only work for simple csv strings: if one uses quotes and the other doesn't, or one uses whitespace around the comma and the other doesn't, you need something more sophisticated.

Now your homework question to make sure you are paying attention: sort() takes an optional parameter of flags, to specify sorting items numerically, as strings, or as strings using current locale. Do I need to specify any flags to have my code work with any type of array data, and to work on a machine using a different locale?

What about comparing arrays of arrays?

And final question, the one of most interest to me: do you have a better way? (Better could mean any of: less code, quicker execution or fewer restrictions/disclaimers!)

Specify the default, stupid!

2009-03-03T17:46:00.000-08:00

I've been struggling all week to connect to a Microsoft SQL server. It was SQL Express. I'm using "ADODB.Connection" COM object from PHP.

(The reason to use that is you can specify the 3rd param as CP_UTF8 (without quotes) and then it will convert from UTF-8 to the UCS-2 that SQL Server works in, all behind the scenes.)

First, if it says it cannot connect due to there being no server: you have to enable TCP/IP connections in SQL Server. They are off by default.

That got us to the next error, which said "接続が正しくありません。" which translates as "The connection is incorrect." Not much to go on. I think this may be the same as "error 26", but I'm not 100% sure on that.

My code could connect fine to the production DB machine, running SQL workstation (which is the more expensive version I believe). And a client on another machine that could also connect to that production DB, could not connect to our SQL Express machine.

So, with all fingers pointing at the SQL Express server, we tried things and googled things and swore at things. Wrong barking tree!

I've given the answer in the subject. SQL Server listens on port 1433 by default. Our SQL server was listening on that port, so you would think no need to specify it. But this DSN fails:

DRIVER={SQL Server};
SERVER={127.0.0.1};UID={myuser};PWD={mypassword}; DATABASE={mydatabase}

whereas this one works:
DRIVER={SQL Server};
SERVER={127.0.0.1,1433};UID={myuser};PWD={mypassword}; DATABASE={mydatabase}

Only for SQL Express it seems. No need to specify the default port when connecting to SQL workstation.

Nice one Microsoft. Just got to work on quality control and documentation a bit more, oh, and consistency, and then they could probably go professional with this software business of theirs.

PHP comments from PHP!

2009-02-15T17:34:00.000-08:00

php|architect December 2008 edition's most interesting article was on phpdoc. I'm a fanatic when it comes to the javadoc style of commenting. When I create a new function the first thing I do is write function somename(){}, then the second thing I do is write /** */ above it.

If I'm refactoring then I might copy and paste in some code straightaway, but usually the third thing I do is document. I describe what the function does, er, will do, and usually define the @return tag (if I've not worked out what data, if any, it should return then I've got ahead of myself - how can I even name a function if I don't know what it produces??). The @param tags I usually leave until after I've written and tested the function, as they can change a lot (i.e. I need an algorithm before I know what inputs my algorithm takes).

During the writing of the function body I will usually jump up to the comment block and add an @internal comment (describing the implementation), or a @todo comment ("Need to add error-checking on call to somefunc()" is a very common one!).

To quote from the php|a article: "What's really exciting,though, is that since PHP 5.1, the Reflection classes can also supply the subject's "doc comment".

See http://php.net/reflection

The php|a article introduced a base class for automatically creating setters for class vars. My own interest is in somehow using it for contract based programming (also called DBC: Design By Contract).

I believe the Java implementation of DBC uses docblock entries, but with a pre-compiler. Pre-compilers suck: being able to edit and run the same source code file is a big bonus, and maybe reflection of docblocks enables that?

Of course using comments is poor cousin of DBC built into the language itself (especially if it can do compile-time analysis). But, for some reason I've not yet grasped, language designers just don't understand how important it is. They limp along with lint and asserts instead.

Back to the docblocks, I've not thought out the details; does PHP provide some hook that can be called when each function is entered and exited? If not I guess you'd have to write VALIDATE() as the first line of each function, and a RETURN() function everywhere you use return, which would be rather intrusive.

Either way, it could then look for @paramvalidate tags which could describe valid data to be found in parameters. It could look for @variant tags in the class doc block to check the object is valid on each of entry and exit.

Did I mention DBC finds loads of bugs, without you having to write a single unit test; all you have to do is document properly. Which, of course, you are going to do anyway if you are a professional programmer.

But did you realize DBC can also be used for optimization, by prioviding hints to the optimizer? As a concrete example (I'm thinking about C++ here) you have two input variables both described as being in the range 1..100. When they are multiplied together the range is 1..10000. Therefore there is no need to consider overflow. If you then divide by that variable then a divide by zero error is impossible. I'm not a compiler writer, and haven't written assembly in about 15 years, but I'm sure the above information, or similar, can help.

As another example, that also applies to PHP, static analysis of possible values can tell you that for an if/elseif/elseif/else block only the second elseif block can ever be reached. This can be both reported as a compiler warning and all the conditional stuff thrown away.

But, that is all a pipe dream, it requires DBC being implemented in the language itself, and I've drifted from the point of this article. Which was: a PHP script can read its own comments! Cool!

Merging PDF files (and more!)

2009-01-29T18:20:00.000-08:00

My mobile phone company decided, 7 months ago, to save the world by not sending paper invoices any more. Unfortunately they didn't tell me (and registering to see invoices online required jumping through lots of hoops, while holding a PIN number I didn't know).
All sorted now, and I only lost one invoice (they only keep the last six online). So I downloaded six shiny new PDF files. Like my phone company I also want to save the world, so wanted to print them four to a page.

The solution is pdftk (in Ubuntu's package list; also available for other major distributions I believe). This simple command:
pdftk 2008*.pdf cat output all.pdf

It just worked. Nice. It also does lots of other cool stuff, like adding or removing encryption, remove just one page out of the middle of a PDF, rotate, extract text. No mention of i18n, so I don't know (yet) how it will cope with Japanese PDFs. Or PDFs where the text is actually an image. The projects home page is http://www.accesspdf.com/pdftk/

It appears there are also Windows and Mac versions.

Gnome file extensions are screwing up my life!

2009-01-21T17:10:00.000-08:00

Okay, perhaps a touch of frustration crept into that choice of subject.
When you double click a file in gnome file manager a lot of the time it works. When it doesn't you are in deep trouble. I've visited this problem before:
http://darrendev.blogspot.com/2008/06/gnome-file-associations-air-and-zip.html
http://darrendev.blogspot.com/2008/05/gnome-whether-under-redhat-or-ubuntu-is.html

I never did fix the AIR problem - I had to uninstall AIR completely.
Now, since yesterday, all my inc files refuse to open in SciTE. They say "this claims to be an inc file, but the contents look like PHP. Even though SciTE is your editor of choice for both inc and php files I'm not going to let you open it. Please also check behind the sofa for terrorists."

I suspect it was because I was editing an inc file in jedit a few days ago and manually chose PHP for the syntax highlighting. But if so, how and why is jedit allowed to alter my system configuration??
(I've also noticed that when clicking links within zip files gedit is now associated with txt files; it used to be open office writer. Something weird is going on, but at least this one is an improvement: application associations in zip files are a world of their own, and I've never tracked down any method of configuring them.)

Following my advice in the second of my above blog entries I tried making a php.xml file for *.inc files. But no luck there.

Somebody help! How do I restore this? Or, even better, how do I switch off Gnome's stupid wrong-extension-is-a-security-risk check? In many years of using gnome it has never once complained in a useful way, or detected a genuine problem. Every single time it has just got in the way.

(If you type "gnome-control-center" there is supposed to be an icon in there for file types. It is missing in ubuntu 7! Or perhaps it was only in there in gnome 1 and got removed in gnome 2.)

UPDATE: Problem half-solved. I've installed PC File Manager, and replaced Nautilus everywhere. See the instructions here:
http://linuxondesktop.blogspot.com/2008/05/replacing-nautilus-with-quicker-and.html
(they say Ubuntu 8 but everything seems to apply to Ubuntu 7 fine.)
I had to go through each text file extension and assign it to /usr/bin/scite (it was defaulting to jedit for most of them, and defaulting to nothing for the troublesome *.inc), but after I did that it is working nicely. File associations appear to be much more transparent than in Nautilus.
However following instructions in 2nd post here: http://ubuntuforums.org/showthread.php?t=692238 for replacing the desktop didn't go so well. The icons cannot be moved around on the desktop, and launchers were not working. So I undid those changes, and will now try and change my directory launchers on my desktop to use pcmanfm. It seems I'll need to go through and replace each "location launcher" with an application launcher that calls pcmanfm. A bit of a pain, but looks like it will work.

Perhaps I'll investigate kubuntu... Surely KDE cannot be worse than Gnome/Nautilus?

linux shell: two directories quick switch

2009-01-13T17:02:00.000-08:00

Sometimes I have to work in two directories from the commandline. For instance to check a log file got created, then back to the directory where the command I am testing is run from. The quick tip is simply "cd -". This takes you back to the previous directory. If you type it again it takes you back to where you were, acting as a nice toggle.

"cd" with no parameters takes you to your home directory. "cd ~/somewhere" takes you directly to a directory relative to your home directory (i.e. tilde expands to your home directory). I had a hunt around but couldn't find any other special characters to use with "cd", but if you know one please let me know.

(This tip was found in Linux Magazine, June 2008)

PHP and outputting floats

2008-11-04T16:32:00.000-08:00

I was writing a script in PHP to compare two data sources, and in the report I was outputting using code like this:
echo "A={$vA}, B={$vB}\n";

What was strange is I was getting entries like
A=3.4E+6, B=340000
A=1.4E+6, B=1500000
A=2146666.66667, B=2.3E+6

My first guess was some data was int, so I added asserts for that, but, no, everything is float.

As often happens, the answer was to be found in the user contributed notes in the PHP online manual: http://jp.php.net/manual/en/language.types.float.php#83577

I just have to repeat his final sentence: I have to be honest: this is one of the strangest things I have seen in any language in over 20 years of coding, and it is a colossal pain to work around.

Also in the notes are complaints that output of floating point numbers uses the locale. Using number_format is one solution for that, or using %F in sprintf is also locale-independent apparently. So my code becomes:
echo sprintf("A=%F, B=%F\n",$vA,$vB);

Unfortunately that now gives me entries like:
A=481.000000,B=1150000.000000

I have hit this zero-noise issue using sprintf in C++ too.

So here is my solution (be aware that this adds CPU cycles):

/**
* Format a float, only showing significant decimal places
*
* @param float $v
* @return String
*/
function fmt_float($v){
$s=(string)$v;
if(strpos($s,'E+')===false)return $s;
$s=sprintf("%.9F",$v); //Write with all decimal places
$s=rtrim($s,'0'); //Chop off trailing zeroes
$s=rtrim($s,'.'); //Chop off decimal point if it is left at the end.
return $s;
}
...
echo "A=".fmt_float($vA).", B=".fmt_float($vB)."\n";

I'll throw that into the next fclib release. Can anyone make a better version?

Note: The first two lines mean use PHP's built-in float to string conversion by default and just use our own when it ended up in scientific notation. I found this gave better results (because using %.9F sometimes writes 120.0000000001 instead of simply 120.000000000, whereas the PHP built-in conversion is fine (giving "120") ). However if you also needed a locale-independent solution, then uncomment those first two lines (as PHP's built-in conversion uses locale).

Being emailled followups

2008-10-20T03:44:00.000-07:00

A question about Blogspot seems best posted here! I noticed there had been some comments on other entries. Surely I'm not supposed to poll each article to see if there has been a new comment. I'm sure there was a setting to say to email me all follow-ups to all articles, but a recent search turned up nothing.
Anyway, thanks for the replies!! I want to try out the Actionscript suggestions I received and will then update the articles if they work (or even if they don't).

Microsoft SQL server and PHP

2008-10-15T03:35:00.000-07:00

I'm a big fan of Pear::DB. (And, as an aside, I think the Pear decision to deprecate it was a bit strange.) But it didn't work for me for Microsoft SQL server (aka mssql), for one particular project (I'm fairly sure it has worked on other projects, so I think it was something specific to this environment). What did work was using the COM object.

COM objects from PHP are a bit of a black art, perhaps as most code examples seem to be in Visual Basic, which has a very different syntax. I found the following page useful, as it shows all three ways of running the same query for a mssql database:
http://www.webcheatsheet.com/PHP/connect_mssql_database.php

I think I also used the user comments in the online PHP manual.

One thing the above URL does not show is error reporting. Here is the code I developed, which also takes care of not treating 5701 and 5703 as errors (congrats to Microsoft on another poorly thought out API!):

//$db is the return from the new COM() call.
//...run some Open or Execute command on $db...
$errors=$db->Errors;
if($errors->Count==0)return ""; //It worked
$e='';
foreach($errors as $ix=>$error){
if($error->NativeError==5701 || $error->NativeError==5703)continue; //Information messages, not errors
$e.="Number={$error->Number}, Source={$error->Source}, SQLState={$error->SQLState}, NativeError={$error->NativeError}, Description={$error->Description}\n";
}
return $e;

Centering multiline text fields in AS3?

2008-07-20T21:01:00.001-07:00

This seems to be a regression-without-cure between AS2 and AS3. In AS2 I could write:
txt.wordWrap=true;
txt.align="center";
txt.htmlText="blah, blah, blah";

and each line in that text field would be centered.

In AS3 I am writing:
txt.wordWrap=true;
txt.align=TextFieldAutoSize.CENTER;
txt.htmlText="blah, blah, blah";

but it ends up left-aligned.
The bigger problem is that this is the documented behaviour: if wordWrap is set then align will not center text.

So, how do I center a multi-line text field?!!

(Time passes.)

I cannot believe there is nothing on the web about this. Have all the flash developers in the world stopped centering their text, or am I missing something obvious?

(More time passes.)

OK, after some trial-and-error I have a workaround, which sucks but will get the job done:

CopyA.htmlText='<body><p align="center">blah, blah, blah</p></body>';

Why do I say this sucks? Because I cannot (easily) apply it afterwards: I will need to get the current htmlText contents, detect the body tag and insert after it; and a corresponding closing-P tag at the end.

But in this application the text string comes from a DB and is allowed to contain HTML tags. What if it too already has a P tag? Will I get an extra blank line? Are P tags allowed to be nested?
(Well, I just tried, and it seems fine still, and if the inner P tag has an align that is the one that gets applied. So all good there.)

UPDATE: I discovered I can also write this:
CopyA.htmlText='<p align="center"><body>blah, blah, blah</body></p>';
That is dubious, and I wonder if it is future-proof, or if I am exploiting a bug? But it is obviously far easier, as it saves me having to do any parsing - I can just wrap whatever the existing htmlText is:
copyA.htmlText='<p align="center">'+copyA.htmlText+'</p>';