Rackspace vs. AWS Oct 2014

I’ve previously blogged about Rackspace vs. AWS. For small servers, Rackspace has generally come out on top. But this year the pendulum has swung the other way.

First, Amazon released their t2.micro instance. The t1.micro was really just for learning the AWS API on, but the t2.micro is the kind of instance you can do real work on. (In fact I’ve been running the test version of a DB-backed website on one, for a month or two now, with no issues.)

A t2.micro server is $11.35/month in Ireland, ($15.64/month in U.S. East or Tokyo - I didn’t expect that! Europe cheaper than the U.S.!). This includes a 20GB magnetic disk, at about $1/month. (SSD is $2/month). $11.35 is £7.10/month (at today’s $1.60/£).

The second change this year is that Rackspace have introduced a compulsory service level fee, of £35/month. This is per data centre. (They might claim it is pay as you go, but small customers won’t do enough go-ing, so will always simply be paying £35/month.)

The third change is Rackspace have done away with their low-end servers. You used to be able to get a “next generation” 512MB server for 2p/hour (http://www.rackspace.co.uk/cloud/servers/existing-customer-pricing), and if you are an existing customer you still can. That works out at £14.40/month. The new minimum server is £52.52/month (including the £35 service level charge). (Of course, the new minimum server is higher spec., but as all I needed was the previous minimum spec, that does not matter.)

By the way, existing customers get a different pricing system, and don’t need to pay the service level charge. However, the per-hour prices are higher, e.g. for “Performance 1” (1GB RAM, 20GB SSD) it is £21.60/month, compared to £17.52/month (+£35) for new customers. (I guess this is related to their point - the service level was hidden in the prices, and now they’re just breaking it out… well, if the price was exactly the same, without the minimum fee, I’d have no problem with that.)

In Rackspace’s favour is their locations. They have a London data centre, the closest Amazon has is Ireland. They have a Hong Kong data centre, the closest Amazon can manage is Tokyo or Singapore.

In AWS’s favour, you have one account, globally, whereas with Rackspace it is one account per data centre.

BTW, one way they both make life difficult is by hiding their calculators. Here is AWS:
http://calculator.s3.amazonaws.com/index.html
And here is Rackspace UK’s:
http://www.rackspace.co.uk/calculator

So, to sum up, if all you need is minimum spec, Rackspace are now either £14.40 or £52.52/month (depending on if you are an existing customer at your desired data centre or not) while Amazon is now £7.10/month.

(Be aware that the prices are always in motion; but if you think I’ve misunderstood something above, please let me know in the comments.)

Written with StackEdit.

Fixing SVG font glyphs by hand

In a web font, I had one glyph where the background and foreground was reversed. Being Mr.Pragmatic, I just went with it, setting foreground and background colours appropriately. But it is an RWD site, and at certain scalings a 1 pixel white line was appearing. Here is how I fixed it.

I opened it in a text editor. What I had was:

<svg ... x="0px" y="0px" width="71px" height="67px" viewBox="0 0 71 67" enable-background="new 0 0 71 67">
<rect fill="#000000" width="71" height="67"/>
<g>
 <path fill="#FFFFFF" d="M49.562,...,31.799z"/>
 <path fill="#FFFFFF" d="M46,...z"/>
</g>
</svg>

The fix was as simple as deleting that <rect> line, and then setting the fill colour of the two <path> tags to be #000000.

When I made the webfont (I use the excellent icomoon.io site to do this) it looked fine. But it looked smaller than the other glyphs in the font. I can add padding easily with CSS, but removing it can be more work.

Conclusion: I don’t know how to do this. I could play around with the viewBox="..." in the <svg> tag, and change the appearance in Inkscape, but it made no difference in icomoon. Similarly, I could select the whole path, scale it in Inkscape, but still no change in icomoon. So, having done the important fix, I gave up on this one (and will stick to controlling the size and padding from CSS.)

Annoying keyboard shortcuts in Xfce

Quickie for mint16/17 using Xfce. You may have found the ctrl-Fn keys have been mapped to change the workspace you are on? This behaviour is highly annoying as applications often define ctrl-Fn to do something.

Do not try to fix it from the graphic configuration window. It does not work.

Instead, as root, open this file in a text editor:

/etc/xdg/xdg-default/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml

(Because or links there are about three or four files pointing to this file; it doesn’t matter which one you edit.)

Look for lines like:

<property name="&lt;Control&gt;F5" type="string" value="..."/>

Then delete all those lines, there are 12 in total, from F1 to F12. (In my case they were not in a block, so you had to hunt around to catch all twelve.)

Then you need to logout of your desktop, and log back in again for the change to take effect.

Written with StackEdit.

C++ Logging: EasyLogging++

The Basics Of EasyLogging++

I wanted a basic logging library for C++. The first one I looked at required I first install Java to be able to compile it. Eh? For a modern C++ library? So then I added “header-only” to be key requirements. And shortly after that I sadly started to resign myself to writing my own. Then, luckily, I stumbled across EasyLogging++.

I retrofitted a couple of projects today, stripping out the ad hoc logging, and replacing it with this. The documentation is good, but gets lost in the details some times, and I felt a simpler tutorial was needed. This is my attempt at it:

Here is the Hello World example:

#define _ELPP_THREAD_SAFE
#include "easylogging++.h"
_INITIALIZE_EASYLOGGINGPP

int main(int,char**){
LOG(INFO)<<"Hello World!";
}

I’ve decided to request it be thread-safe, right from this first example, because most C++11 apps use threads. Remove that line if you definitely have a single-threaded application. (Speaking of which, this library is C++11 only; but there is a link on their website to an earlier version that supports older C++).

The above program prints “Hello World!” to stdout. But, obviously, you want to log to a file. And for a project of any worthwhile size you will end up with a logging configuration file anyway, so let’s add one now. Here is “logging.conf”:

-- default 

* GLOBAL:
    TO_FILE = true
    FILENAME = "info.%datetime{%Y%M%d}.log"
    TO_STANDARD_OUTPUT   =  false
* WARNING:
    TO_STANDARD_OUTPUT   =  true
* ERROR:
    TO_STANDARD_OUTPUT   =  true
* FATAL:
    TO_STANDARD_OUTPUT   =  true

Here I am saying I want to have one log file per day, using the YYYYMMMDD datestamp in the log filename, and that it will store messages of all log levels. I’m also saying that I want TRACE and INFO messages to only go to the file, but WARNING, ERROR and FATAL to go to both the file and stdout. There may be more elegant ways to do that, but the above works.

You use the config file by adding just one line at the start of main():

#define _ELPP_THREAD_SAFE
#include "easylogging++.h"
_INITIALIZE_EASYLOGGINGPP

int main(int,char**){
el::Loggers::configureFromGlobal("logging.conf");
LOG(INFO)<<"Hello World!";
}

That code will write “Hello World\n” to e.g. “info.20140827.log”
 
And that is all you need to know; all your other questions will be answered by the documentation. Do please spend some time with the documentation as there is a lot of functionality in this library (e.g. Conditional logging, Occasional Logging, log output of STL containers, log output for your own classes, datestamps to customizable sub-second accuracy, run-time disabling of certain log levels, and even more.)

One feature it does not have, that I wanted, is an asynchronous log queue. I.e. a thread grabs the lock just long enough to push a string on to a queue, with another dedicated thread doing the actual writing to disk. This makes sure your worker threads do not get caught up waiting for a lock because another thread is waiting for disk I/O to finish. However another feature, that EasyLogging++ does have, lessens the impact of this: it only flushes to disk every N log messages. So effectively strings are being pushed to a queue, and it is only once every N times that a thread gets caught waiting for disk I/O to finish. N defaults to 256; I reduced it in my config file to 5, which gives a fair balance between thread wait and log latency (and the risk of losing log messsages). This is not as good as an asynchronous log queue, but I can live with it.

Written with StackEdit.

Using C++11 std::future to push data from producer to multiple consumers

This is an example of using C++11's std::future to move data from a data producer to multiple consumer threads, in a very stable and thread-safe way. And hopefully it is at least as efficient as the alternatives:

http://stackoverflow.com/a/25339704/841830

Critiques of my approach (or of the alternatives) are very welcome.

QR Codes in Handlebars

User’s don’t like to type in long URLs, and if they are on mobile they don’t like to type at all. Here is a simple Handlebars helper to make QR codes. But first, here is how easy it is to use:

  {{{qrcode "http://darrendev.blogspot.com/"}}}

Why would you want a QR code in a web site, when you could just as well put a clickable link? One example is to move a URL from viewing it on a desktop to your mobile. Another example is for ticketing: you put a booking code in the URL, make a QR code from it, and the venue will scan it (and the URL takes them directly to the validation function of their web app). No typing. Quick, safe and easy for everyone involved.
Handlebars is a JavaScript templating engine, easy to use and understand, and it allows expanding it with your own plugins, which thankfully are equally easy to write. Here is the above plugin, in full:

Handlebars.registerHelper("qrcode", function(data) {
var qrcode  = new QRCode(-1,QRErrorCorrectLevel.H);
qrcode.addData(data);
qrcode.make();

var bg="#fff",fg="#000";
var n=qrcode.getModuleCount();
var tilesize = 4;
var sz=n*tilesize;

var s='<table style="width:'+sz+'px;height:'+sz+
  'px;border:0;border-collapse:collapse;background:'+
  bg+'>';
for(var row = 0; row < n; row++ ){
    s+='<tr style="height:'+tilesize+'px">';
    for(var col = 0; col < n; col++ ){
        s+='<td style="background:'+
          (qrcode.isDark(row, col) ? fg:bg)+
          ';width:'+tilesize+'px"></td>';
        }
    s+='</tr>';
    }
return s;
});

To use this plugin you need to fetch an open-source JavaScript library unzip it and include it on your site with:

<script type="text/javascript" src="qrcode.min.js">

Aside: the core of the above plugin was taken from one of the examples that come with the library. I chose to use tables, rather than canvas, as it gives much wider browser support, with no disadvantage.

Written with StackEdit.

Casper, d3, jquery and clicking

I spent (wasted? invested?) an awful lot of time yesterday on trying to use CasperJS to click a link inside an SVG diagram that had been made by d3.js. I’ll start this article by showing my Foolish Mistake, but then will document what I did learn.

Foolishness

Normally, in CasperJS, we write this.click('#myButton'), where “#myButton” can be just about any CSS selector.

Aside: it is normally this.click() rather than casper.click()
because it is normally in the handler function of a casper.waitXXX()
function.

This wasn’t working for me trying to click an SVG <g> tag, that starts a search running. I was taking a screenshot 0.5s later to check it had worked, and it showed the page had not changed.
And that turned out to be foolish. The click was working, the search was working, d3 and SVG had nothing to do with anything. The problem was simply were no search results found, and so it never moved to the next page. When I changed the search string, suddenly click() was working.
So my troubleshooting turned out to be barking up the wrong tree. But, still, I did learn a few things from it.

Using evaluate() With jQuery

Instead of calling this.click() you can do something like this:

this.evaluate(function(){
  $('#myButton').click();
  };

This runs JavaScript from within the browser’s context. In this case I use jQuery. This works just as well as using Casper’s click() outside the evaluate().
Here there is no advantage. But, by being in a different scope, we have extra flexibility: we could call other functions, or add new event handlers, etc, etc.

Using evaluate() With d3

Here was another of my attempts, but this one does not work:

this.evaluate(function(){
  d3.select('#myButton').click();
  };

The reason: a d3 selection does not have a ready-made click() function.

Making events happen

this.evaluate(function(){
  var evt = document.createEvent("MouseEvents");
  evt.initMouseEvent("click", true, true, window,
    0, 0, 0, 0, 0, false, false, false, false, 0, null);
  return d3.select('#myButton').node().dispatchEvent(evt);
  };

This is how you do a click with d3 (you could use this approach with jQuery too (see a helper function), but are unlikely to ever need to). The first lines create a (simulated) mouse click. The final line sends that event to the DOM item of interest.

Aside: dispatchEvent() returns false if any of the event handlers
cancelled it, true otherwise.

I learned this here; that answer also says this should have worked:

this.evaluate(function(){
  d3.select('#myButton').on("click")()
  };

This definitely does not work for me. Why? It is actually a cheat, trying to find and call the click handler for the button. In my case the click handler was attached to the parent object (a <g>) not the object I was calling select on. (Also, because it is a cheat, this approach does not work when you’ve attached multiple handlers.)
By the way, if not using jQuery or d3, you can use querySelector() and do it this way:

this.evaluate(function(){
  var evt = document.createEvent("MouseEvents");
  evt.initMouseEvent("click", true, true, window,
    0, 0, 0, 0, 0, false, false, false, false, 0, null);
  return document.querySelector('#myButton').dispatchEvent(evt);
  };

Summary

The real lesson here, for me, was when something doesn’t work, make sure you are judging success in the right way!
Beyond that, it turns out there are a whole host of ways to click a button in a CasperJS script. Use the simplest when you can, bear the others in mind for special occasions.

Written with StackEdit.

Captcha never changed (securimage)

It took a while of testing before I noticed, then I finally realized the captcha on a web app I was working never changed!

Thankfully, before launching into some heavy-duty troubleshooting, I had a flash of inspiration: to make my unit tests work nicely I was setting the random seed to a constant (i.e. to get repeatable test data). That test data is actually being generated for every page request of the main web app (just during development).

So the fix was as trivial as calling mt_srand(time());  just before creating the Captcha  (as I'm using the Securimage library, just before the $img = new Securimage(); call).

If only all troubleshooting was this quick!

Iterate a map in added order, not key order, using C++11

I’m loving C++11. Most of the things only save you a few keystrokes. But C++ was a verbose language, and those saved keystrokes mean the code becomes more readable and more maintainable.
Here is a quick example refactor. I had some key-value pairs, and was using a std::map (I’ve shown the value type as T to avoid cluttering this example):

std::map< std::string, T> data;
...
data [ key ] = value;
...
for(const auto &entry : data){
    std::cout<<"Label:" << entry.first << "\n";
    std::cout<< entry.second.something << "\n";
    }

You can see I’m already using a C++11-ism, with the range-based for loop, and auto. That is so much nicer that I’m not even going to show you the old way it would’ve been done.
The problem was I didn’t want to iterate through the map in key order, I wanted to iterate through in the order the items was added. My first idea was to switch to using two vectors, one for the keys, one for the values. But parallel data structures are a nasty code smell.
Then I thought of using std::pair. And the above code became this:

std::vector< std::pair<std::string, T> > data;
...
data.push_back( {key,value} );
...
(as above)

First thing to observe is that even though the data structure is now completely different, my for(){...} loop is completely the same. This was partly luck: a std::map iterator has first and second elements, and so does a std::pair. But we also need to give a big thanks to auto: it saved us having to even think about what the new iterator type is.
Did you spot the second C++11-ism? It slipped by so naturally it might not even have registered. Here it is: {key,value}
In earlier versions of C++ I would have had to write:

data.push_back( std::make_pair(key,value) );

Only 14 keystrokes saved, but those 14 were pure distraction, and won’t be missed.
(This refactor is the correct approach in older versions of C++ too, by the way. All C++11 brings to the party is that it is now easier to do.)

Written with StackEdit.

PHP Date vs. gettimeofday

I just had a lot of trouble with a unit test failing. Unfortunately lots of things had changed: new computer, new linux distro, new version of PHP (from 5.3 to 5.5), new version of PHPUnit (from 3.4 all the way to 4.0 then back to 3.7), etc.

I eventually tracked it down; and made this minimal example:

for($n=0;$n<250;++$n){
        $t = time();
        $s = date("Y-m-d H:i:s");
        $tod = gettimeofday();
        echo "$t,$s,{$tod['sec']},{$tod['usec']},".
          date("Y-m-d H:i:s")."\n";
    usleep(10000);  //0.01s
    }

It is getting the time 100 times a second, and comparing with time(), date() and gettimeofday(). I’ll just show the interesting bit:

1398601641,2014-04-27 21:27:21,1398601641,972424,2014-04-27 21:27:21
1398601641,2014-04-27 21:27:21,1398601641,982545,2014-04-27 21:27:21
1398601641,2014-04-27 21:27:21,1398601641,992669,2014-04-27 21:27:21
1398601641,2014-04-27 21:27:21,**1398601642**,2793,2014-04-27 21:27:21
1398601642,2014-04-27 21:27:22,1398601642,12919,2014-04-27 21:27:22
1398601642,2014-04-27 21:27:22,1398601642,23041,2014-04-27 21:27:22

I’ve highlighted the problem point. It appears gettimeofday() is slightly ahead of time() (and date() is always consistent with time()). More precisely, for the first 10000 microseconds of each second, time() is still in the previous second.

My unit test was watching gettimeofday() to wait for a new second to roll by. But the code being tested was using date to get a timestamp. (That timestamp was then used as a unique ID for a database insert, and the unit test asserted it was always an insert, not an update!)

Fascinating stuff, eh? I’ve no idea if this is a behaviour change in PHP, or if this new machine is simply a bit quicker, and the problem never cropped up before. However I doubt the latter as the unit test now fails 100% of the time, but never failed this way before.

Oh, the fix? Simply changed the calls to gettimeofday() to time(). To be honest I’ve no idea why I used the more complicated function in the first place.

Using RewriteRule instead of Alias in .htaccess

Subtitle: Why aren't Apache rewrites working?!?!?!?!

I had a few directories (e.g. each a mini website or webapp) sharing a fonts directory, and I was doing this with symlinks. But symlinks are a pain to rsync, so I thought let's use Apache to do the symlinking, i.e. use Alias. I put this in .htaccess:

  Alias /mysite/fonts/ ../shared/fonts/

500 Internal Server Error.

Or, in other words, Alias cannot be used in .htaccess, only in httpd.conf. I could've done it that way (added the Alias to httpd.conf) but as I was already using RewriteRule in the .htaccess file I decided to persevere finding a way to do it using just the .htaccess file.

This was my first attempt:

   RewriteRule ^/mysite/fonts/(.*)$ /shared/fonts/$1  [L]

Completely ignored. Now, when you start googling, or searching on StackOverflow, for "Apache rewrite does not work", 99% of the hits will tell you how to turn it on. I.e. you need AllowOverride All in your httpd.conf, which allows you to use .htaccess files, and you need Options FollowSymLinks so that the rewrite engine will work. And you need RewriteEngine On in your .htaccess file.


Unfortunately I'd done all those things. I knew the rewrite engine was working, as I had other rules in this same .htaccess file, and commenting them out changed behaviour. My problem was that my new rewrite rule was just being ignored, even though it looked correct.


I'll spare you the pain of seeing the 187 other experiments I tried. The forehead-slap moment came when I realized the first parameter of the rewrite rule is relative to the directory the .htaccess file is in. I.e. my .htaccess file is in the /mysite/ directory. And, thus, the correct rule turned out to be:

   RewriteRule ^fonts/(.*)$ /shared/fonts/$1  [L]

Note the lack of leading forward slash. Very important. Incidentally, the second parameter is relative to the server root if you start it with a forward slash! In other words, it is only the first parameter that doesn't work that way.

The [L] at the end of the RewriteRule means "last", and is normally what you want for this kind of rewrite rule. Use [R] ("redirect") instead if you wanted the URL to be rewritten (i.e. an HTTP redirect to be sent back). Useful troubleshooting tip: if you set the flag to [F] then you can immediately see if the first parameter to the rule is being recognized: if it is getting matched, you get a 403 (and if you don't then you have to fix the first parameter). If you get a 403 with an [F], but don't get the expected results when you change the the [F] to an [L] then it is the second parameter you need to fix.

So, to summarize, if your .htaccess file is in a subdirectory, remember that the RewriteRule has to be relative to that subdirectory.

Clearing cookies between CasperJS tests

I spent absolutely ages searching for how to do this, and it turns out to be really easy: phantom.clearCookies();

One page I found suggested casper.clear() would do it. I'm not fully sure what it does clear, but cookies are not covered.

In context here is what it looks like:

  var username = ..., password = ...;
  (function f(label,url){    casper.test.begin(label, {
      test:function(test){ runTheTests(test,label,url,username,password); }
    });
  })("first", "http://127.0.0.1/first");

  (function f(label,url){
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,label,url,username,password); }
    });
  })("second", "http://127.0.0.1/second");
     function runTheTests(test,label,url,username,password){
  casper.start();
  casper.clear();
  phantom.clearCookies();
  casper.thenOpen(url,function(){
    console.log('Loaded:'+url);
    });
 
  ...Other actions and tests here...

  casper.then(function(){
    test.done();
    });

  casper.run();
  }

I.e. first test does a successful login. Second test was supposed to then see a different login page, but in fact it was still logged-in. Adding phantom.clearCookies() did the trick.