Showing posts with label javascript. Show all posts
Showing posts with label javascript. Show all posts

Friday, March 31, 2017

The Seven Day A Year Bug

I’ll cut straight to the chase: when you use d.setMonth(m - 1) in JavaScript, always set the optional second parameter.

What’s that, you didn’t know there was one? Neither did I until earlier today. It allows you to set the date. Cute, I thought at the time, minor time-saver, but hardly worth complicating an API for.

Ooh, how wrong I was. Let me take you back to when it happened. Friday, March 31st….

After a long coding session, I did a check-in. And then ran all unit tests. That is strange, three failing, but in code I hadn’t touched all day. I peer at the code, but it looks correct - it was to do with dates, specifically months, and I was correctly subtracting 1.

Aside: JavaScript dates inherit C’s approach of counting months from 0. In the first draft of this blog post I used a more judgemental phrase than “approach”. But to be fair, it was a 1970s design decision, and the world was different back then. Google “1970s men fashion”.

So, back to the test failures. I start using “git checkout xxxx” to go back to earlier versions, to see exactly when it broke. Running all tests every time. I know something fishy is going on, by the time I’ve gone back 10 days and the tests still fail. I am fairly sure I ran all tests yesterday, and I am certain it hasn’t been 10 days.

Timezones?! Unlikely, but we did put the clocks back last weekend. But a quick test refutes that. (TZ=XXX mocha . will run your unit tests in timezone XXX.)

So, out of ideas, I litter the failing code with console.log lines, to find out what is going on.

Here is what is happening. I initialize a Date object to the current date (to set the current year), then call setMonth(). I don’t use the day, so don’t explicitly set it. I was calling setMonth(8), expecting to see “September”, but the unit test was being given “October”. Where it gets interesting is that the default date today is March 31st. In other words, when I set month to September the date object becomes “September 31st”, which isn’t allowed. So it automatically changes it to October 1st.

You hopefully see where the title of this piece comes from now? If I was setting a date in February I would have discovered the bug two days earlier, and if my unit test had chose October instead of September, the bug would never have been detected. If I’d thought, “ah, I’ll run them Monday”, the bug would not have been discovered until someone used the code in production on May 31st. I’d have processed their bug report on June 1st and told them, “can’t reproduce it”. And they’d have gone, “Oh, you’re right, neither can I now.”

To conclude with a happy ending, I changed all occurrences of d.setMonth(m - 1) into d.setMonth(m-1, 1), and the test failures all went away. I also changed all occurrences of d.setMonth(m-1);d.setDate(v) (where v is the day of the month) into: d.setMonth(m-1, v) not because it is shorter and I can impress people with my knowledge of JavaScript API calls, but because two separate calls was a bug that I simply didn’t have a unit test for.

But writing that unit test can wait until Monday.

Friday, March 18, 2016

Timestamp helper in Handlebars

Handlebars is a widely-used templating language for web pages. In a nutshell, the variables to insert go between {{ and }}. Easy. It offers a few bits of logic, such as if/else clauses, for-each loops, etc. But, just as usefully, Handlebars allows you to add helper functions of your own.
In this article I will show a nice little Handlebars helper to format datestamps and timestamps. Its raison d’etre is its support for multiple languages and timezones. The simplest use case (assuming birthday is their birthday in some common format):
<p>Your birthday is on {{timestamp birthday}}.</p>
It builds on top of sugar.js’s Date enhancements; I was going to do this article without using them, to keep it focused, but that would have made it unreasonably complex.
There are two ways to configure it: with global variables, or with per-tag options. For most applications, setting the globals once will be best. Here are the globals it expects to find:
  • tzOffset: the number of seconds your timezone is ahead of UTC. E.g. if in Japan, then tzOffset = 9*3600. If in the U.K. this is either 0 or 3600 depending on if it is summer time or not.
  • lang: The user-interface language, e.g. “en” for English, “ja” for Japanese, etc.
(By the way, if setting lang to something other than “en”, you will also need to have included locale support into sugar.js for the languages you are supporting - this is easy, see the sugar.js customize page, and check Date Locales.)
The default timestamp format is the one built-in to sugar.js for your specified language. All these configuration options (the two above, and format) can be overridden when using the tag. E.g. if start is the timestamp of when an online event starts, you could write:
<p>The live streaming will start at
{{timestamp start tzOffset=0}} UTC,
which is {{timestamp start tzOffset=32400}}
in Tokyo and {{timestamp start tzOffset=-25200}}
in San Francisco.</p>
Here is the basic version:
Handlebars.registerHelper('timestamp', function(t, options){
var offset = options.hash.tzOffset;
if(!offset)offset = tzOffset;

if(!Object.isDate(t)){
    if(!t)return "";
    if(Object.isString(t))t = Date.create(t + "+0000").setUTC(true).addSeconds(offset);
    else t = Date.create(t*1000).setUTC(true).addSeconds(offset);
    }
else t = t.clone().addSeconds(offset);

if(!t.isValid())return "";

var code = options.hash.lang;
if(!code)code = lang;   //Use global as default

var format = options.hash.format ? options.hash.format : '';
return t.format(format, lang);
});
The first two-thirds of the function turn t into a Date object, coping whether it was already a Date object, or a string (in UTC, and in any common format the Date.create() can cope with), or a number (in which case it is seconds since Jan 1st 1970 UTC). However, be careful if giving a pre-made Date object: make sure it was the time in UTC and specifies that is in UTC.
The rest of the function just chooses the language and format, and returns the formatted date string.
If you were paying attention you would have noticed t stores a lie. E.g. for 5pm BST, t would be given as 4pm UTC. We then turn it into a date that claims to be 5pm UTC. Basically this is to stop format() being too clever, and adjusting for local browser time. (This trick is so you can show a date in a browser for something other than the user’s local timezone.)
But it does mean that if you include any of the timezone specifiers in your format string, they will wrongly claim it is UTC. {{timestamp
theDeadline format="{HH}:{mm} {tz}" }} will output 17:00 +0000.
To allow you to explicitly specify the true timezone, here is an enhanced version:
Handlebars.registerHelper('timestamp', function(t, options){
var offset = options.hash.tzOffset;
if(!offset)offset = tzOffset;   //Use global as default
if(!Object.isDate(t)){
    if(!t)return "";
    if(Object.isString(t))t = Date.create(t + "+0000").setUTC(true).addSeconds(offset);
    else t = Date.create(t*1000).setUTC(true).addSeconds(offset);
    }
else t = t.clone().addSeconds(offset);
if(!t.isValid())return "";

var code = options.hash.lang;
if(!code)code = lang;   //Use global as default

var format = options.hash.format ? options.hash.format : '';
var s = t.format(format, lang);
if(options.hash.appendTZ)s+=tzString;
if(options.hash.append)s+=options.hash.append;
return s;
});
(the only change is to add a couple of lines near the end)
Now if you specify appendTZ=true then it will append the global tzString. Alternatively you can append any text you want by specifying append. So, our earlier example becomes one of these:
{{timestamp theDeadline format="{HH}:{mm}" appendTZ=true}}
{{timestamp theDeadline format="{HH}:{mm}" append="BST"}}
{{timestamp theDeadline format="{HH}:{mm}" append=theDeadlineTimezone}}
The first one assumes a global tzString is set. The second one hard-codes the timezone, which is unlikely to be the case; the third one is the same idea but getting timezone from another variable.
VERSION INFO: The above code is for sugar.js v.1.5.0, which is the latest version at the time of writing, and likely to be so for a while. If you need it for sugar.js 1.4.x then please change all occurrences of setUTC(true) to utc().

Wednesday, March 9, 2016

Comparison Of Three WebGL Libraries

Comparison Of Three WebGL Libraries

For many people, WebGL is a technology for making browser-based games, but I am more interested in all the other uses: data visualization, data presentation, making web sites look fantastic, new and interesting user experience (UX), etc. (I have spent many years using Flash for similar things.)

What is WebGL?

WebGL is an API to allow browsers to use a GPU to speed up 2D and 3D graphics; you write in a mix of JavaScript and a shader language. Because it is low-level and complex I recommend against writing in raw WebGL; use a library instead.

It is supported on just about any popular OS/browser combination, including working on tablets and mobile phones. Your device does not need to have a dedicated GPU to run WebGL.

What libraries are there?

There are actually quite a few choices, but for this article I will focus on the three libraries I have made (non-trivial) WebGL applications with:
The first two are fairly low-level (Babylon.JS has a few more abstractions built-in), meaning you will be thinking in terms of vertices, faces, 3D coordinates, cameras, lighting, etc. A 3D graphics background will be useful. Superpowers is higher-level, but more focused on games development. Some Blender (or equivalent) skills will also come in handy, whichever library you go for.

Three.js And Its Resources


Three.JS is the most established WebGL library, with some published books, many demos (http://threejs.org/, https://stemkoski.github.io/Three.js/ and others), even a Udacity course.

However it has scant regard for backwards compatibility, meaning that frequently the code in the published books (or the source code of older demos and tutorials) will not work with the latest library version. It has a relatively aggressive developer community, who think that having an uncommented demo of a feature counts as documentation.

It uses the MIT license (the most liberal open source - fine for commercial use), hosted on github; bug reports to github, but support questions to StackOverflow’s [three.js] tag.

Babylon.js And Its Resources

Babylon.JS is now two years old, and was developed at Microsoft in France, though it is open source (Apache license, so fine for commercial use). It is primarily intended for making games, but is flexible enough for other work.

Like Three.JS, it has plenty of demos, and again they are often undocumented. There is an active web forum; explanations and experiments there often link to the Babylon Playground, which is a live coding editor. There is also a very useful eight hour training video course (free), presented by the two David’s who created Babylon.JS. (There is a just-released book, https://www.packtpub.com/game-development/babylonjs-essentials, but I’ve not seen it, so cannot comment.)

Superpowers And Its Resources

Superpowers is a bit different: it is a gaming system, with its own IDE. It is very new, only released as open source (ISC license, which is basically the nice liberal MIT license again) in the middle of January 2016, though appears to have a year’s closed development behind it. (The IDE is cross-platform; it has been running nicely for me on Linux, I’ve not tried it on other platforms.)

Some of the initial batch of demos and games have been released on GitHub (kind of as open-source - the licenses are a bit vague, especially regarding re-use of assets), which has been my main source of learning. A few tutorials have also appeared recently (GameFromScratch.com, and https://itch.io/board/11494/tutorials-guides).

What grabbed my attention was the quality and completeness of the Fat Kevin game, combined with the fact that I could download all source and assets for it, to learn from. (The Discover Superpowers demo is similar, but simpler, so easier to learn from.)

Support is through forums on itch.io, with separate English and French sections. This requires yet another user account; I find it a shame they didn’t use StackOverflow, Github, or at least HTML5 Game Devs (as Babylon did). I’d not heard of itch.io (“an open marketplace for independent digital creators with a focus on independent video games”) before, but I think their choice tells you how they see Superpowers being used.

The coding language is TypeScript, basically JavaScript 1.8 plus types; it is worth specifying those types, as then the IDE’s helpful autocomplete can work. Note that Superpowers is closely tied to the IDE - you need to be clicking and dragging things; doing everything in code is not realistic (though this might just be the style of the initial few games). Superpowers is built on Three.JS, but I’m not seeing anything exposed, so I don’t think you can take a Three.JS example and use it directly.

Conclusion

Which library to choose? I suggest you try out the demos for each of these, and choose the library that has demos that cover all the things you want to do. If the choice comes down to Three.JS vs. Babylon.JS, and you cannot find a killer reason to choose one over the other, this is because it doesn’t really matter, they can each do 95+% of what the other can: follow your hunch, choose one or the other, and dive in and learn it.

Finally I should say that WebGL for website development is hard: your programmer(s) will need 3D experience, as will your graphic designer(s). If you are using RWD/mobile-first to target both mobile and desktop, it is even more complex. My company, QQ Trend Ltd. can help (contact me at dc [at] qqtrend.com).

Monday, March 7, 2016

Gradients in Three.JS

Gradients in Three.js

3D charts with gradientsMany years ago, I made some charts in Flash: 3D histograms using boxes and pyramids. More as proof of concept than anything else, I used two types of gradients on the sides:
  • Colour gradients (e.g. red at the bottom of the bar, gradually changing to be yellow at the top)
  • Opacity gradients (e.g. solid at the bottom, gradually changing to be only 20% opaque at the top)
Recently I’ve been trying to reproduce (and go beyond) those charts in WebGL. Gradients seem to be both harder, and less flexible, than they were in ActionScript/Flash.

I’ve been working with two libraries, Three.JS and Babylon.JS. In Babylon.JS I couldn’t find any examples of how to do either type of gradient. In Three.JS I believe there is no support for opacity gradients, but colour gradients are possible, and that will be the theme of this article.

Three.js: mesh, geometry, material

I will assume some familiarity with WebGL and Three.JS concepts, but the most essential knowledge you need to follow-on with this article is:
  • geometry is a shape.
  • material is the appearance
  • mesh is a geometry plus a material.
Most of the time your geometry and your material are orthogonal. E.g. if you have a red shiny material you can apply it equally easily to your pyramid or your torus. And you can just as easily tile a grass image on either of those shapes.

A more tightly coupled (less orthogonal) example is a game character (a mesh) you have made in, say, Blender, with a special texture map (a material) to give it a face and clothes. The mesh and the material are basically tied together. However, if the mesh comes with multiple poses, or animations, the same texture map works for all of them. And you can repaint the texture map to give your mesh (in any of its poses) new clothes.

In contrast, gradients are highly coupled; at least in the way I will show you here. Like coupling in software engineering, this is bad: I cannot prepare a red-to-yellow gradient material, and then apply it to any mesh; instead I have to embed the gradient description into that mesh, in a way specific to that mesh.

Vertex Colours

The way it works is you can switch a material to use VertexColors. E.g.
  var mat = new THREE.MeshPhongMaterial({vertexColors:THREE.VertexColors});
And then over in the mesh you specify a colour for each vertex. If you do this, then Three.JS will, for each triangle, blend the vertex colours in a smooth gradient. All faces in Three.JS are triangles, and vertices are referenced through each face, so you end up with lots of lines like this:
  myGeo.faces[ix].vertexColors = [c1, c2, c3];
where each of c1, c2 and c3 are THREE.Color instances.

By the way, I said opacity gradients were not possible with this technique (because vertexColors takes an RGB triplet, not RGBA), but it is still possible to make the whole mesh semi-transparent.

BoxGeometry Vertices

A THREE.BoxGeometry is used to make a 6-faced cuboid shape. To be more precise it is a shape made up of 12 triangles (two on each face). To be able to set vertexColors you need to know the order of the those 12 triangles. I reverse-engineered it, by colouring each in turn, to get the following list:
  • 0: top-left of one of the side faces. 0 is top-left, 1 is bottom-left, 2 is top-right (anti-clockwise)
  • 1: bottom-right. vertex 0 is bottom-left, 1 is bottom-right, 2 is top-right. (anti-clockwise)
  • 2/3: same for opposite side
  • 4/5: is top. (4’s vertex zero touches 2’s vertex zero)
  • 6/7 is bottom
  • 8/9 is one side
  • 10/11 is the other side

A Factory Function

Here is a factory function to make a box mesh with colour c1 on the base, color c2 on the top, and each side having a smooth linear gradient from c1 at the bottom to c2 on the top.

You can specify c1 and c2 as either a hex code (e.g. 0xff0000) or as a THREE.Color object. w, d, h are the three dimensions of the cube. opacity is optional, and can be from 0.0 (invisible) to 1.0 (full opaque - the default).

function makeGradientCube(c1, c2, w, d, h, opacity){
if(typeof opacity === 'undefined')opacity = 1.0;
if(typeof c1 === 'number')c1 = new THREE.Color( c1 );
if(typeof c2 === 'number')c2 = new THREE.Color( c2 );

var cubeGeometry = new THREE.BoxGeometry(w, h, d);

var cubeMaterial = new THREE.MeshPhongMaterial({
    vertexColors:THREE.VertexColors
    });

if(opacity < 1.0){
    cubeMaterial.opacity = opacity;
    cubeMaterial.transparent = true;
    }

for(var ix=0;ix<12;++ix){
    if(ix==4 || ix==5){ //Top edge, all c2
        cubeGeometry.faces[ix].vertexColors = [c2,c2,c2];
        }
    else if(ix==6 || ix==7){ //Bottom edge, all c1
        cubeGeometry.faces[ix].vertexColors = [c1,c1,c1];
        }
    else if(ix%2 ==0){ //First triangle on each side edge
        cubeGeometry.faces[ix].vertexColors = [c2,c1,c2];
        }
    else{ //Second triangle on each side edge
        cubeGeometry.faces[ix].vertexColors = [c1,c1,c2];
        }
    }

return new THREE.Mesh(cubeGeometry, cubeMaterial);
}
 
Given the earlier explanation, I hope the code is self-explanatory: make a material where all we set is that we will use VertexColors (and, optionally, that it is partially transparent), make a box mesh, and then go through all 12 faces, work out which face it is, and set the colours of the three corners accordingly.

A Full Example

Here is a complete example (you’ll need to paste in the above code, where shown), to quickly demonstrate that it works. (This was tested with r74, but as far as I know it should work back to at least r65.)

The code is minimal: make a scene, with a camera and a light, and put a gradient box (2x3 units at the base, 6 units high) at the centre of the scene. The gradient goes from red to a pale yellow. I made it slightly transparent (the 0.8 for the final parameter), but as it is the only object in the scene this has no effect (except to dim the colours a bit, because of the black background)!

<!DOCTYPE html>
<html>
<head>
  <title>Gradient test</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r74/three.min.js"></script>
</head>
<script>
function makeGradientCube(c1, c2, w, d, h, opacity){/*As above*/}

function init() {
  var scene = new THREE.Scene();
  var camera = new THREE.PerspectiveCamera(45,
    window.innerWidth / window.innerHeight, 0.1, 1000);

  var renderer = new THREE.WebGLRenderer();
  renderer.setClearColor(0x000000, 1.0);
  renderer.setSize(window.innerWidth, window.innerHeight);

  var dirLight = new THREE.DirectionalLight();
  dirLight.position.set(30, 10, 20);
  scene.add(dirLight);

  scene.add( makeGradientCube(0xff0000, 0xffff66, 2,3,6, 0.8) );

  camera.position.set(10,10,10);
  camera.lookAt(scene.position);

  document.body.appendChild(renderer.domElement);
  renderer.render(scene, camera);
  }

window.onload = init;
</script>
<body>
</body>
</html>


Sources

The above code was based on studying this example and trying to work out how it was doing that. It is undocumented - par for the course with Three.JS examples, sadly. I also peeked at the Three.JS source code. If you want more undocumented code examples of using THREE.VertexColors, see https://stemkoski.github.io/Three.js/Vertex-Colors.html

Future Work

First, if you write your own shaders I believe anything and everything is possible.

Second, I wonder about making a gradient in a 2D canvas, and using that as a texture map. And/or using it as the alpha map to create an opacity gradient.

Either of those may be the subject of a future article. In the meantime, if you know a good tutorial on using gradients in either Babylon.JS or Three.JS, please link to it in the comments. Thanks, and thanks for reading!

Thursday, September 3, 2015

Format Japanese date with kanji day-of-week

In Japanese, there are single kanji for each day of the week.
var days = ['日','月','火','水','木','金','土'];
(If you want to mutter them under your breath, at work, to impress colleagues, nichi-getsu-ka-sui-moku-kin-do.)
In JavaScript, to put them in a date use days[d.getDay()] (where d is a Date object).
I use sugar.js, which adds a format() function (amongst loads of other useful stuff) to the Date class; I now extend it further with this:
Date.prototype.format_ja_MMDDK = function(){
var days = ['日','月','火','水','木','金','土'];
return this.format("{MM}月{dd}日") + "(" + days[this.getDay()] + ")";
};
(If you hate underlines feel free to use `formatJaMMDDK() or anything you like, for the matter.)
Here is one way you might use it (jQuery-syntax):
$('.todaysDate').html(new Date().format_ja_MMDDK());

Monday, June 30, 2014

QR Codes in Handlebars

User’s don’t like to type in long URLs, and if they are on mobile they don’t like to type at all. Here is a simple Handlebars helper to make QR codes. But first, here is how easy it is to use:
  {{{qrcode "http://darrendev.blogspot.com/"}}}
Why would you want a QR code in a web site, when you could just as well put a clickable link? One example is to move a URL from viewing it on a desktop to your mobile. Another example is for ticketing: you put a booking code in the URL, make a QR code from it, and the venue will scan it (and the URL takes them directly to the validation function of their web app). No typing. Quick, safe and easy for everyone involved.
Handlebars is a JavaScript templating engine, easy to use and understand, and it allows expanding it with your own plugins, which thankfully are equally easy to write. Here is the above plugin, in full:
Handlebars.registerHelper("qrcode", function(data) {
var qrcode  = new QRCode(-1,QRErrorCorrectLevel.H);
qrcode.addData(data);
qrcode.make();

var bg="#fff",fg="#000";
var n=qrcode.getModuleCount();
var tilesize = 4;
var sz=n*tilesize;

var s='<table style="width:'+sz+'px;height:'+sz+
  'px;border:0;border-collapse:collapse;background:'+
  bg+'>';
for(var row = 0; row < n; row++ ){
    s+='<tr style="height:'+tilesize+'px">';
    for(var col = 0; col < n; col++ ){
        s+='<td style="background:'+
          (qrcode.isDark(row, col) ? fg:bg)+
          ';width:'+tilesize+'px"></td>';
        }
    s+='</tr>';
    }
return s;
});
To use this plugin you need to fetch an open-source JavaScript library unzip it and include it on your site with:
<script type="text/javascript" src="qrcode.min.js">
Aside: the core of the above plugin was taken from one of the examples that come with the library. I chose to use tables, rather than canvas, as it gives much wider browser support, with no disadvantage.
Written with StackEdit.

Wednesday, June 25, 2014

Casper, d3, jquery and clicking

I spent (wasted? invested?) an awful lot of time yesterday on trying to use CasperJS to click a link inside an SVG diagram that had been made by d3.js. I’ll start this article by showing my Foolish Mistake, but then will document what I did learn.

Foolishness

Normally, in CasperJS, we write this.click('#myButton'), where “#myButton” can be just about any CSS selector.
Aside: it is normally this.click() rather than casper.click()
because it is normally in the handler function of a casper.waitXXX()
function.
This wasn’t working for me trying to click an SVG <g> tag, that starts a search running. I was taking a screenshot 0.5s later to check it had worked, and it showed the page had not changed.
And that turned out to be foolish. The click was working, the search was working, d3 and SVG had nothing to do with anything. The problem was simply were no search results found, and so it never moved to the next page. When I changed the search string, suddenly click() was working.
So my troubleshooting turned out to be barking up the wrong tree. But, still, I did learn a few things from it.

Using evaluate() With jQuery

Instead of calling this.click() you can do something like this:
this.evaluate(function(){
  $('#myButton').click();
  };
This runs JavaScript from within the browser’s context. In this case I use jQuery. This works just as well as using Casper’s click() outside the evaluate().
Here there is no advantage. But, by being in a different scope, we have extra flexibility: we could call other functions, or add new event handlers, etc, etc.

Using evaluate() With d3

Here was another of my attempts, but this one does not work:
this.evaluate(function(){
  d3.select('#myButton').click();
  };
The reason: a d3 selection does not have a ready-made click() function.

Making events happen

this.evaluate(function(){
  var evt = document.createEvent("MouseEvents");
  evt.initMouseEvent("click", true, true, window,
    0, 0, 0, 0, 0, false, false, false, false, 0, null);
  return d3.select('#myButton').node().dispatchEvent(evt);
  };
This is how you do a click with d3 (you could use this approach with jQuery too (see a helper function), but are unlikely to ever need to). The first lines create a (simulated) mouse click. The final line sends that event to the DOM item of interest.
Aside: dispatchEvent() returns false if any of the event handlers
cancelled it, true otherwise.
I learned this here; that answer also says this should have worked:
this.evaluate(function(){
  d3.select('#myButton').on("click")()
  };
This definitely does not work for me. Why? It is actually a cheat, trying to find and call the click handler for the button. In my case the click handler was attached to the parent object (a <g>) not the object I was calling select on. (Also, because it is a cheat, this approach does not work when you’ve attached multiple handlers.)
By the way, if not using jQuery or d3, you can use querySelector() and do it this way:
this.evaluate(function(){
  var evt = document.createEvent("MouseEvents");
  evt.initMouseEvent("click", true, true, window,
    0, 0, 0, 0, 0, false, false, false, false, 0, null);
  return document.querySelector('#myButton').dispatchEvent(evt);
  };

Summary

The real lesson here, for me, was when something doesn’t work, make sure you are judging success in the right way!
Beyond that, it turns out there are a whole host of ways to click a button in a CasperJS script. Use the simplest when you can, bear the others in mind for special occasions.
Written with StackEdit.

Sunday, March 2, 2014

Clearing cookies between CasperJS tests

I spent absolutely ages searching for how to do this, and it turns out to be really easy: phantom.clearCookies();

One page I found suggested casper.clear() would do it. I'm not fully sure what it does clear, but cookies are not covered.

In context here is what it looks like:

  var username = ..., password = ...;
  (function f(label,url){    casper.test.begin(label, {
      test:function(test){ runTheTests(test,label,url,username,password); }
    });
  })("first", "http://127.0.0.1/first");


  (function f(label,url){
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,label,url,username,password); }
    });
  })("second", "http://127.0.0.1/second");

     function runTheTests(test,label,url,username,password){
  casper.start();
  casper.clear();

  phantom.clearCookies();
  casper.thenOpen(url,function(){
    console.log('Loaded:'+url);
    });
 

  ...Other actions and tests here...

  casper.then(function(){
    test.done();
    });

  casper.run();

  }

I.e. first test does a successful login. Second test was supposed to then see a different login page, but in fact it was still logged-in. Adding phantom.clearCookies() did the trick.


Thursday, November 7, 2013

Saving downloaded files in SlimerJS (and Casper and Phantom)

It seems a common request is to be able to see not just the HTML of the main page that PhantomJS/SlimerJS are downloading, but also all the other files (images, CSS, JavaScript, fonts, etc.) that are being fetched. You can use onResourceReceived to see them being fetched, but not their body.

The situation with PhantomJS is a bit confusing: I believe there is a patch to allow this, but it hasn't been applied yet. There is also a download API being proposed (or possibly already implemented), but that appears to be for the special case of files that have a Content-Disposition: attachment header. (?)

In SlimerJS it is possible to use response.body inside the onResourceReceived handler. However to prevent using too much memory it does not get anything by default. You have to first set page.captureContent to say what you want. You assign an array of regexes to page.captureContent to say which files to receive. The regex is applied to the mime-type. In the example code below I use /.*/ to mean "get everything". Using [/^image/.+$/] should just get images, etc.

The below code sample will download and save all files. It is complete; you just have to edit the url at the top.

var url="http://...";

var fs=require('fs');
var page = require('webpage').create();

fs.makeTree('contents');

page.captureContent = [ /.*/ ];

page.onResourceReceived = function(response) {
//console.log('Response (#' + response.id + ', stage "' + response.stage + '"): ' + JSON.stringify(response));
if(response.stage!="end" || !response.bodySize)return;

var matches = response.url.match(/[/]([^/]+)$/);
var fname = "contents/"+matches[1];

console.log("Saving "+response.bodySize+" bytes to "+fname);
fs.write(fname,response.body);
};

page.onResourceRequested = function(requestData, networkRequest) {
//console.log('Request (#' + requestData.id + '): ' + JSON.stringify(requestData));
};

page.open(url,function(){
    phantom.exit();
    });


It is verbose in that it says what it is saving. If you want it much more verbose, to see what other information is passing back and forth, there are two logging lines commented out.

WARNING: this works in SlimerJS 0.9 (and should work in 0.8.x), but the API may change in future (to keep in sync with PhantomJS).


Thursday, October 24, 2013

Why does JavaScript's LocalStorage behave like SessionStorage?

I added LocalStorage to a web page. It seemed like it was working well: when I reload the page, it found and used the existing data. Close the window, when another browser window is still open, then load the URL again: this also works fine, it finds the previous data.

But when I closed all browser windows, and restarted, the data was gone. In other words it appears to just be storing the data per-session, not indefinitely as should be happening. This was happening with both Firefox and Chrome!

It turns out it was configuration, in each browser! See below for the solutions for each. What is frustrating is that neither browser gave any error message; I'm not even sure there is a way to query that the storage is going to end up session-only. I didn't even expect LocalStorage to have the same privacy policy as cookies: cookies get sent back to the server, whereas LocalStorage does not.

Anyway, to the fixes:

Firefox

In Firefox Preferences, Privacy tab, I had both "Accept cookies from sites" and accept "third-party cookies" checked. But under 3rd party cookies I had "Keep Until I Close Firefox". When I changed it to "Keep until they expire" then LocalStorage started working.

This is pretty strange, as it sounds like LocalStorage is considered a 3rd party cookie.

By the way, in about:config, there is also dom.storage.enabled. I had this set to true. If it was false that would be another reason it would not work (though I think then it would not work when just pressing reload either).

Chrome

In Chrome, settings, advanced settings, content settings: under cookies I had "Keep local data only until I quit browser"
One solution is to the change that to "Allow local data to be set (recommended)". The alternative is to add an exception of "allow" for the domain name in question.


Tuesday, October 15, 2013

SlimerJS: getting it to work with self-signed HTTPS

SlimerJS (as of 0.8.3) lacks the commandline options of PhantomJS to say "relax about bad certificates". Unfortunately the self-signed SSL certificate, that developers typically use during development, counts as a bad certificate.

Here are the steps needed to handle this:

1. slimerjs --createprofile AllowSSL
  Make a note of the directory it has created.
  (You can call your new profile anything, "AllowSSL" is just for example.)

2. Go to normal desktop Firefox, browse to the URL in question, see the complaint, add it as a security exception.
  Chances are, if you have been testing your website already, that you've already done this and you can skip this step.

3. Go to your Firefox profile, and look for the file called "cert_override.txt". Copy that to the directory you created in step 1.

4. Have a look at the copy you just made of "cert_override.txt".
  If it only has the entry you added in step 2, you are done.
  Otherwise, remove the entries you don't want.
  (The file format is easy: one certificate per line.)

5. Now when you need to run slimerjs you must run it with the "-P AcceptSSL" commandline parameter.
  E.g. "slimerjs -P AcceptSSL httpstest.js"

If you are using SlimerJS with CasperJS (requires CasperJS 1.1 or later), do the same, e.g.
   casperjs test --engine=slimerjs -P AcceptSSL tests_involving_https.js


Monday, October 14, 2013

Cursed Closures In Javascript Loops

I hit this so many times, scratch my head for a while, then spit out: "Closures!" like it is a really nasty curse that could cause your Grandmother to faint.

I then waste half an hour trying to remember how to get around them, struggling to squint at the various StackOverflow answers to see how they relate to my own loop. So, here is a step-by-step example, that hopefully will make sense to me next time I hit this.

I'm using CasperJS here, but that is not too important. Just consider those lines as "something that gets executed later but is using local variables".

Here is the before:
var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
    for(var ix2 = 0;ix2 < B.length;++ix2){
        var label = A[ix1] + "-" + B[ix2];
        casper.test.begin(label, {
            test:function(test){ runTheTests(test,url,B[ix2],A[ix1]); }
            });
        }
    }
And here is the intermediate stage:
var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
  for(var ix2 = 0;ix2 < B.length;++ix2)(function f(){

    var label = A[ix1] + "-" + B[ix2];
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,url,B[ix2],A[ix1); }
      });
    })();
  }
Then you need to pass into that new function anything outside it that is changing on each pass of the loop. I.e. anything involving ix1 or ix2. It ends up looking like this:
var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
  for(var ix2 = 0;ix2 < B.length;++ix2)(function f(a,b){

    var label = a + "-" + b;
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,url,b,a); }
      });
    })(A[ix1],
B[ix2]);
  }

Thursday, July 4, 2013

Some updating line charts in d3


There are so many lovely d3 demos... but I think this might be more due to the creativity and stubbornness of the people who have chosen to use d3 than because of some innate quality of the library.
(d3 is a charting library, that claims it is not a charting library; instead it likes to see itself as an alternative to JQuery, I think.)

As a case in point, there is this page showing three beautiful examples:
   http://bost.ocks.org/mike/path/

In particular scroll down to the bottom and it is tracking how much scrolling you've done on the page, with the time on the x-axis. But the script for this cleverness is about 110 lines. And it is quite opaque. To put that comment in context I've read the O'Reilly d3 book cover-to-cover, then played with every d3 example in the book.

By the way, another example of scrolling charts, this one with multiple charts overlaid: http://lyngbaek.com/real-time-stream-graph.html

Here is a similar chart done in Rickshaw (which is another library built on top of d3): http://code.shutterstock.com/rickshaw/examples/fixed.html

And another example done in Rickshaw, that is quite heavy-duty: http://code.shutterstock.com/rickshaw/examples/extensions.html

Monday, April 8, 2013

PhantomJS: POST, auth and timeouts

I recently discovered PhantomJS, which (for me) is an alternative to Selenium, with two key differences that (again, for me) make it very useful:
  1. It is headless, meaning you don't see anything graphical. That means I can run it on a server, from the commandline, without needing X installed. It also means it causes less load.
  2. It embeds webkit, rather than attempting to interface with many browsers, and control them as a user would.
The second point allows me to POST to a URL, which is great for testing how web services work in a real browser. Selenium refuses to offer this because it is not something a user can do with a browser. (The workaround in Selenium was to make a temporary page that uses an AJAX call to POST to the URL, then does something with what is returned.)

There are two things that PhantomJS makes difficult, which I will show techniques for here. The first is that authorization is kind-of-broken. The second is timeouts for requests that never finish (e.g. an http streaming web service). But, first, the basic example, without auth or timeouts, and using GET:
var page=require('webpage').create();
var callback=function(status){
    if (status=='success'){
        console.log(page.plainText);
        }else{
        console.log('Failed to load.');
        }
    phantom.exit();
    };
var url="http://example.com/something?name=value";
page.open(url,callback);
Now here is the same code with basic auth (shown in orange), and a five second time-out (shown in red):
var page=require('webpage').create();
page.customHeaders={'Authorization': 'Basic '+btoa('username:password')};
var callback=function(status){
    if(timer)window.clearTimeout(timer);
    if (status=='success' || status=='timedout') {
        console.log(page.plainText);
        }else{
        console.log('Failed to load.');
        }
    phantom.exit();
    };
var timer=window.setTimeout(callback,5000,'timedout');
var url="http://example.com/something?name=value";
page.open(url,callback);
Don't use page.settings.userName = 'username';page.settings.password = 'password'; because it has a bug as of PhantomJS 1.9.0 (it uses two connections for GET requests and doesn't work at all for POST requests). Instead make your own basic auth header as shown here (thanks to Igor Semenko, on the PhantomJS mailing list for this trick).

For the time-out code I still call the same callback, but pass a status of "timedout" instead of "success" (so the callback could react differently, if timedout was a bad thing - here I treat them the same). So, if the URL finishes loading within 5000ms, then callback is called (by the page.open() call) with status equal to "success". If it has not finished within 5000ms then callback is called (by the javascript timer), with status equal to "timedout".

I explicitly clear the timer immediately when entering callback(). This is not really necessary, as we're about to shutdown (the phantom.exit() call) anyway. But it feels safer because otherwise callback() might be called twice (i.e. if the page loaded in exactly 5000ms); the more computation being done in callback(), especially if asynchronous, the more this might occur. (Well to be precise: that catches the case when page loads in just under 5000ms and triggers the callback before the timer does. But, if the timer gets in first, and then the page loads in just over 5000ms, and callback computation takes a while, then we may still get two calls. I think calling page.close() in callback() might prevent this, but that is untested.)

Finally, here is the same code using POST instead of GET:
var page=require('webpage').create();
page.customHeaders={'Authorization': 'Basic '+btoa('username:password')};
var callback=function(status){
    if(timer)window.clearTimeout(timer);
    if (status=='success' || status=='timedout') {
        console.log(page.plainText);
        }else{
        console.log('Failed to load.');
        }
    phantom.exit();
    };
var timer=window.setTimeout(callback,5000,'timedout');
var url="http://example.com/something";
var data="name=value";
page.open(url,'post',data,callback);
The differences are shown in red. It couldn't be easier!

Monday, September 12, 2011

node.js: Good Tutorial

Chatting with a friend at the weekend, node.js came up; I'd only vaguely heard of it, but apparently it is what all the kewl kids are into. Server-side javascript, right up my street.

I took a very quick look yesterday, and it sounded interesting: especially for speed-critical websites. So, today I took a deeper look. First up you should know the official website has a documentation link that only goes to the API documentation. In real terms that means node.js is officially undocumented. Then from the Wiki I found a link to a free e-book called Mastering Node. I won't give the link as it is (currently) over-priced: poorly organized and unfinished.

I was  getting a bit demoralized when a search found http://www.nodebeginner.org/. Now this is more like it. In fact this is an excellent tutorial, that goes right from raw beginner to a reasonably complex mini-application. (I read it in HTML format, but as an ebook it is 60 pages, so you can get an idea of how involved it is going to get.)

I followed along, and it was fairly easy, though I did have a lot of trouble outputting the POST-ed data. It always said "undefined" in my browser. I could do a console.log() of the variable just before and just after writing it to the browser and it was set correctly. I cleared the cache repeatedly, and tried a second browser. Annoyingly I didn't solve that problem in the end.

The main point of this blog post is to recommend the above tutorial/ebook if you're interested in getting a feel for node.js, but What Do I Think About node.js?

Hhhmmm. First and foremost, it should only be used by expert programmers. It is a bit like C++ in that it is going to be easy to shoot yourself in the foot. Asynchronous programming is hard. But if you use a synchronous programming-style you will lock up the whole server. I'm thinking in terms of the web server example application here (which is the use-case I had in mind for it). Asynchronous programming is hard. Yes, I know I already said that but I don't think you thought through what that means in the Real World. What will happen is programmers will take a short-cut: they'll use little bits of synchronous code for jobs they know are so quick that no-one will notice. (Even the above tutorial does this: fs.renameSync)  The problem is any job involving any I/O device (like a hard disk file system or a TCP/IP socket) will take 10 times longer to finish than average, about 1% of the time. (I made that statistic up, but the principle is true, so stay with me...)

What does that mean? When that happens it will lock the whole web server up, and all the other requests will block. Take the extreme case of doing a sync action that results in a time-out because the resource has gone offline. If the time-out is 30 seconds, the whole web server is down for 30 seconds. All of your customers are getting time-outs. Every image comes up broken.

Another nice thing about the Node Beginner tutorial is the links to deeper information... and nested in one of the comments is a link to a paper comparing threads and events: Why Events Are A Bad Idea  Well, their conclusion is in the title, but if you look at their charts the important thing to learn is that well-written event handling code and well-written threading code are basically as quick as each other. (You won't ever reach the right-side of the charts in a real website on a single server; their example is just serving a static image. So the differences are only of interest to academics.)

But, although dealing with threads is hard, asynchronous programming is even harder, IMHO.

Now, if node.js had a web server module where it maintained a thread pool and each new request got its own thread, then I could program in a synchronous style in my thread, happy in the knowledge I won't break the web server, and also happy I'll be able to meet my deadlines...

Tuesday, August 9, 2011

Debugging Regexes

Cue: dramatic music. There I was under pressure, enemy fire going off in all directions, and my unit test had started complaining. The test regex was 552 characters long, the text being tested was almost as long, and each run of the unit test takes 30 seconds. Talk about finding a needle in a haystack. James Bond only had to choose between cutting the red or the blue wire. He had it easy.

But I lived to tell the tale. Playing the Bond Girl in this scenario was http://www.regextester.com/ (I actually used version 2 which, though alpha, worked fine).

It still wasn't smooth sailing. The above site assumes the regex is surrounded by /.../ but mine wasn't. So, first I had my unit test output the regex, then I escaped it correctly for use with /.../ then pasted it into the Regex Tester. I also pasted in the text to test. It should match; it doesn't. So I put the cursor at the end of my regex and deleted a few characters at a time. After deleting about two-thirds (!!) of it, finally the text turned red and I had a match. I could see exactly where the match stopped and realize what was missing in my regex. I fixed the regex (simultaneously in RegexMatcher and in my unit test script) and repeated. I had to delete back to almost the same point. It took half a dozen passes before the whole regex matched.

The code looks to be open source javascript. So maybe I will hack on it, to automate the above process (my better Bond Girl, if you like): I would give the regex, the target text, say I expect a match, and it will find the longest regex that matches and show me how much of the target text got matched. (Ah, it uses ajax requests to back-end PHP for the preg and ereg versions, and that code is not available; but at least I could do this for javascript regexes.)

Enough with poking around inside today's Bond Girl. Down the cocktail, jacket on, back to the field...