Why does JavaScript's LocalStorage behave like SessionStorage?

I added LocalStorage to a web page. It seemed like it was working well: when I reload the page, it found and used the existing data. Close the window, when another browser window is still open, then load the URL again: this also works fine, it finds the previous data.

But when I closed all browser windows, and restarted, the data was gone. In other words it appears to just be storing the data per-session, not indefinitely as should be happening. This was happening with both Firefox and Chrome!

It turns out it was configuration, in each browser! See below for the solutions for each. What is frustrating is that neither browser gave any error message; I'm not even sure there is a way to query that the storage is going to end up session-only. I didn't even expect LocalStorage to have the same privacy policy as cookies: cookies get sent back to the server, whereas LocalStorage does not.

Anyway, to the fixes:

Firefox

In Firefox Preferences, Privacy tab, I had both "Accept cookies from sites" and accept "third-party cookies" checked. But under 3rd party cookies I had "Keep Until I Close Firefox". When I changed it to "Keep until they expire" then LocalStorage started working.

This is pretty strange, as it sounds like LocalStorage is considered a 3rd party cookie.

By the way, in about:config, there is also dom.storage.enabled. I had this set to true. If it was false that would be another reason it would not work (though I think then it would not work when just pressing reload either).

Chrome

In Chrome, settings, advanced settings, content settings: under cookies I had "Keep local data only until I quit browser"
One solution is to the change that to "Allow local data to be set (recommended)". The alternative is to add an exception of "allow" for the domain name in question.

SlimerJS: getting it to work with self-signed HTTPS

SlimerJS (as of 0.8.3) lacks the commandline options of PhantomJS to say "relax about bad certificates". Unfortunately the self-signed SSL certificate, that developers typically use during development, counts as a bad certificate.

Here are the steps needed to handle this:

1. slimerjs --createprofile AllowSSL
  Make a note of the directory it has created.
  (You can call your new profile anything, "AllowSSL" is just for example.)

2. Go to normal desktop Firefox, browse to the URL in question, see the complaint, add it as a security exception.
  Chances are, if you have been testing your website already, that you've already done this and you can skip this step.

3. Go to your Firefox profile, and look for the file called "cert_override.txt". Copy that to the directory you created in step 1.

4. Have a look at the copy you just made of "cert_override.txt".
  If it only has the entry you added in step 2, you are done.
  Otherwise, remove the entries you don't want.
  (The file format is easy: one certificate per line.)

5. Now when you need to run slimerjs you must run it with the "-P AcceptSSL" commandline parameter.
  E.g. "slimerjs -P AcceptSSL httpstest.js"

If you are using SlimerJS with CasperJS (requires CasperJS 1.1 or later), do the same, e.g.
   casperjs test --engine=slimerjs -P AcceptSSL tests_involving_https.js

Cursed Closures In Javascript Loops

I hit this so many times, scratch my head for a while, then spit out: "Closures!" like it is a really nasty curse that could cause your Grandmother to faint.

I then waste half an hour trying to remember how to get around them, struggling to squint at the various StackOverflow answers to see how they relate to my own loop. So, here is a step-by-step example, that hopefully will make sense to me next time I hit this.

I'm using CasperJS here, but that is not too important. Just consider those lines as "something that gets executed later but is using local variables".

Here is the before:

var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
    for(var ix2 = 0;ix2 < B.length;++ix2){
        var label = A[ix1] + "-" + B[ix2];
        casper.test.begin(label, {
            test:function(test){ runTheTests(test,url,B[ix2],A[ix1]); }
            });
        }
    }

And here is the intermediate stage:

var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
  for(var ix2 = 0;ix2 < B.length;++ix2)(function f(){
    var label = A[ix1] + "-" + B[ix2];
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,url,B[ix2],A[ix1); }
      });
    })();
  }

Then you need to pass into that new function anything outside it that is changing on each pass of the loop. I.e. anything involving ix1 or ix2. It ends up looking like this:

var url="http://example.com/";
var A=['a','b','c'];
var B=['','99'];

for(var ix1 = 0;ix1 < A.length;++ix1){
  for(var ix2 = 0;ix2 < B.length;++ix2)(function f(a,b){
    var label = a + "-" + b;
    casper.test.begin(label, {
      test:function(test){ runTheTests(test,url,b,a); }
      });
    })(A[ix1],B[ix2]);
  }

Backing-up a bunch of small files to a remote server

I have a directory, containing lots of files, and I want an off-site, secure backup.

Even though the remote server might be a dedicated server that only I know root password for, I still don't trust it. Because of the recent NSA revelations I no longer consider myself paranoid. Thanks guys, I can look at myself in the mirror again!

As a final restriction, I don't want to have to make any temp files locally: disk space is tight, and the files can get very big.


Here we go:

cd BASE_DIR
tar cvf - MY_FOLDER/ | gpg -c --passphrase XXX | ssh REMOTE_SERVER 'cat > ~/MYFOLDER.tar.gpg'

(the bits in capitals are the things you replace.)

Notes

• The "v" in "tar cvf" means verbose. Once you are happy it is working you will want to use "tar cf" instead.

• The passphrase has to be given in the commandline because stdin is being used for the data!! A better way is to put the passphrase in another file: --passphrase-file passfile.txt. However note that this is only "better" on multi-user machines; on a single-user machine there is no real difference.

• I'm using symmetric encryption. You could encrypt with your key pair, in which case the middle bit will change to: gpg -e -r PERSON  Then you won't need to specify the passphrase.

• In my case REMOTE_SERVER is an alias to an entry in ~/.ssh/config. If you are not using that approach, you'll need to specify username, port number, identity file, etc. By the way, I'm not sure this method will work with password login, only keypair login, because stdin is being used for the data.

• Any previous MYFOLDER.tar.gpg gets replaced on the remote server. So, if the connection gets lost halfway during the upload then you've lost your previous backup. I suggest using a datestamp in the filename, or something like that.

What about to get the data back?

cd TMP_DIR
ssh REMOTE_SERVER 'cat ~/MYFOLDER.tar.gpg' | gpg -d --passphrase XXX | tar xf -

You should now have a directory called MYFOLDER, with all your files exactly as they were.


Outstanding questions

Is it possible to use this approach in conjunction with Amazon S3, Google Drive, Rackspace cloud files, or similar storage providers? E.g. 100GB mounted as a Rackspace drive is $15/month (plus the compute instance of course, but I already have that), whereas 100GB as cloud files is $10/month, or $5/month on google drive. ($9.50/month on S3, or $1/month for glacier storage). Up to 15x cheaper: that is quite an incentive.

2013-10-08 Update: The implicit first half of that question is: is there a way to stream stdout to the remote drive (whether using scp or a specific commandline tool).
For Amazon S3 the answer is a clear "no": http://stackoverflow.com/q/11747703/841830 (the size has to be known in advance).
For Google Drive the answer is maybe. There is a way to mount google drive with FUSE: https://github.com/jcline/fuse-google-drive   It looks very complicated, describes itself as alpha, and the URL for the tutorial is a 404.
For Rackspace CloudFiles (and this should cover all OpenCloud providers), you can use curl to stream data! See "4.3.2.3. Chunked Transfer Encoding" in the cloud files developer guide HOWEVER, note that there is a 5GB limit on a file. That is a show-stopper for me. (Though by adding a custom script instead of "ssh REMOTE_SERVER 'cat > ~/MYFOLDER.tar.gpg'", I could track bytes transferred and start a new connection and file name at the 5GB point, so there is still hope. Probably only 10 lines of PHP will do it. But if I'm going to do that, I could just as easily buffer say 512MB in memory at a time, and use S3)

NOTE: Because I've not found an ideal solution yet, I never even got to the implicit second part of the question, which is if the need to "cat" on the remote server side will cause problems. I think not, but need to try it to be sure.