Captcha never changed (securimage)

It took a while of testing before I noticed, then I finally realized the captcha on a web app I was working never changed!

Thankfully, before launching into some heavy-duty troubleshooting, I had a flash of inspiration: to make my unit tests work nicely I was setting the random seed to a constant (i.e. to get repeatable test data). That test data is actually being generated for every page request of the main web app (just during development).

So the fix was as trivial as calling mt_srand(time());  just before creating the Captcha  (as I'm using the Securimage library, just before the $img = new Securimage(); call).

If only all troubleshooting was this quick!

Backing-up a bunch of small files to a remote server

I have a directory, containing lots of files, and I want an off-site, secure backup.

Even though the remote server might be a dedicated server that only I know root password for, I still don't trust it. Because of the recent NSA revelations I no longer consider myself paranoid. Thanks guys, I can look at myself in the mirror again!

As a final restriction, I don't want to have to make any temp files locally: disk space is tight, and the files can get very big.


Here we go:

cd BASE_DIR
tar cvf - MY_FOLDER/ | gpg -c --passphrase XXX | ssh REMOTE_SERVER 'cat > ~/MYFOLDER.tar.gpg'

(the bits in capitals are the things you replace.)

Notes

• The "v" in "tar cvf" means verbose. Once you are happy it is working you will want to use "tar cf" instead.

• The passphrase has to be given in the commandline because stdin is being used for the data!! A better way is to put the passphrase in another file: --passphrase-file passfile.txt. However note that this is only "better" on multi-user machines; on a single-user machine there is no real difference.

• I'm using symmetric encryption. You could encrypt with your key pair, in which case the middle bit will change to: gpg -e -r PERSON  Then you won't need to specify the passphrase.

• In my case REMOTE_SERVER is an alias to an entry in ~/.ssh/config. If you are not using that approach, you'll need to specify username, port number, identity file, etc. By the way, I'm not sure this method will work with password login, only keypair login, because stdin is being used for the data.

• Any previous MYFOLDER.tar.gpg gets replaced on the remote server. So, if the connection gets lost halfway during the upload then you've lost your previous backup. I suggest using a datestamp in the filename, or something like that.

What about to get the data back?

cd TMP_DIR
ssh REMOTE_SERVER 'cat ~/MYFOLDER.tar.gpg' | gpg -d --passphrase XXX | tar xf -

You should now have a directory called MYFOLDER, with all your files exactly as they were.


Outstanding questions

Is it possible to use this approach in conjunction with Amazon S3, Google Drive, Rackspace cloud files, or similar storage providers? E.g. 100GB mounted as a Rackspace drive is $15/month (plus the compute instance of course, but I already have that), whereas 100GB as cloud files is $10/month, or $5/month on google drive. ($9.50/month on S3, or $1/month for glacier storage). Up to 15x cheaper: that is quite an incentive.

2013-10-08 Update: The implicit first half of that question is: is there a way to stream stdout to the remote drive (whether using scp or a specific commandline tool).
For Amazon S3 the answer is a clear "no": http://stackoverflow.com/q/11747703/841830 (the size has to be known in advance).
For Google Drive the answer is maybe. There is a way to mount google drive with FUSE: https://github.com/jcline/fuse-google-drive   It looks very complicated, describes itself as alpha, and the URL for the tutorial is a 404.
For Rackspace CloudFiles (and this should cover all OpenCloud providers), you can use curl to stream data! See "4.3.2.3. Chunked Transfer Encoding" in the cloud files developer guide HOWEVER, note that there is a 5GB limit on a file. That is a show-stopper for me. (Though by adding a custom script instead of "ssh REMOTE_SERVER 'cat > ~/MYFOLDER.tar.gpg'", I could track bytes transferred and start a new connection and file name at the 5GB point, so there is still hope. Probably only 10 lines of PHP will do it. But if I'm going to do that, I could just as easily buffer say 512MB in memory at a time, and use S3)

NOTE: Because I've not found an ideal solution yet, I never even got to the implicit second part of the question, which is if the need to "cat" on the remote server side will cause problems. I think not, but need to try it to be sure.

Actual costs: rackspace cloud

A few months back I decided to put a 24/7 script on a Rackspace Cloud instance, instead of the more obvious Amazon EC2 choice. The reason at the time was my needs were low CPU but relatively high bandwidth and diskspace usage and it worked out cheaper.

Now I've had a few invoices in I am relieved to say there was no catch. My past three invoices have been $11.99, $11.99 and $12.20 (USD). This is for a minimal CPU spec (256MB, 1.6% of a quad core CPU, 10GB disk), 1.1 to 1.3 GB/month of outgoing bandwidth each month (there is no charge for incoming bandwidth), and cloud storage rising from 4 to 8GB. 90% of the monthly cost is for the machine, and the cloud storage has risen from $0.63 to $1.15. The bandwidth is not costing much at all.

In contrast on Amazon EC2, the micro instance would cost $15.65 (including $1 for 10GB of EBS storage), while a small instance would cost $62.25/month, of which $0.03 is the bandwidth usage. (The first year of that micro instance would be free if you are a new customer, but I am not.)

So, at the CPU bottom-end, Rackspace is winning on cost. The other feature of Rackspace Cloud that I love is there is an automatic daily backup of the full disk image, and that backup is stored in the cloud storage. (Storing that backup is basically all my $1/month cloud storage costs.)

What do I not like? I keep using up my 10GB disk space. But there seems no way to move to 20GB without doubling the CPU spec and doubling the monthly cost; with Amazon micro I'd just increase the EBS storage space. With an Amazon small instance I'd get 160GB and would not care.

What do I not like about Rackspace and Amazon? It is that you just get a basic linux distro. You have to spend time installing, configuring and maintaining. And the configuration is not trivial; I've kept a log of all I've had to do, and it includes things like moving ssh off of port 22, setting up an iptables firewall, installing a mail server (not a POP server, just enough so I can *send* email alerts), and writing my own low-diskspace email alert script. The latter was done just the other day after my application broke, yet again, because the machine had run out of disk space.

P.S. As I want 24/7, and have not mentioned the need to scale, what about cheaper shared hosting? Well, I couldn't find a VPS, that gives me root access and no restrictions, for under $10/month. It seems Rackspace is winning that fight too?

(2012-08-15 UPDATE: Rackspace are stopping their minimal server config: no more 256MB option in their "nextgen" V2 API. Also scheduled images are not available in the V2 API (yet). In other words the two things I pointed out that were good, in the above article, are going! I guess the Rackspace marketing department will have to work out a positive spin on "cutting out our competitive advantage so we look just like the competition now" ;-)

svn, ssh, /bin/false, git, /etc, etc.

I just spent almost two hours troubleshooting an svn server. It was set up to allow ssh access from a few users over ssh as described in this helpful blog post.

Today I realized none of the svn clients could run svn update: "svn: Network connection closed unexpectedly"

Part of the challenge was that I'd not run "svn update" in 1-2 months, so I had a big time frame for breakage. But all I could remember changing recently was commenting out some lines in my iptables firewall, so I spent quite a while staring at that. And I'd updated some packages and rebooted.

I spent a lot of time looking at ssh verbose output. To do this use

export SVN_SSH="ssh -vv"

(-v for quite verbose, -vv for more, -vvv for even more, etc.) No error messages: it connects, runs the svnserve command, passes up some environmental variables, sends some data, and then simply loses the connection.

Then I wondered if the svn user had somehow lost permission for something important. It has /bin/false as its shell, so I gave it /bin/bash, used su to become root, then "su - svn". Ran the svnserve command. It is fine. Looked at the repository. It is fine. Ran svnadmin to check for locks. None.

Giving up, I started crafting an email to post somewhere asking for help. I went back to get the exact error message svn update was giving me... and it worked. Yep, the other client works now too. My first guess was running svnadmin had quietly fixed something. But then, on a hunch, I changed svn's shell back to /bin/false. Yep, that broke it. It seems the svn user has to have a valid login shell.

Okay, problem solved. But I'm sure that svn has always had /bin/false, and now I wanted to know if that is true, and if and when it changed. It is too late now, but ready for next time, I decided to put all of /etc into a git repository (no need for a central repository, so git is far better choice than svn for this). The git commands (all run as root) are trivial:

cd /etc
git init
git add .
git commit -m "Initial files" -a

I found this page of someone who has done something similar, and he suggested sending the git status report from a daily cron, so I did that too.

It is so easy, and disk space so plentiful, that I think I will do it on all my linux machines.

UPDATE: /etc has some files that get edited a lot, so to reduce noise this is my .gitignore file so far:

/cups/printers.conf*
/cups/subscriptions.conf*
/emacs/site-start*
/mtab
/adjtime
/ld.so.cache
/*-
*~

(Use "git rm --cached cups/printers.conf*" (for instance) to stop git tracking the files if they've already been added.)

Buffalo Air Station setup on existing home LAN

I found this surprisingly difficult; though it turns out the steps involved are quite easy...once you know how.

For this article I'll assume your existing LAN is 10.0.0.0/8 with 10.0.0.1 as the default gateway. I'm going to give the wireless LAN router 10.1.2.3. Substitute 10.0.0.1 and 10.1.2.3 below for something suitable for your LAN (and change the 255.0.0.0 network mask accordingly).

These instructions assume factory reset status, with the "router" switch (on the case) set to "On", rather than "Off" or "Auto". The AirStation is connected to the LAN hub using the Internet socket (the blue one).

1. Connect a computer to the Air Station directly (with ethernet cable, to one of the four LAN sockets), and set your computer to use DHCP, so that it gets an IP address that can see the Air Station! (Most likely your computer will receive 192.168.11.2.)
Note: The computer you connect with does not need to have a wireless card in it; it does have to have wired ethernet though.
Tip: Don't use your main computer for this step; that way you will be able to still use your main computer for: a) googling when you hit problems; b) ping tests (see step 5 below).

2. Connect by browser to 192.168.11.1
Use root as the username, with blank password.

3. Internet/LAN | Internet
IP Manual: 10.1.2.3
255.0.0.0
Extended:
Default GW: 10.0.0.1
DNS: (whatever DNS server you use: this is what will be handed out to wireless clients that connect using DHCP)

4. Wait for it to restart. (I also enabled pings at this point too.)

5. Then cycle the power; this appears to be essential.
Now you should be able to ping 10.1.2.3 from outside, and also ping from the connected computer to 10.0.0.1. And if you can do that then you can also connect to internet! You are done, time for a nice cuppa.

6. Test from a wireless device to make sure you are required to type in the key.

7. Set a root password.

scp with multiple targets: ssh-add

I sometimes have the need to upload the same file to two or more locations on the target server. In fact I even have a script to help me, which goes something like this:

scp file1 file2 remote01:/path/to/somewhere/
 scp file1 file2 remote01:/path/to/another/

The problem is I need to type the password for remote01 twice. I had never managed to find some clever scp syntax to allow specifying two destinations, and a post on the TLUG mailing list confirmed that. But what I did learn from TLUG is that there is something called ssh-agent that can store passphrases for key pairs; this plugged a gap in my knowledge. There are three ways to login in via ssh/scp:

1. Give your password
2. Make a keypair with no passphrase
3. Make a keypair with a passphrase

The second way is used to allow scp from cron jobs, to automate copying files between two machines. I'd never really got the point of the third way: if you have to type a passphrase why not just use the first method, and save messing around making a key pair. (Well, yes, there is better security: a log-in then requires both something I have and something I know; you have to turn off PasswordAuthentication on your ssh server for this to have any meaning though. Thanks to Kalin for this comment.)

But it turns out there is this program called ssh-agent that remembers passphrases for you. And I found it is already running in the background on Ubuntu.

Enough chat, let's look at the solution. First, I created a one-off keypair for this script, on my machine, using something like this:

cd ~/.ssh
  ssh-keygen -t dsa -C me@example.com -f key_for_remote01
  chmod 600 key_for_remote01*
  scp -p key_for_remote01.pub remote01:~

When generating the keypair give a reasonably secure passphrase (you will have to type it in each time, and it is only of use to people in possession of key_for_remote1, so no need for a 20-random-character monster; I believe it is perfectly fine for it to be the same as your normal ssh login password for remote01).

Then log in to remote01 and append key_for_remote01.pub to ~/.ssh/authorized_keys. If that file does not exist then you can just rename key_for_remote01.pub to authorized_keys and move it into ~/.ssh/

(By the way, there is no need to put your private half of the keypair in ~/.ssh/ but that seems as good as place as anywhere else.)

Now, I modified my script as follows:

ssh-add -t 120 ~/.ssh/key_for_remote01
 scp -i ~/.ssh/key_for_remote01 file1 file2 remote01:/path/to/somewhere/
 scp -i ~/.ssh/key_for_remote01 file1 file2 remote01:/path/to/another/
 ssh-add -d ~/.ssh/key_for_remote01

What happens is the ssh-add line will ask you for your passphrase. The two scp lines then work automatically. Finally the ssh-add -d stops it caching your passphrase (forcing you to type your passphrase each time you run this script).

The -t 120 parameter says the passphrase will expire after two minutes. This is just in case the batch file doesn't complete and so does not get chance to run ssh-add -d.


Note: you can use that same key pair for other machines. Basically anywhere you put the *.pub half of the key pair will let you login. And you can login from anywhere you have the private half of the key pair.

Note: the timeout/deletion code above is deliberate for this application, but you don't have to do it this way. By allowing it to cache it permanently you would only be prompted for your passphrase once, and then all future ssh and scp logins would be automatic. They will be cleared when you log-out of gnome (on ubuntu, at least) or shutdown your machine.

Note: If you don't want to specify -i each time you use ssh/scp then you can add an entry to your ~/.ssh/config file, like this:

Host remote01
        Hostname 10.1.2.3
        Port 22
        IdentityFile ~/.ssh/key_for_remote01

Note: I used -C with my real email address. This is put in the public key, and I wanted the administrator of remote01 to know who put the key there. Without -C it defaulted to "myusername@myhost". The administrator of remote01 knows nothing about my machine names so this seemed unreasonable and I decided to use -C. But the advantage of that default is it seems ssh-agent knows about that name and will prompt automatically the first time you try to use ssh/scp, which means there is no need to run ssh-add first. I have not worked out yet if ssh-agent can be told to know about my email address too.

Image Protection in Zend Framework

Sometimes you want to only show certain images (or other media) to certain users (e.g. those that are logged in, or those that have paid for it). This quick tutorial will show how to write the controller for Zend Framework for serving images. This is for ZF 1.10, but as far as I can tell it is not using anything new or unusual.

This tutorial does not cover the validation part of the code.

We start with a controller that does not use views, layout and all that stuff:

class ImgController extends Zend_Controller_Action
{
public function init()
{
$this->_helper->viewRenderer->setNoRender(true);
$this->_helper->layout()->disableLayout();
}
}

Now add an action that will serve all images:

public function imgAction(){
$type=$this->getRequest()->getParam(1);
$fname=$this->getRequest()->getParam(2);
$ext=$this->getRequest()->getParam(3);

echo "type=$type, fname=$fname, ext=$ext";
}

public function badAction(){
$this->getResponse()->setHeader('Content-Type','image/jpeg');
}

The idea is that URLs such as http://127.0.0.1/img/folder/abc.jpg will end up at the imgAction() function, and $type will get set to "folder", $fname to "abc" and $ext to "jpg". To do that we need to set up what is called a router, which we will do next. The badAction() will handle any problems; this crude code will send back a 0 byte jpeg.

To create a router, jump to your bootstrap file and create a function something like this:

public function _initCustomRouting(){
$frontController=Zend_Controller_Front::getInstance();
$router=$frontController->getRouter();
$router->addRoute(  //To catch any that are not formatted correctly
'imgHandlingBad',
new Zend_Controller_Router_Route('img/*',
array('controller'=>'img', 'action'=>'bad')
)
);
$router->addRoute(  //Formatted as /url/type/fname.ext
'imgHandling',
new Zend_Controller_Router_Route_Regex('img/(.+)/(.+)\.(.+)',
array('controller'=>'img', 'action'=>'img')
)
);
}

Note that order: imgHandlingBad must come before imgHandling. I found this introduction (7 minute screencast) to using Zend_Controller_Router_Route very useful; then some trial and error and study of the ZF source code.

Now you should be able to test http://127.0.0.1/img/folder/abc.jpg and see our debug comment. But without a folder (e.g. http://127.0.0.1/img/abc.jpg) it will get handled by the badAction(), as will those without an extension: http://127.0.0.1/img/folder/abcjpg

Now the final step is to feed back some images, so in imgAction() replace the echo line with this code:

$path=$basePath.$type;
$fullPath=$path.$fname.'.'.$ext;
if(!file_exists($fullPath))return $this->badAction();

switch($ext){
case 'jpg':$mime='image/jpeg';break;
case 'png':$mime='image/png';break;
case 'gif':$mime='image/gif';break;
default:return $this->badAction();
}
$this->getResponse()->setHeader('Content-Type',$mime);

readfile($fullPath);

It decides the filename, decides the mime-type based on extension, and then readfile passes on the binary data.

In the above code we use the $type directly to decide the image path. You could instead map type to different directories. E.g.

switch($type){
case 'www':$path='/var/www/main_images/';break;
case 'www/articles':$path='/var/www/html/special/articles/';break;
case 'family':$path='/home/user/images/family/';break;
default:return $this->badAction();
}

Another way you might use $type is if certain users can only see certain types of images. You could do different validation checks in each case statement above.

Got any suggestions to make this code better? Let me know!

P.S. One special note about this technique. In the typical MVC Zend Framework setup, if you create public/img/ then apache will serve images from there; only if the image is missing in that location will Apache ask the Zend Framework to serve it. You might use this to your advantage to speed up delivery of certain common images. But it also opens the way to accidentally serving images that are supposed to be protected.

How much is that password worth?

These people have published details about how they used Amazon EC2 to crack passwords:
http://news.electricalchemy.net/2009/10/cracking-passwords-in-cloud.html

Personally, I skipped all the details and went straight to the interesting conclusions page:
http://news.electricalchemy.net/2009/10/password-cracking-in-cloud-part-5.html

It tells you how much it will cost someone (in EC2 charges) to crack your passwords, based on their lengths and the number of characters you use.

I used to think 8 characters was a good password. Seems it is worth about $3, or $45 if I've mixed in some numbers. Gulp. And all this is assuming there are no dictionary words in there. Double gulp.

Flex/Flash accessing both network and local files

Flash wants you to be safe, which means a swf you make cannot access the network (not even localhost!) and read local files. This can be a pain when you have a swf file you trust (e.g. when you wrote it yourself)! But the bigger pain is how badly Adobe document the workaround, especially if you are on linux.

First the answer. As root, make /etc/adobe/FlashPlayerTrust/. Then if your trusted swf, and its files are in /home/georgebush/killforoil/ you make a file called:
/etc/adobe/FlashPlayerTrust/killforoil.cfg
and in that cfg file you put this line:
/home/georgebush/killforoil/

George is now able to continue to screw the world without those Adobe do-gooders getting in his way.

Adobe document this (for Windows/Mac) here:
http://livedocs.adobe.com/flex/3/html/help.html?content=05B_Security_01.html

(Why am I saying the document it so badly? Because everywhere else I looked, they say using use-network=true when you make your swf file means it can never access local files.)

Once I knew that FlashPlayerTrust was the key word, I was able to find these linux instructions (which also tell you how to do per-user configurations):
http://www.abdulqabiz.com/blog/archives/flash_and_actionscript/flash_player_trust_f.php