EC2: move a large file between Windows instances

Moving a file between linux machines is easy-peasy, just use scp. (You can be sure ssh/scp is on all your linux ec2 machines.) Windows? Sigh, Windows. To get ssh/scp on Windows you need to install cygwin, and that is a non-trivial step to take.

So, how to move a 60GB file to move from one Windows ec2 instance to another Windows ec2 instance (in a different region)? Here is what I did:

1. Install CloudBerry Pro on the source server. (Must be the pro version: you can get a 14-day free trial; when that expires it is a $30 cost)[1]
2. Install CloudBerry free version on the target server.
3. Test copying over a small file, via your S3 account, to make sure it works. I'm assuming you already know how to use this type of two-pane file-copy application. (I created the bucket in the same region as the target machine: that means the upload takes longer than the download.)
4. In CloudBerry Pro, Tools menu, then Options, then choose Compression And Encryption tab. Check "Use Compression".[2]
5. Copy the big file. It gives no progress.
6. When it had finished it said it was 21% done. Very confusing. And on the server it just showed as 13GB file, not a 60GB one.[3]
7. Download to your target server, using CloudBerry free version. (yes it works fine to download large files, to download compressed files.)
8. Rename your downloaded file with a ".gz" extension, as that is what it actually is.[3]
9. Install 7-zip, if you don't have a program that can deal with gzip files. It tells me the file is 2GB compressed, 13GB uncompressed. Ignore that, it is just being stupid. Decompress it, and you get a 60GB file.

Phew,  hard work. If you needed to do it regularly you should install cygwin and use scp! <soap-box>Or port your applications to linux where the living is easy. Apart from the fact that running Windows machine is harder, it is also significantly cheaper to run the cloud instances.</soap-box>

[1]: I've heard, but not confirmed, that you can uninstall it from one machine, then use the same install key on a different machine. If true, that is quite a fair license, and I encourage you to support them.

[2]: As we saw, this creates more work, so uncheck after doing your big file.

[3]: I think CloudBerry Pro should have put a .gz extension on the file, when it uploaded it, to make it clear what was going on.

g++ linker: behaviour change

I was working late on a Friday night, setting up an Ubuntu 12.04 machine. Final step was compiling a couple of C++ programs, that worked fine on Ubuntu 10.04, 11.04 and Centos 5. I got linker complaints regarding boost::program_options. Sounded like it wasn't installed. Strange, as I'd installed "libboost-all-dev".

Poke around... all the files seem to be there. The lib files are in /usr/lib/. Surely g++ is already looking there, but I added "-L/usr/lib/" to be sure. No help. I'd tried all the various ideas I found on StackOverflow, until eventually I found http://stackoverflow.com/a/11250976/841830 where "panickal" suggested the list of libraries has to come last. Yeah sure. But getting desperate by this point I give it a try... and it works!

Specifically, in my Makefile, I had to change this line:
    $(TARGET): $(OBJS)
        $(CXX) -s $(LDFLAGS) $(OBJS) -o $@

To look like this:
    $(TARGET): $(OBJS)
        $(CXX) -s $(OBJS) -o $@ $(LDFLAGS)

So somewhere between g++ 4.5.2 and g++ 4.6.3 it has gone from being easy-going and taking parameters in any order, to requiring certain ones to come at the end. A strange evolution. (I understand why the ordering amongst the library files matters, I just don't see why they now have to all go together at the end.)

But luckily I was using a Makefile with some structure, so the fix was trivial, and my Friday night did not become a Saturday morning!

The cloud and Wall Street

An enjoyable video on how Wall Street uses the cloud, and why generally they don't: http://www.infoq.com/presentations/Cloud-Wall-Street

If you only have one minute, here is my summary:

Yes, banks could not just save money but also make their development more nimble and perhaps even more reliable, by moving to the cloud. But because they have loads of legacy systems that integrate in complex ways, there is the element of "if it ain't broke then don't try to fix it." An even bigger reason is moving would be a major project that would distract energy at all levels of the company from their real business of making money. They'd rather make money from increasing sales than make money from reducing costs.

If you have 62 minutes, and have an interest in the intersection of IT and finance, the whole thing is worth your time.
If less time, and are interested in why they should be using the cloud more, that is 35:30 to 39:00.
If you want to understand why they don't, 32:00 to 35:30, and questions from 39:00-44:00. Then 48:00 to 52:00.
If you are interested in Apache Ambari, and creating your own clouds, which is that is 52:00 to 60:00; it is only loosely related to the main theme of the talk.
Question at 60:00 on application rot is interesting.

Note: he generally uses cloud in the sense of virtualization on heavy-duty hardware, that you own and install. (As opposed to the sense of compute units running out there somewhere, that you pay for by the hour, and that you can start and stop just when you need them.)

Problems with .net4 regasm.exe on shared disk

I've been struggling trying to use the .NET4 version of regasm on a C#4 DLL. The error was:

    RegAsm : error RA0000 : Count not load file or assembly '...' or one of its dependencies. Operation is not supported.

It turns out this is a security thing, due to not trusting my network drive (where the DLL is kept).

The fix was to go to c:\windows\Microsoft.NET\Framework\v4.0.30319\ and open regasm.exe.config in a text editor. Mine now looks like this (the lines I added are shown in red):


<configuration>
    <startup uselegacyv2runtimeactivationpolicy="true">
        <requiredruntime imageversion="v4.0.30319" safemode="true" version="v4.0.30319">
        <supportedruntime sku="client" version="v4.0">
    </supportedruntime></requiredruntime></startup>
<runtime>
    <loadfromremotesources enabled="true"> </loadfromremotesources></runtime>
</configuration>

I didn't have to do this with the .NET2 version of regasm.

By the way, if you get an error complaining about "CoCreateInstance failed with error 0x80040154", that (in my case) was due to having previously registered this DLL as a .NET2 assembly with .NET2 version of regasm. Recompiling the DLL for any of .NET3 or .NET3.5 was fine, but targetting .NET4 gave that error. Hence the need to register it again using v4  of regasm.