<- vs. = in R

One of the first thing to strike a programmer new to R is <- all over the place. "Ugly!" may be a first reaction. But probe a bit deeper and you'll find you can use = just as well. "So why bother with <- at all?!" is a common reaction.The most pragmatic advice I've seen from experienced R programmers is it really just comes down to personal preference.

As I'm coming from experience in C++, PHP, Javascript, python, java, (the list goes on), I use = in my R code. Except in one case, which I'll come to in a moment. I've used = in almost all my R-related blog posts and StackOverflow questions and answers and no-one has ever taken offence (AFAIK), shunned me (AFAIK), or done an edit to change them all to <-. So it appears to be acceptable.

Downsides

But there are three downsides I can think of, one important, one social and one bogus.

The first downside is R also uses the single equals sign for a named parameter assignment in a function call. This generally doesn't matter because using assignment in parameters is bad form in C-like languages, so I don't get confused. Just about the only time it ever matters to me is timing code. I *have* to write:

     timing = system.time( x <- do_calculation(a=1,b=2) )


If I write the following then x won't get assigned:

  timing = system.time( x = do_calculation(a=1,b=2) )

The second downside is the R community considers <- to be standard. So all packages use it, all books use it. If you want to be part of the in-crowd you have to use it too.

The third downside is it is easier to use search-and-replace to convert all your "<-" to "=", than it is to go the other way. But this is suspicious: the above example shows why you cannot do "<-" to "=" completely automatically anyway.

Upsides

Are there any upsides to preferring = to compensate? Yes, though they'll sound petty to people who believe using <- is the Only Way. First it is one less keystroke. Second, in comparisons, this does not work:

    if(x<-5)cat("x is less than minus five\n")

So you must put spaces around < and > in R; it is not just a style thing as it is in other languages.

The third reason is when I'm using <- it is communicating intent: I'm deliberately doing an assignment to a variable in the parameter list of a function call. As that is considered bad form in most languages, I like how it stands out.

I've saved the biggest upside for last: familiarity to programmers coming from any of the other C-like languages. (R is a C-like language too.)

Comparisons

You thought I'd mention ==, and how it can be confused with = ? That potential confusion exists in all C-like languages, and we just know to look out for it. And, anyway, it still exists in R:

   f(x=1)   vs.    f(x==1)

When I write one of those, did I mean the other?

Language Design

If I was designing R from scratch, how would I do it differently? I love how I can assign to named parameters in R - it is perhaps the most beautiful feature of the language. But it is an overload of the = sign, and one not found in other C-like languages, so I'd be tempted to change it to use ":" (it looks a bit like object notation in Javascript, but without the curly brackets). So the above example would become:

   timing = system.time( x = do_calculation(a:1,b:2) )

Notice how I can use "x=" safely, because it now has no other meaning. Also notice how this helps with the f(x=1)   vs.    f(x==1) confusion too. But I'm contradicting my comment above, about liking the way <- stands out. So maybe I want it to look like this:

   timing = system.time( x <- do_calculation(a:1,b:2) )

Now a lint tool can warn about use of a single equals sign in a function call, because there should never be one. Hhhmmm, I'm not convinced but it is something to chew on.

Do you have any thoughts or constructive criticism? Please add a comment.

They are SPOFs hiding everywhere!

One of my test systems sent me a few hundred emails between 2:38 and 7:45am JST. Just a test server so it didn't go to my phone and I noticed the problem around 7:30am. No paying clients were affected.

I dived straight in and found my Rackspace UK box couldn't find api.qqtrend.com. But DNS lookup worked for me. Other DNS was  working on the Rackspace box. I also found it couldn't ping the DNS server (at GoDaddy). But I could from my office LAN. And I also could from a U.S. server.

Conclusion (totally wrong - see below): Rackspace UK data centre had issues.

So I logged in to Rackspace, to check for alerts, and post a support ticket. There was a mention, phrased very vaguely, saying "someone else" has DNS problems. Hhmmm. By this time I could ping the GoDaddy server, but DNS lookup still failed. Uncertain I pinged around a bit more, and by that time DNS had started working.

I.e. The underlying problem had already been fixed, it was just taking time to spread through the internet, and I could have "fixed it" with no effort by just staying in bed 15 minutes longer. Oh well.

Anyway, it turns out it was a sociopath attacking GoDaddy: http://www.bbc.co.uk/news/business-19549367


Here are his reasons:
    "i'm taking godaddy down bacause well i'd like to test how the cyber security is safe and for more reasons that i can not talk now." (sic)

OK, English may not be his first language. But even allowing for that, he is not coming across as an upstanding member of society. Not protesting, no cause, just wanting to see if he had the skills to annoy a lot of people. This is a guy (gal?) who badly needs a girlfriend/boyfriend. If you know him, introduce him to someone. Please.

I know GoDaddy had a big PR screw-up by initially supporting SOPA, but they had the courage and sense to listen to people and change their position. Still a good company in my mind.

But, the silver lining is it nicely illustrated we (QQ Trend) have a Single Point Of Failure, at the DNS and registrar level, that had been overlooked. We'd previously got server1 and server2 as the two endpoints. We have them in different continents, and different cloud providers (no secret: Amazon and Rackspace). And I thought that was solidity to boast about. I was about to add server3 in a third continent as an option for customers who really, really need 100% uptime.

But what today's problem reveals is that if all three servers are on the same domain: server1.example.com, server2.example.com, server3.example.com, then we have a potential issue with DNS, and even with the registrar.

I think we need an alternative domain, at a completely different registrar, with DNS at an independent ISP. Then at the script level we add that in as one of the failover endpoints. They'll point to same three servers.

For instance, however big Amazon or GoDaddy (or any infrastructure provider) get, however many data centres they have around the world, they are still open to attacks, politics and human error inside their organization. We're service providers building on top of their infrastructure. It is our job to accept their limitations and do something about it.

Small R/XTS Code Snippets And Tips

1. Splitting by week

The split function in xts can split your data into weeks:  split(data,f="weeks")
Under the covers it uses endpoints, which in turn uses a C function of the same name in the XTS package. The problem I discovered it that it considers the start of the week as Monday, 00:00:00, and the end of the week as Sunday, 23:59:59. This is a problem because the FX markets start at 9pm or 10pm on a Sunday night in the UTC/GMT timezone (at Sunday, 5pm in New York timezone).

What we want is for it to treat Sunday 00:00:00 as the start of the week. That gives us good buffer on both sides (about 24hrs after Nasdaq after-market hours finish, and about 21hrs before the FX markets open). If you look at the source of endpoints (just type endpoints in an interactive R session) you'll see it has the magic constant: 3L * 86400L. After some trial and error I found we want that to be 4L not 3L. Though it may be brittle, as you're using XTS internals, you can use this line in your code:

    ep= .Call("endpoints", .index(data)+4L*86400, 604800L, k=k, PACKAGE="xts")

(I don't suppose the XTS package can consider this a bug, as there may be people relying on the current "weeks" behaviour; but maybe they could add a "sunweeks" option to endpoints and split? What do you think?)

P.S. A minor optimization for the XTS endpoints function:
    if (on %in% c("years", "quarters", "months", "weeks", "days"))
could read:
    if (on %in% c("years", "quarters", "months", "days"))
because weeks is not using posixltindex, and it takes CPU time to generate.

2. Put an XTS object in a string

Say you want to show the first 4 rows of an xts object in a string. The first trick you need is capture.output(). This takes the print() output and puts it in a string. But it returns each row as an entry in a vector. So we'll use paste() to convert that to a single string. Here is our code:


    msg = paste("Here are the first 5 rows:",paste(capture.output(x[1:5,]),collapse="\n") )

See item 3 for an example of this.

3. Show me just the NA rows

Here is some test data:
    library(xts)
    x=xts(  data.frame(a=c(1,2,3,NA,5,6), b=c(100,99,NA,NA,NA,95)),
            as.Date("2012-10-01")-6:1)
Which looks like:

                a   b
    2012-09-25  1 100
    2012-09-26  2  99
    2012-09-27  3  NA
    2012-09-28 NA  NA
    2012-09-29  5  NA
    2012-09-30  6  95

Imagine there should be none, and you want to print them out in an error message. This is the command to just show the rows with no NAs:

    x[complete.cases(x),]

And therefore just the rows with  NAs:


    x[!complete.cases(x),]

So, to print a fatal error message that shows the NA rows:

    if(any(is.na(x))){
      stop( paste("We have NAs:",
        paste(capture.output(x[!complete.cases(x),]),collapse="\n")
        ))
      }

(see Item 2, "Put an XTS object in a string", if the second half of that line looks scarier than a rabid dog who has just bitten through his leash.)

4. Loop through as key=>value

In many languages there is a foreach(container as key=>value) type construct. I've not found something so compact for XTS objects, so I use this:
    for(ix in 1:nrow(x)){
      datestamp=index(x)[ix]
      b=x$b[ix]  #Or, b=coredata(x$b)[ix]
      #print(datestamp);print(b)
      }

I've shown two ways of setting 'b'. The first way leaves 'b' as an XTS object. In the second way 'b' is a number. If using the latter, I'm sure it is more efficient to do the coredata() call before the loop. I.e. like this:
    xcd=coredata(x$b)
    for(ix in 1:nrow(x)){
      datestamp=index(x)[ix]
      b=xcd[ix]
      #print(datestamp);print(b)
      }

 

5. Merging, but excluding values only in one xts object

Here is some test data:
  library(xts)
  Sys.setenv(TZ = "UTC")
  d=xts(1:5,Sys.Date()+(2:6))
  ix=Sys.Date()+(1:5)

So, d is our data, whereas ix lists the days we should have data for.
ix looks like:
  [1] "2013-01-10" "2013-01-11" "2013-01-12" "2013-01-13" "2013-01-14"

and d looks like:
             [,1]
  2013-01-11    1
  2013-01-12    2
  2013-01-13    3
  2013-01-14    4
  2013-01-15    5

In other words, d is missing an entry for 2013-01-10, so we need to add an NA entry for it. But also d has an entry for 2013-01-15, which it shouldn't have yet. (In a finance context it might be a bar that we are still collecting ticks for, so we don't have final values for yet; in a business context it might be a value that has not been approved by management for release yet.)

When we do merge(d,xts(,ix),all=F) we get:
             d
  2013-01-11 1
  2013-01-12 2
  2013-01-13 3
  2013-01-14 4


When we do merge(d,xts(,ix),all=T) we get:
             d
  2013-01-10 NA
  2013-01-11  1
  2013-01-12  2
  2013-01-13  3
  2013-01-14  4
  2013-01-15  5
Neither is what we want. The solution is  merge(d,xts(,ix),join='right') which gives us:
  2013-01-10 NA
  2013-01-11  1
  2013-01-12  2
  2013-01-13  3
  2013-01-14  4

6a. Combining two xts objects that have same column but different timestamps

Here is some test data:
  library(xts)
  Sys.setenv(TZ = "UTC")
  a=xts(1:5,as.Date("2013-02-01")+1:5);colnames(a)="v"
  b=xts(0:-2,as.Date("2013-02-01")-0:2);colnames(b)="v"

When we do merge(a,b) we get:
             v v.1
 2013-01-30 NA  -2
 2013-01-31 NA  -1
 2013-02-01 NA   0
 2013-02-02  1  NA
 2013-02-03  2  NA
 2013-02-04  3  NA
 2013-02-05  4  NA
 2013-02-06  5  NA

You can play with various values for the join parameter, but it won't help. The solution is  rbind(a,b):
             v
 2013-01-30 -2
 2013-01-31 -1
 2013-02-01  0
 2013-02-02  1
 2013-02-03  2
 2013-02-04  3
 2013-02-05  4
 2013-02-06  5

What if both a and b have a timestamp in common? Then you get two rows. In that case you can remove the duplicates afterwards:
  x=rbind(a,b)
  x=x[!duplicated(index(x)),]

The duplicate from b is the one that gets removed. If you wanted the one from a to be removed instead, then pass fromLast = TRUE to duplicated(), or simply do rbind(b,a) !

6b. Combining two xts objects that have same columns but different timestamps

This follows on from 6a, but when the xts object have more than one column. Here is some test data:
  library(xts)
  Sys.setenv(TZ = "UTC")
  a=xts(data.frame(x=1:5,y=5:9),as.Date("2013-02-01")+1:5)
  b=xts(data.frame(x=0:-2,y=10:12),as.Date("2013-02-01")-0:2)

Again,  rbind(a,b) is the answer: 
             x  y
 2013-01-30 -2 12
 2013-01-31 -1 11
 2013-02-01  0 10
 2013-02-02  1  5
 2013-02-03  2  6
 2013-02-04  3  7
 2013-02-05  4  8
 2013-02-06  5  9

Mock The Socket, in PHP

I wanted to put a unit test around some PHP code that use a socket, and of course hit a problem: how do I control what a call to fgets returns? You see, in PHP, you cannot replace one of the built-in functions: you get told "Fatal error: Cannot redeclare fgets() ...".

Rename Then Override Then Rename Again!

I asked on StackOverflow, not expecting much response, but almost immediately got told about the rename_function(). Wow! I'd never heard of that before. The challenge then was that this is in the apd extension, which was last released in 2004 and does not support php 5.3. I've put instructions on how to get it installed on the StackOverflow question so I won't repeat them here.

The next challenge you'll meet is that naively using rename_function to a move a function out of the way fails. You still get told "Fatal error: Cannot redeclare fgets() ..." ?! You need to use override_function to replace its behaviour. All Done? Not quite, what you discover next is that you can only override one function. Eh?! But all is not lost: the comments in the marvellous PHP manual described the solution, which goes like:

1. Use override_function to define the new behaviour
2. Use rename_function to give a better name to the old, original function.

However, when you go to restore a function (see further down), it turns out that does not work. What you actually need to do is:

1. Use rename_function to give a name to the old, original function, so we can find it later.
2. Use override_function to define the new behaviour
3. Use rename_function to give a dummy name to __overridden__

You do those three steps for each function you want to replace. Here is a complete example that shows how to override fgets and feof to return strings from a global array. NOTE: this is a simplistic example; I should really be overriding fopen and fclose too (they'd be set to do nothing).


$GLOBALS['fgets_strings']=array(
    "Line 1",
    "Line 2",
    "Line 3",
    );

rename_function('fgets','real_fgets');
override_function('fgets','$handle,$length=null','return array_shift($GLOBALS["fgets_strings"])."\n";');
rename_function("__overridden__", 'dummy_fgets'); 

rename_function('feof','real_feof');
override_function('feof','$handle','return count($GLOBALS["fgets_strings"])==0;');
rename_function("__overridden__", 'dummy_feof');

$fname="rename_test.php";
$fp=fopen($fname,"r");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }
fclose($fp);

Mock The Sock(et)

So, what about the original challenge, to unit test a socket function? Here is some very minimal code to request a page from a web server; let's pretend we want to test this code:

$fp=fsockopen('127.0.0.1',80);

fwrite($fp,"GET / HTTP/1.1\r\n");
fwrite($fp,"Host: 127.0.0.1\r\n");
fwrite($fp,"\r\n");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }

fclose($fp);

To take control of its behaviour, we prepend the following block; the above code does not have to be touched at all.

rename_function('fwrite','real_fwrite');

override_function('fwrite','$fp,$s','');
rename_function("__overridden__", 'dummy_fwrite');

rename_function('fsockopen','real_fsockopen');
override_function('fsockopen',
    '$hostname,$port=-1,&$errno=null,&$errstr=null,$timeout=null',
    'return fopen("socket_mock_contents.txt","r");'
    );
rename_function("__overridden__", 'dummy_fsockopen');

I.e. We replace the call to fsockopen with a call to fopen, and tell it to read our special file. (If you don't want to use an external file, and instead want a fully self-contained test, with the contents in a string, you could use phpstringstream or if you don't want Yet Another Dependency, you could write your own, as the code is fairly short and straightforward )

The other thing to note about the above code is that we had to replace fwrite as well. This is needed because we're creating a read-only stream to be the stand-in for a read-write stream. If you are using other functions (e.g. ftell or stream_set_blocking) you will need to consider if those functions need a mock version too.

The Fly In The Ointment

The thing about replacing a global function is that you're replacing a global function, as in, globally! Any other code that calls that function is going to call your mock version. Maybe we can get away with this with fsockopen, but it becomes quite a major problem if you are replacing things like fgets or fwrite, in a phpUnit unit test, as phpUnit is quite likely to call those functions itself!

So, we want to restore the functions once we're done with them? It is very easy to get this wrong and get a segfault. You must use rename_function before using override_function, the first time. The following code has been tested and restores the behaviour of the above example:

override_function('fwrite','$fp,$s','return real_fwrite($fp,$s);');
rename_function("__overridden__", 'dummy2_fwrite');

override_function('fsockopen','$hostname,$port','return real_fsockopen($hostname,$port);');
rename_function("__overridden__", 'dummy2_fsockopen');

Notice how we still need to deal with __overridden__ each time.

Food For Thought

There is another approach, which might be more robust. It involves checking inside each overridden function if this is the stream you want to be falsifying data for, and whenever it isn't you call the original versions. Here I'll show how to just do that with the fwrite function:

$GLOBALS["mock_fp"]=null;

rename_function('fwrite','real_fwrite');
override_function('fwrite','$fp,$s','
if($fp==$GLOBALS["mock_fp"]){echo "Skipping a fwrite.\n";return;}   //Do nothing
return real_fwrite($fp,$s);
');
rename_function("__overridden__", 'dummy_fwrite');

rename_function('fsockopen','real_fsockopen');
override_function('fsockopen','$hostname,$port=-1,&$errno=null,&$errstr=null,$timeout=null',
    'return $GLOBALS["mock_fp"]=fopen("socket_mock_contents.txt","r");');
rename_function("__overridden__", 'dummy_fsockopen');

Then we can test it, as follows:
$fp=fsockopen('127.0.0.1',80);

fwrite($fp,"GET / HTTP/1.1\r\n");
fwrite($fp,"Host: 127.0.0.1\r\n");
fwrite($fp,"\r\n");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }

fclose($fp);

$fp=fopen("tmp.txt","w");
fwrite($fp,"My output\n");
fclose($fp);