Showing posts with label php. Show all posts
Showing posts with label php. Show all posts

Thursday, February 5, 2015

PHP Sending mails twice?? (doing stuff twice)

Just had a mammoth troubleshooting session, because a very simple PHP script to send an email kept sending them twice. I’d only just configured postfix to send email, so I kept looking for problems there. I kept staring at the PHP code, and could see no problem, but it looked more and more like PHP was calling postfix twice. Commenting php.ini settings in and out made no difference. This was commandline PHP, so nothing to do with browser reloads, or anything like that.

Then I had the brainwave to append a random number to the bottom of the body text. Different numbers; in fact, not just that, but the 2nd email got both numbers! So it is definitely my PHP script. But I still couldn’t see it.

Stripped down, so the problem is more obvious, it looked like this:

$bodyText = "Whatever";

class Test{

function test(){
$bodyText.="RANDOM=".mt_rand(1,1000);
mail("me@example.com", "Test", $bodyText);
}
}

$R = new Test;
$R->test();

I’ve been spending too much time jumping between languages. And I’d also got used to PHP constructors being called __construct() and forget it still offered backwards compatibility for using the class name. Yep, that’s right, PHP functions (and classes) are case-insensitive, and so test() was being treated as the constructor of class Test. So one mail was being sent from the constructor, the second from my explicit function call. Grrrr….

(The above is also a possible explanation for problems like “PHP calls web service twice”, or “PHP has double log entries” or “PHP does something twice”!)

Written with StackEdit.

Friday, June 20, 2014

Captcha never changed (securimage)

It took a while of testing before I noticed, then I finally realized the captcha on a web app I was working never changed!

Thankfully, before launching into some heavy-duty troubleshooting, I had a flash of inspiration: to make my unit tests work nicely I was setting the random seed to a constant (i.e. to get repeatable test data). That test data is actually being generated for every page request of the main web app (just during development).

So the fix was as trivial as calling mt_srand(time());  just before creating the Captcha  (as I'm using the Securimage library, just before the $img = new Securimage(); call).

If only all troubleshooting was this quick!

Tuesday, November 12, 2013

Just discovered a new way to loop through months, up to the current month. I used to have horrible code like this:

for($y=2009;$y<=date("Y");++$y){
  for($m=1;$m<=12;++$m){
    if($y==date("Y") && $m>=date("m"))continue;
    $t = strtotime("$Y-$m-01");
    echo date("M_Y",$t)."\n";
    }
  }


If wanting days, you get $t then do $days_in_month = date("t",$t);

Here is my new solution:

$start = strtotime("2009-01-15");
$now = time();
$secs_per_month=(int)((365.25*86400)/12);
for($t=$start;$t<$now;$t+=$secs_per_month){
    echo date("M_Y",$t)."\n";
    }


I.e. choose a day in the middle of the month, and do everything in seconds. Fewer calls to date() and shorter code. Obvious in hindsight!



Friday, September 6, 2013

PHP: A jump in performance for free

Not often I just post a link to another blog, but Lorna Jane has some very interesting numbers on PHP versions:
  http://www.lornajane.net/posts/2013/php-version-adoption

But what is even more interesting is the benchmarks of each PHP version which I'm going to reproduce here:
  • 5.2.17: 3.77 seconds
  • 5.3.23: 2.63 seconds
  • 5.4.15: 1.98 seconds
  • 5.5RC1: 2.11 seconds

Assuming that benchmark accurately reflects your own application, then your code on php 5.4 will run in 75% of the time it previously took on 5.3.  (PHP 5.5 appears to only be 80% of php 5.3 speed; not sure if the slowdown was due to working with a release candidate, rather than the final release?)

(PHP 5.5 stops supporting Windows XP and Windows 2003, and adds nothing that caught my interest. PHP 5.4 on the other hand had some interesting features, and now I see a worthwhile speed-up I will target that for new projects.)



Monday, February 4, 2013

Making sense of vfsStream

vfsStream is a very cool PHP library that abstracts the file system. Its primary use case is in unit tests, as a way to mock file system activity, and it integrates nicely with PHPUnit, even being mentioned in the PHPUnit manual.

Sadly the documentation is a bit lacking, so this article will try provide some middle ground behind the unexplained usage example you'll find in a few places, and the dry API docs. (In fact skip the API docs completely - the source is more understandable.)

I'm going to show this completely outside PHPUnit, as that clouds the issues. Let's start with the include we need:
   require_once 'vfsStream/vfsStream.php';

(Aside: all tests here are being done with version 0.12.0, installed using pear.)

Here is our minimal example:

   vfsStream::setup('logs');
   file_put_contents(vfsStream::url('logs/test.log'),"Hello\n");


The first line creates our filesystem.
The second line creates a file called "test.log", and puts a string inside it.

There is something I want to emphasize here, as it confused me no end. We have not created a directory called "logs". We have created a virtual file system called "logs". We have no sub-directories at all. test.log lives in the root of the filesystem.

Here is our troubleshooting tool:
   require_once 'vfsStream/visitor/vfsStreamStructureVisitor.php';
   print_r(vfsStream::inspect(new vfsStreamStructureVisitor())->getStructure());

It outputs:
    Array
    (
        [logs] => Array
            (
                [test.log] => Hello
            )
    )


(Yes, I know it still looks like logs is a directory name there too. It isn't.)

Another way to troubleshoot this is using PHP functions directly:
    $dir=dir("vfs://logs");
    while(($entry=$dir->read())!==false)echo $entry."\n";


This outputs:
    test.log

Note: you cannot use "vfs://" as a url to get the root. This is like trying to access the root of a website with "http://". You need to use "http://example.com/" and you need to use "vfs://logs" for a virtual filesystem. As I said, I found this confusing, so I prefer to rewrite my above examples as follows (this also shows the complete code):

   
    require_once 'vfsStream/vfsStream.php';
    vfsStream::setup('_virtualroot_',null,array('logs'=>array()));
    file_put_contents(vfsStream::url('_virtualroot_/logs/test.log'),"Hello\n");
    print_r(vfsStream::inspect(new vfsStreamStructureVisitor())->getStructure());
    $dir=dir("vfs://_virtualroot_");
    while(($entry=$dir->read())!==false)echo $entry."\n";

    ?>

This time we do have a directory called "logs", which is in the root of a virtual file system called "_virtualroot_". You may hate that approach for its horrible verbosity. Me? I like it.

A few random other points about vfsStream:
  • chdir("vfs://logs") does not work. The Known Issues page lists a few more.
  • vfsStream::url("logs/test.log") simply returns "vfs://logs/test.log".
  • fopen($fname,"at") fails. The "t" is not supported. You have to change your code to use "a" or "ab" (aside: "ab" is better, as "a" could work differently on different systems).
  • Calling vfsStream::setup  a second time removes the previous filesystem, even if you use a different virtual filesystem name.
  • I've not worked out how to use the git version. It may be that my above examples do not work with the very latest version. If I work it out, and that is the case, I'll post additional examples.

Sunday, July 1, 2012

Mock The Socket, in PHP

I wanted to put a unit test around some PHP code that use a socket, and of course hit a problem: how do I control what a call to fgets returns? You see, in PHP, you cannot replace one of the built-in functions: you get told "Fatal error: Cannot redeclare fgets() ...".

Rename Then Override Then Rename Again!


I asked on StackOverflow, not expecting much response, but almost immediately got told about the rename_function(). Wow! I'd never heard of that before. The challenge then was that this is in the apd extension, which was last released in 2004 and does not support php 5.3. I've put instructions on how to get it installed on the StackOverflow question so I won't repeat them here.

The next challengyou'll meet is that naively using rename_function to a move a function out of the way fails. You still get told "Fatal error: Cannot redeclare fgets() ..." ?! You need to use override_function to replace its behaviour. All Done? Not quite, what you discover next is that you can only override one function. Eh?! But all is not lost: the comments in the marvellous PHP manual described the solution, which goes like:
  1. Use override_function to define the new behaviour
  2. Use rename_function to give a better name to the old, original function.
However, when you go to restore a function (see further down), it turns out that does not work. What you actually need to do is:
  1. Use rename_function to give a name to the old, original function, so we can find it later.
  2. Use override_function to define the new behaviour
  3. Use rename_function to give a dummy name to __overridden__
You do those three steps for each function you want to replace. Here is a complete example that shows how to override fgets and feof to return strings from a global array. NOTE: this is a simplistic example; I should really be overriding fopen and fclose too (they'd be set to do nothing).


$GLOBALS['fgets_strings']=array(
    "Line 1",
    "Line 2",
    "Line 3",
    );

rename_function('fgets','real_fgets');
override_function('fgets','$handle,$length=null','return array_shift($GLOBALS["fgets_strings"])."\n";');
rename_function("__overridden__", 'dummy_fgets'); 

rename_function('feof','real_feof');
override_function('feof','$handle','return count($GLOBALS["fgets_strings"])==0;');
rename_function("__overridden__", 'dummy_feof');

$fname="rename_test.php";
$fp=fopen($fname,"r");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }
fclose($fp);


Mock The Sock(et)


So, what about the original challenge, to unit test a socket function? Here is some very minimal code to request a page from a web server; let's pretend we want to test this code:

$fp=fsockopen('127.0.0.1',80);

fwrite($fp,"GET / HTTP/1.1\r\n");
fwrite($fp,"Host: 127.0.0.1\r\n");
fwrite($fp,"\r\n");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }

fclose($fp);

To take control of its behaviour, we prepend the following block; the above code does not have to be touched at all.

rename_function('fwrite','real_fwrite');
override_function('fwrite','$fp,$s','');
rename_function("__overridden__", 'dummy_fwrite');

rename_function('fsockopen','real_fsockopen');
override_function('fsockopen',
    '$hostname,$port=-1,&$errno=null,&$errstr=null,$timeout=null',
    'return fopen("socket_mock_contents.txt","r");'
    );
rename_function("__overridden__", 'dummy_fsockopen');

I.e. We replace the call to fsockopen with a call to fopen, and tell it to read our special file. (If you don't want to use an external file, and instead want a fully self-contained test, with the contents in a string, you could use phpstringstream or if you don't want Yet Another Dependency, you could write your own, as the code is fairly short and straightforward )

The other thing to note about the above code is that we had to replace fwrite as well. This is needed because we're creating a read-only stream to be the stand-in for a read-write stream. If you are using other functions (e.g. ftell or stream_set_blocking) you will need to consider if those functions need a mock version too.


The Fly In The Ointment

The thing about replacing a global function is that you're replacing a global function, as in, globally! Any other code that calls that function is going to call your mock version. Maybe we can get away with this with fsockopen, but it becomes quite a major problem if you are replacing things like fgets or fwrite, in a phpUnit unit test, as phpUnit is quite likely to call those functions itself!

So, we want to restore the functions once we're done with them? It is very easy to get this wrong and get a segfault. You must use rename_function before using override_function, the first time. The following code has been tested and restores the behaviour of the above example:

override_function('fwrite','$fp,$s','return real_fwrite($fp,$s);');
rename_function("__overridden__", 'dummy2_fwrite');

override_function('fsockopen','$hostname,$port','return real_fsockopen($hostname,$port);');
rename_function("__overridden__", 'dummy2_fsockopen');

Notice how we still need to deal with __overridden__ each time.

Food For Thought

There is another approach, which might be more robust. It involves checking inside each overridden function if this is the stream you want to be falsifying data for, and whenever it isn't you call the original versions. Here I'll show how to just do that with the fwrite function:

$GLOBALS["mock_fp"]=null;

rename_function('fwrite','real_fwrite');
override_function('fwrite','$fp,$s','
if($fp==$GLOBALS["mock_fp"]){echo "Skipping a fwrite.\n";return;}   //Do nothing
return real_fwrite($fp,$s);
');
rename_function("__overridden__", 'dummy_fwrite');

rename_function('fsockopen','real_fsockopen');
override_function('fsockopen','$hostname,$port=-1,&$errno=null,&$errstr=null,$timeout=null',
    'return $GLOBALS["mock_fp"]=fopen("socket_mock_contents.txt","r");');
rename_function("__overridden__", 'dummy_fsockopen');


Then we can test it, as follows:
$fp=fsockopen('127.0.0.1',80);

fwrite($fp,"GET / HTTP/1.1\r\n");
fwrite($fp,"Host: 127.0.0.1\r\n");
fwrite($fp,"\r\n");

if($fp)while(!feof($fp)){
    echo fgets($fp);
    }

fclose($fp);

$fp=fopen("tmp.txt","w");
fwrite($fp,"My output\n");
fclose($fp);

Monday, May 28, 2012

php-webdriver bindings for selenium: how to add time-outs

Not all webpages finish loading. In particular I've a page that keeps streaming data back to the client, and never finishes. (For instance it might be used from an ajax call.) I want to test this from Selenium, but have been hitting problems. The main problem is Selenium's get() function, which is used to fetch a fresh URL, does not return until the page has finished loading [1]. In my case that meant never, and so my test script locked up!

However all is not lost; you can specify a page load timeout. It is hidden in the protocol docs, but I've added it to the php webdriver library I use (v0.9). See the three functions below [2]; just paste them in to the bottom of WebDriver.php.

I also needed one bug fix in WebDriver.php's public function get($url). It currently ends with:
    $response=curl_exec($session);

Just after that line you should add this:
    return $this->extractValueFromJsonResponse($response);


The time-out, and that bug fix, can be used like this:

require_once "/usr/local/src/selenium/php-webdriver-bindings-0.9.0/phpwebdriver/WebDriver.php";
$webdriver = new WebDriver("localhost", "4444");
$webdriver->connect("firefox");
$webdriver->setPageLoadTimeout(2000);   //2 seconds
$url="http://example.com/forever.php"; //A page that never finishes loading
$obj=$webdriver->get($url);
if($obj===null){
    $current_url=$webdriver->getCurrentUrl();
    if(!$current_url){
        //Selenium-server not running
        }
    else{
        //It worked! (it completed loading in under two seconds)
        }
    }
elseif($obj->class=='org.openqa.selenium.TimeoutException'){
    //It timed out
    }
elseif($obj->class=='org.openqa.selenium.remote.UnreachableBrowserException'){
    //Browser was closed (or selenium-server was shutdown)
    }
else{
    echo "FAILED:";print_r($obj);
    }

This is useful stuff. There is still one problem left for me: I wanted to load two seconds worth of data and then look at it. But I cannot. The browser refuses to listen to selenium while it is loading a page! So though get() returned control to my script after two seconds, I cannot do anything with that control (except close the browser window), because the URL is still actually loading. And it will do that forever!!  (I've played with an interesting alternative approach, which also fails, but suggests that a solution is possible. But that is out of the scope of this post, which is to show how to add the time limit functions to php-webdriver-bindings.)

[1]: This is browser-specific behaviour, not by Selenium design. Firefox and Chrome, at least, behave this way.


[2]: Consider this code released, with no warranty, under the MIT license, and permission granted to use in the php-webdriver-bindings project with no attribution required.

    /**
     * Set wait for a page to load.
     *
     * This timeout is for the get() function. (Firefox and Chrome, at least, won't return from get()
     * until a page is fully loaded.  If remote server is streaming content, they would never return
     * without this time-out.)
     *
     * @param Number $timeout Number of milliseconds to wait.
     * @author Darren Cook, 2012
     * @internal http://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/timeouts
     */
    public function setPageLoadTimeout($timeout) {
        $request = $this->requestURL . "/timeouts";       
        $session = $this->curlInit($request);
        $args = array('type'=>'page load', 'ms' => $timeout);
        $jsonData = json_encode($args);
        $this->preparePOST($session, $jsonData);
        curl_exec($session);       
    }

    /**
     * Set wait for a script to finish.
     *
     * @param Number $timeout Number of milliseconds to wait.
     * @author Darren Cook, 2012
     * @internal http://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/timeouts
     */
    public function setAsyncScriptTimeout($timeout) {
        $request = $this->requestURL . "/timeouts";       
        $session = $this->curlInit($request);
        $args = array('type'=>'script', 'ms' => $timeout);
        $jsonData = json_encode($args);
        $this->preparePOST($session, $jsonData);
        curl_exec($session);       
    }
    /**
     * Set implict wait.
     *
     * This is for waiting for page elements to appear. Not useful for scripts or
     * waiting for the initial get() call to time out.
     *
     * @param Number $timeout Number of milliseconds to wait.
     * @author Darren Cook, 2012
     * @internal http://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/timeouts
     */
    public function setImplicitWaitTimeout($timeout) {
        $request = $this->requestURL . "/timeouts";       
        $session = $this->curlInit($request);
        $args = array('type'=>'implicit', 'ms' => $timeout);
        $jsonData = json_encode($args);
        $this->preparePOST($session, $jsonData);
        curl_exec($session);       
    }

Sunday, November 6, 2011

PHP, Proxies, HTTPS: v2, v3 or v23?!

From a PHP http client, using HTTPS via a proxy, I started getting a "400 bad request" error from Apache. I knew it could work because it worked last week. The apache error log message was:
   Hostname 127.0.0.1 provided via SNI and hostname mytest.local provided via HTTP are different

My first troubleshooting mistake was messing around with server-side settings: I found out what SNI meant, but as far as I could see I wasn't using it. Then, finally, I remembered I could use curl as a test http client, and it was working fine. I removed all my server-side changes and curl was still connecting fine. So now I knew I'd broken something client-side. I added the -v flag to curl to see the exact headers it is sending. We're both sending the same headers.

Finally I remembered I'd changed this line:
  stream_socket_enable_crypto($fp,true,STREAM_CRYPTO_METHOD_SSLv23_CLIENT);

The PHP docs give no guidance on which option to choose, but "v23" sounded like it would work with version 2 or version 3, and maybe do all kinds of auto-negotiation behind the scenes. Which had to be a good thing. I'm sure I'd tested after changing, but I must have tested non-HTTPS or without the proxy by mistake. When I changed back to this line, everything worked again:
  stream_socket_enable_crypto($fp,true,STREAM_CRYPTO_METHOD_SSLv3_CLIENT);

I hope that helps someone, as google was no use for me (all the hits about the SNI name and HTTP name difference were due to Apache being case-sensitive about the name comparison, which was not the problem here).

By the way if you want to know how to do http connections, with a proxy, using PHP, supporting both HTTP and HTTPS, I've described it here.

Saturday, October 22, 2011

Frapi (PHP web service API system)

I read about Frapi in PHP Architect (May 2011), and spent a couple of hours trying it out. It is quite interesting, but I don't think I will be using it. It is a full web-interface for making the API. This is what makes it cool, but also its biggest disadvantage. There is a lot of code involved, meaning there is a lot to learn if you need to change it and lots of places for bugs and security exploits to crop up.

It comes with a documentation generator, which could be really useful. This feature is still incomplete (for instance there are no links to it yet, see here, PDF generation didn't work properly), but it looks okay.

There is one specific limitation: I could not create an action with an optional parameter, at least not using the router.  E.g. if my action is called "ddd", then I can call /ddd (no parameter). But I cannot call "/ddd/77". (I can give an optional parameter with /ddd?id=77). Or I can define a router as "/ddd/:id" so that I can call "/ddd/77". But in that case id is now required and I cannot use "/ddd".

Another disadvantage is no built-in support for oAuth. (I did find an oauth extension for Frapi but did not try it as the integration seems quite rough still.)

Incidentally if you were looking for a full application built on Zend Framework this may make the perfect study case. Apparently only the admin interface uses ZF, and the actual web services do not; but as far as I can tell they are closely tied and you need to keep both together even on your production servers.

Overall, because making a web service in PHP is not that hard, the advantages are slight and not enough to outweigh the disadvantages (large codebase, inflexible structure, etc.).

Friday, September 2, 2011

Using twitter oauth from commandline

The twitteroauth library is the most recommended PHP library for using oauth for API access to Twitter. It also supports the commandline approach (which can work completely behind a firewall, no need for a web server to host a callback page), but it is not very well documented.
(Note: when I say behind a firewall you do still need web access, because you need to login to twitter to get a PIN code; but this is much less demanding than needing to set up a web server on a public IP address.)

Anyway, after I'd worked it out, I put my sample code up on github. The three files to look at are oob1.php, oob2.php and oob3.php. Here is how you use them:

Step One:

Edit config.php, to set:
      define('OAUTH_CALLBACK', 'oob');

(If not already done, go to https://dev.twitter.com/apps, create and configure the app, and add the consumer key and secret to config.php.)

Step Two:
   php oob1.php

Step Three:
Visit the URL it tells you to, and approve the application

Step Four:
   php oob2.php 1234567
(where 1234567 is the PIN number you got at the end of step three)

Step Five:
   php oob3.php account/verify_credentials
(this command will show your account; see the oob3.php source code for other supported commands, and some shortcuts.)

Let me know if you have any problems with, or questions about, these files.
If you find bugs, or want to encourage its inclusion in the main twitteroauth library, you can comment on the github pull request.

Sunday, July 3, 2011

Apache: use both PHP module and PHP-CGI

I had a need the other day to configure apache so it uses the PHP Apache module for all directories except one, where I wanted it to use the cgi version of PHP. The apache module is more efficient, but runs as part of the apache process. I wanted one URL to run in its own process. (I was troubleshooting and had a hunch this might help; luckily it did in this case!) Instructions for both Linux and Windows follow.

On Ubuntu I needed some preparation: 1. install the php5-cgi package (different from php5-cli, which is what we use when using php from the commandline); 2. enable the actions module.
On Windows I was using xampp, which follows an Everything-But-The-Kitchen-Sink philosophy, and all I needed was already there.

Here is the code I added (to my VirtualHost). First for Ubuntu:
ScriptAlias /usephpcgi /usr/bin
Action  application/x-httpd-php-cgi  /usephpcgi/php5-cgi
And then for Windows XAMPP:
ScriptAlias /usephpcgi C:/xampp/php/
Action  application/x-httpd-php-cgi  /usephpcgi/php-cgi.exe
(That is copy and paste from working configurations, so I think the trailing slash must be optional!)

Then for both Apache and Windows:
AddType application/x-httpd-php-cgi .phpcgi
Now, I have to admit my dirty secret: I cheated. Instead of enabling php-cgi for all *.php in one directory, I left *.php going to the Apache PHP module, and I created the *.phpcgi extension to use the cgi binary. Initially this was simply because I managed to get it working that way; but on reflection I realized I preferred it: I can switch a script between using php module and php-cgi just by changing the extension; also I can use php-cgi anywhere in my VirtualHost. If that does not sound so useful I should explain the script in question is already hidden between an Alias, something like this, so no public-facing URLs need to change:
AliasMatch ^/admin/(.*?)/(.*)$ /path/to/admin.phpcgi

What about my original plan to configure it for a directory, without changing the file extensions? I had trouble with this, and gave up on this; but Ben kindly left a comment, so now I know how to do it. First, you still need the ScriptAlias and Action directives shown above. Then it is simply this.
<Directory /var/www/path/to>
    <FilesMatch "\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php-cgi
    </FilesMatch>
</Directory>
As Ben explains (see comment#1 below) the reason we need the FilesMatch inside the Directory is because mod-php is setting a global FilesMatch directive; and that takes priority over our attempts to use AddType or AddHandler for a directory.

Thursday, June 16, 2011

PHP PDO: so hard to debug

I wrote a simple PDO helper function to update fields in a certain database table. The fields are given as the key/value pairs in $d, and my function looked like this:
$q='UPDATE MyTable SET lastupdate=:lastupdate';
foreach($d as $key=>$value)$q.=', '.$key.'=:'.$key;
$q.=' WHERE username=:username';

$statement=$dbh->prepare($q);
foreach($d as $key=>$value)$statement->bindParam(':'.$key,$value);
$statement->bindParam(':lastupdate',date('Y-m-d H:i:s'));
$statement->bindParam(':username',$username);
It all looks reasonable doesn't it? Create the SQL, then assign the values to. But it didn't work. My $d array looked like:
array( 'status'=>'expired', 'mode'=>'' )
Instead of getting set to expired, the status field ended up blanked out. Yet lastupdate and username got set. This had me scratching my head for ages.
PDO has a debug function that is next to useless: it tells you the parameters, but not the values you've assigned to them. Incredibly annoying.

Have you spotted my bug yet?

Here's the answer. Though all the examples in the PHP documentation use bindParam(), the function to assign a value is bindValue(). You should always use bindValue(), unless you actually need the advanced functionality that bindParam() gives you. What advanced functionality you wonder? Instead of assigning the value immediately, it attaches a reference, and uses the final value of that reference variable. You're ahead of me: in my foreach loop the $value variable changes on each iteration.

If PDO had a decent debug function I'd have discovered that in half the time. Oh well, now I know!

Tuesday, May 24, 2011

PHP, PDO, SQLite, mysterious lock problem

Let's start with the conclusion:
If doing a prepare() or query() with PDO and sqlite, and then you want to do something else with sqlite in the same PHP function then unset the first PDOStatement, before trying to do that something else.

As an example here is my code, to get a unique ID ($dbh is a PDO connection to an sqlite):

function get_next_id($dbh){
$q='SELECT next FROM MyNextId';
$obj=$dbh->query($q); //Throws on error
$d=$obj->fetch(PDO::FETCH_NUM);
$next_id=$d[0];

$q='UPDATE MyNextId SET next=next+1';
$row_count=$dbh->exec($q); //Throws on error
if($row_count==0)throw new Exception("Failed to execute ($q)\n");

return $next_id;
}

It works on Ubuntu 10.04, with sqlite 3.6.22, but fails on Centos 5.6, with sqlite 3.3.6, with this message:
exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 6 database table is locked'

I went through the whole changelog from 3.3.6 to 3.6.22, but got no clues (though I am now impressed with how active and organized the sqlite development is). But finally I tracked down this article on someone getting similar errors.

And that was it. I could have used $obj->closeCursor(), but deleting $obj is just as good:

function get_next_id($dbh){
$q='SELECT next FROM MyNextId';
$obj=$dbh->query($q); //Throws on error
$d=$obj->fetch(PDO::FETCH_NUM);
$next_id=$d[0];
unset($obj);

$q='UPDATE MyNextId SET next=next+1';
$row_count=$dbh->exec($q); //Throws on error
if($row_count==0)throw new Exception("Failed to execute ($q)\n");

return $next_id;
}

If you are doing just one PDO action per function then there is no need, because exiting the function will automatically do the unset.

(I don't know why this is a problem on sqlite 3.3.6 but not sqlite 3.6.22... in fact, I suspect it may be due a difference in the PDO or PHP version or configuration instead. Apologies for the loose end!)

Sunday, May 22, 2011

oauth for php (and Ubuntu)

There is a "standard" PHP OAuth library, documented in the manual, and which is installed via pecl. There is no package shortcut under Ubuntu; you still have to use pecl
sudo pecl install oauth

If you get a complaint about no "phpize", install the "php5-dev" ubuntu package. And then if you get an error in "php_pcre.h" when compiling, then you need to install the "libpcre3-dev" ubuntu package.

Finally, you need to enable it. Still as root:
cd /etc/php5/conf.d
echo "extension=oauth.so" >oauth.ini

(This creates a config file just for oauth; you could also simply put the extension line in php.ini.)

Finally, "php -m" should list "OAuth", and you can create a OAuth object in your php scripts.

Ubuntu package manager lists "liboauth-php"; the minimal information, and the lack of mention of pecl, should have given me the clue that it is something different.

Also different is this: http://code.google.com/p/oauth-php/

Monday, February 7, 2011

ZendForm: inserting static text

I'm not a fan of Zend Form, or Zend Framework generally: too much structure, too much work to do anything slightly unusual. Trying to add static text to your form epitomizes this. Note, you have to work through the $form object, as in the view page this is all there is:
  <?=$form ?>

If you are using php 5.3, there is a quite reasonable solution:
  $form->addElement('text','whatever_id',array(
    'decorators'=>array(array('Callback',array('callback'=>
    function(){return "<h3>Whatever</h3>";}))),
    ));

Yes, that is as minimal as I can get it, and yes, you must give a 'whatever_id' that is unique on your form, but you have complete control over what is output, and nothing else gets output.

If you are using php 5.2, there are two approaches. The first way is equivalent to the above, just slightly more verbose:
  $form->addElement('text','whatever_id',array(
    'decorators'=>array(array('Callback',array('callback'=>
      create_function('','return "<h3>Whatever</h3>";')
      ))),
    ));

The second way is a different approach:
  $form->addElement('text','whatever_id',array(
     'label'=>E('<h3>Whatever</h3>'),
     'helper'=>'formNote',
     'decorators'=>array(array('Label',array('escape'=>false))),
      ));

Yes, four lines, and every line matters. Here is what gets output:
  <label class="optional" for="whatever_id"></label><h3>Whatever</h3>

I cannot stop the <label> tag appearing; If you can, let me know. (It does not appear to the end user, just in the HTML source, so this is not a serious issue.)

(2011-03-02: Edited to show the create_function() approach in php 5.2)

Sunday, November 28, 2010

Troubleshooting php and xinetd with strace

In a previous entry I mentioned I was trying to track down a difference in a php script between two machines. I've now got closer, close enough in fact to fix the problem but not to understand it. As a troubleshooter, the big discovery for me was how useful strace can be.

First, a PHP code snippet:
while(1){
stream_set_blocking(STDIN,true);
$s=trim(fgets(STDIN));
if($GLOBALS['log_fp'])fwrite($GLOBALS['log_fp'],$s."\n");
$params=explode(" ",$s);
$command=array_shift($params);
...
}

The first two lines say listen on stdin for a command, and wait forever for it to arrive. When it does we log it and then split it up into a command and parameters.

This script is used over xinetd. Xinetd listens on a port for us, and when it gets something it feeds it to stdin; php script output is then fed back over the socket to the client.

Here is the setup:
  • machine A: ubuntu 8.04, 32-bit, php 5.2
  • machine B: ubuntu 10.04, 64-bit, php 5.3

Both machines run an indentical version of xinetd: "xinetd Version 2.3.14 libwrap loadavg".

Machine A runs fine, machine B sometimes sits there. Analyzing this some more, by connecting over telnet to machine B:
  1. I send a command
  2. It replies straightaway
  3. After 60 seconds or so it gives an error message
  4. If I do nothing this error message appears again every 60 seconds.

On machine A, or if I run the php script directly on machine B, it behaves like this:
  1. I send a command
  2. It replies straightaway
  3. It sits there forever waiting for input

Out of desperation I ran strace (using -p you can attach it to a running process; use ps -a | grep php to find the PID), and I was pleasantly surprised to see it was not too verbose.

Over xinetd on machine B (the problem configuration) the interesting snippet is:
...
write(1, "mydata\n", 7)                 = 7
fcntl(0, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(0, F_SETFL, O_RDWR)               = 0
poll([{fd=0, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 0 (Timeout)
write(3, "\n", 1)                       = 1
write(5, "\n", 1) 
...

On machine A over xinetd it instead looks like this:
...
write(1, "mydata\n", 7)                 = 7
fcntl64(0, F_GETFL)                     = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl64(0, F_SETFL, O_RDWR|O_LARGEFILE) = 0
read(0,
(Yes nothing comes after the "0," until I send some input.)

And direct access to the php script on machine B it practically the same:
...
write(1, "mydata\n", 7)                 = 7
fcntl(0, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl(0, F_SETFL, O_RDWR|O_LARGEFILE)   = 0
read(0,

So, the cause is clear: fgets() blocks for only 60 seconds instead of blocking forever, but only on this machine and only over xinetd. And the bug in my script then becomes clear: I'm not prepared to handle blank input!

Here is my fix:
while(1){
stream_set_blocking(STDIN,true);
$s=trim(fgets(STDIN));
if($s=='')continue;
if($GLOBALS['log_fp'])fwrite($GLOBALS['log_fp'],$s."\n");
$params=explode(" ",$s);
$command=array_shift($params);
...
}


As an ironic postscript, the real problem was elsewhere (a mismatch in another program version and the configuration being used for it!), and I also used strace to track that mistake down. But, what was so important about fixing the above was that it stopped distracting me with an error message every 60 seconds on machine B, when the configuration error was actually on machine A!

Saturday, November 27, 2010

php 5.3, ticks, pcntl_signal, pcntl_signal_dispatch

I'm trying to track down a problem with a script that works on php 5.2 but behaves strangely on php 5.3 (there are lots of differences between the environments, and I suspect php version will actually turn out to be completely unrelated). php 5.3 introduced pcntl_signal_dispatch() which processes outstanding signals and I've been investigating if that could somehow explain the behaviour differences I see.

The confusing part is that I've seen people saying that the old way of "declare(ticks=1)" is now deprecated in 5.3, and you must use pcntl_signal_dispatch(). This seemed very silly as you'd have to litter your code with calls to pcntl_signal_dispatch(), as well as have very different code for php 5.2 and 5.3.

If you're confused, like I was, here is what you need to know:
1. declare(ticks=1) still works: no deprecated message (with E_ALL|E_STRICT error reporting);

2. My script, using ticks, behaves identically under 5.2 and 5.3 when I send it a ctrl-C or a kill signal;

3. The docs don't mention it being deprecated; I realized everywhere saying this was user-contributed comments or blogs!

There is a performance aspect with using declare(ticks=1). I believe it is minor, but I think pcntl_signal_dispatch() has been introduced so you can use it instead of ticks if you want to take fine-control over when signals get considered.

Thursday, June 24, 2010

Zend Form: display group and custom decorator

I keep meaning to write a proper review of Zend Framework, but I am waiting to finish a big project that uses it. It is running late, and I think ZF can take some of the blame. So don't hold your breath waiting for a positive review.

Today's topic: I want to have part of my form hidden initially, and instead show a button saying: "toggle more questions".

So, we make it a display group (where ex1 and ex2 are the form elements to hide):
$form->addDisplayGroup(array('ex1','ex2'),'extras');

By default this wraps it in a fieldset, with no place I could see to slip in my CSS, javascript and the toggle link. What I need is a decorator, I said to myself.

Oh, gasp, does Zend make this difficult or what! In another part of this project custom decorators had been used for a minor layout change. 7 classes in 3 directories, almost all of it boilerplate. The real work was being done in CSS; those 7 classes were just to give a way to name the items as far as I could tell.

The problem is the Zend Framework Philosophy of making things complex. Did you realize there is no addDecorator($myclass) function! You have to keep decorators in a special directory and then tell Zend where to get them. Then addDecorator('part_of_my_class_name').

ZF's saving grace is that it is open source. So I poked around, and here is my solution. First, I'll define an alias for readability (optional):
$displayGroup=$form->getDisplayGroup('extras');

Remove all the default stuff I don't need (this step is optional too):
$displayGroup->removeDecorator('Fieldset');
  $displayGroup->removeDecorator('HtmlTag');
  $displayGroup->removeDecorator('DtDdWrapper');

Now insert my decorator (these 4 lines are in lieu of addDecorator($myclass)):
$decorators=$displayGroup->getDecorators();
  $decorators['MyTest']=new MyTestDecorator();
  $displayGroup->clearDecorators();
  $displayGroup->addDecorators($decorators);

(I.e. get the current decorators, add mine, then replace the existing decorators with the new set.)


Finally we get to the meat. All you need is to define a render() function that takes a string (the existing content) and returns that string, optionally modified. Here is the minimal version that does nothing.
class MyTestDecorator extends Zend_Form_Decorator_Abstract
{
  public function render($content){ return $content; }
}

And here is the full version: it hides the elements in the group and uses JQuery to show/hide it. The CSS is inline.
class MyTestDecorator extends Zend_Form_Decorator_Abstract
{
  public function render($content)
  {
    $js="$('#extra_questions').toggle();return false;";
    return '<a href="" onclick="'.$js.'">Toggle Tags Visibility</a>'.
      '<div id="extra_questions" style="display: none;">'.$content.'</div>';
  }
}

Yep, it's that simple. Oooh, the architects of ZF must be turning red with rage ;-)

UPDATE: The best article I've found so far on Zend Form Decorators. As many of the comments say, the length of the article is also a very good argument against using Zend Form. And it still didn't answer my questions, so it really needed to be 2-3 times as long. But if you need to format a form, it is a far more useful resource than the utterly inadequate Zend Form manual.

Tuesday, June 22, 2010

fclib 0.4.20 release

I just put up another fclib release. Fclib is an ad hoc collection of php libraries, started about 10 years ago; the i18n-related functions are perhaps of most interest to people (with functions for Japanese, Chinese and Arabic language processnig).

This new release has some minor bug fixes, a new utf8 function (for truncating a string), and a new file, modify_images.inc, that is a high-level interface to use gd functions to make doing thumbnails, cropping, resizing and basic drawing edits. It can operate on one or a batch of images.

(The previous release was 11 months ago, described here)

Tuesday, June 15, 2010

doctrine: delete object but leave it in database

I've a website where I make a page from a doctrine object which is taken from a database. Nothing special there. For adding the data to the database I have a form that goes to a preview page then a confirm button that actually writes it. For the preview page I share the same display logic, so I construct a doctrine object and simply don't save it. For editing, I do the same: I load the doctrine object from the database, make the changes specified on the form, and then either show the preview page or save to the database (depending on if preview or confirm was clicked).

The problem came with editing a one-to-many relation. As a concrete example let's say the user is registering multiple email addresses. I was doing this:
   $User->Emails->delete();

Then doing this for each address the user gave:
   $User->Emails[]=new Email(...);

It nicely handles when the user has removed an email address, changed one, or added an extra one. But when I realized the flaw I slapped my forehead so hard I gave myself whiplash. Do you see it? Have a moment...

Yep, if they click cancel (or leave before clicking confirm) they're expecting nothing has changed, whereas in fact all the email addresses they'd previously input have vanished.

The problem is that delete() happens immediately, rather than waiting for me to call save(). I hunted high and low for a way to stop that. My final solution was:
   if(confirmed)$User->Emails->delete();
   else $User->unlink('Emails');

I.e. unlink() appears to be the delete-that-does-not-touch-database I was searching for. (Of course, don't do something silly like go and call save() now, or the user will lose their email addresses and you'll have some orphaned records in your Emails table.)

Incidentally this did not work:
 $user->clearRelated('Email');

I'd hoped it would, after reading the description in the manual. But in fact it did nothing at all.