Frapi (PHP web service API system)

I read about Frapi in PHP Architect (May 2011), and spent a couple of hours trying it out. It is quite interesting, but I don't think I will be using it. It is a full web-interface for making the API. This is what makes it cool, but also its biggest disadvantage. There is a lot of code involved, meaning there is a lot to learn if you need to change it and lots of places for bugs and security exploits to crop up.

It comes with a documentation generator, which could be really useful. This feature is still incomplete (for instance there are no links to it yet, see here, PDF generation didn't work properly), but it looks okay.

There is one specific limitation: I could not create an action with an optional parameter, at least not using the router.  E.g. if my action is called "ddd", then I can call /ddd (no parameter). But I cannot call "/ddd/77". (I can give an optional parameter with /ddd?id=77). Or I can define a router as "/ddd/:id" so that I can call "/ddd/77". But in that case id is now required and I cannot use "/ddd".

Another disadvantage is no built-in support for oAuth. (I did find an oauth extension for Frapi but did not try it as the integration seems quite rough still.)

Incidentally if you were looking for a full application built on Zend Framework this may make the perfect study case. Apparently only the admin interface uses ZF, and the actual web services do not; but as far as I can tell they are closely tied and you need to keep both together even on your production servers.

Overall, because making a web service in PHP is not that hard, the advantages are slight and not enough to outweigh the disadvantages (large codebase, inflexible structure, etc.).

Facebook Making Me Redundant

Back in 2007 I wrote an article on how to port your PHP applications into C++ (in the Oct 2007 issue of PHP Architect), showing the speed-up and memory-usage benefits. It's a nice idea, though I've only done it in a production project once. The programmer man-hours is very rarely worth it.

Well, Facebook have gone and automated the process, calling it Hip-Hop. Over 90% of Facebook's (400 billion PHP-based page views every month) traffic is now running on their Hip-Hop system, i.e. on PHP pages that have been compiled into C++.

This is very interesting to me as I have recently been writing less and less in C++, more and more in PHP. Even a tree searcher for computer go I have been doing in PHP, as an experiment to see how it goes. On reflection that particular experiment is probably a failure - I need to set bits in a 32-bit or 64-bit integer and PHP puts a layer of abstraction in the way.

The main php script is very slow (its been running non-stop for 5-6 weeks so far, with months left to run) but ironically all that time is being spent in calling other applications (written in C++!) to do the hard calculations. However I've another script that analyzes the data, needing to hold it all in memory at once, and that is showing signs of cracking - I'm already having to set my memory_limit to 1.5G, and that will probably hit 4G when the other script has chomped through all my data.

Hip Hop might be my saviour.

P.S. PHP Architect magazine also think this is going to be a very important technology.

Doctrine ORM

I mentioned before, when talking about my Step-By-Step Regex article in php|a magazine, that the same issue (August 2009) had some other interesting articles.

One is about Adminer, an alternative to phpMyAdmin which is designed to be very compact and exists in just one file (so it is easy to upload temporarily to a production server for instance); the article is mainly about how Adminer was written, which was interesting.

But the article - and library - I most want to talk about is Doctrine ORM. The main problem this library is solving is this: when you add a field to your database you have to make the change in at least two places, your PHP script and your database. That sucks. The article describes itself as a tour around Doctrine's more advanced features, but I found I didn't need any other introduction, and it sold the features in a very clear and logical way.

The first thing I liked was you can describe your database schema in any of three ways: SQL, PHP, or the compact YAML files, and utilities are provided to convert between all three. By SQL I mean an existing database, that you could have been creating interactively from phpMyAdmin, or it could be a legacy database that has been around for years and now needs to be connected to a php script.

What only the Yaml and PHP schema formats can specify are what Doctrine calls behaviours. By adding the Timestampable behaviour it will add created_at and updated_at fields to the database and keep them up to date for you. The Sluggable behaviour is for making URLs and is demonstrated in the php|a article, but others that caught my eye are SoftDelete (records just get the deleted_at field set when you delete, rather than physically being removed), and Versionable (each time a record is changed the previous record is saved, and can be reverted to). NestedSet helps build trees, I18n is for holding translations (see below), Searchable does indexing, Geographical can find nearby places.

Incidentally I covered I18n in a previous article and Doctrine is using type 1, but without the language table to store collation. There is no consideration of different collations, and no concept of default language. It is sufficient for most purposes though.

Looking at some behaviours not in the Doctrine core distribution, Taggable looks interesting, allowing you to add blog-style tags. Locatable apparently ties in with Google Maps to get latitude and longitude automatically given an address (it could then be used, for instance, with the Geographical behaviour to automatically find all people within 10km of you based on just their address). EventLoggable logs all actions (select, update, delete, etc.) to a disk file. Blameable records (in the table) who last made any change to a record.

Doctrine's behaviours feature is powerful, portable, the existing ones are customizable (all the above seem to have sensible defaults but also can have their behaviour fine-tuned) and new ones look easy to write.

The third big feature of Doctrine that I like is the way data is retrieved on demand, and how that ties in with table relations (see listing 6 in the php|a article). And if this becomes inefficient you can optimize without having to change much code (listing 7). I'm not a big fan of using PHP to write SQL queries. Generally to do anything interesting you end up dropping down to writing SQL anyway. But I was impressed with the example in the article, and you only need to start explicitly describing joins if you have profiled and found you need to optimize.

(By the way, it goes without saying that Doctrine ORM is database-neutral, not tying you into MySQL or any other database.)

The final cool Doctrine feature is migrations. When you make a change to the schema in either your YAML file or your PHP file you run a script and it will make a migration file. This can then be run to move a database between versions (backwards or forwards). I admit I currently do this with some hand-crafted SQL scripts, which are one way only, and painful. And sometimes I'm not even that organized.

I've not used Doctrine ORM for a real project, but I expect I will try it out very soon.

php Architect: no more print version

The subject says it all: the publishers of the excellent php|a magazine have suddenly stopped the print edition. It has gone PDF only.
(Click php|a on the right to see my other blog articles about php|a magazine.)

This is such a shame: I would willingly pay much more for a print version than a PDF version. In fact I was doing exactly that. Then a year ago they turned everyone to be Print+PDF subscribers, cut the price dramatically, and gave me free extra 12 issues to make up for the price cut. In fact that only works out as 6 or so free issues, but as they were free I can hardly complain.

It is just a shame they fiddled with their business model, as obviously the price cut made the print magazine too expensive.

Why do I prefer the print version? It can be read on the train, read in the mountains, is light and portable. It can be kept on my bookshelf, pulled off and opened on my desk while I'm coding. As an author, a print magazine is also much more impressive. It is something physical I can show a potential client, something I can show my Mum.

Regex Article in php|a

The August edition of PHP Architect magazine contained my introductory article on regexes. It has a PHP slant but mainly it is about regexes. Well, most of it is introductory but it contains a bonus section at the end using advanced regexes in SQL to repair a database in situ.

The print magazine arrived by air mail a couple of weeks ago and I finally got to read it. I was pleased (and relieved) that it read well. If you have read it I'd love to get some constructive criticism - especially if you are uncomfortable with regexes.

It was also my first print article to use colour, and I think that worked well too.

One correction in the complicated "Repairing With Regexes" section at the end of the article. The text says "So \1 has to be written \\\\1 (that is four backslashes)." That seemed a strange thing to write, so I took a look at the unit test source code (included with the PDF download version of php|a magazine) I had written. Four backslashes is correct when in PHP, but when in SQL it should be "\\1" not "\1".
(I just checked my emails from when we were proof-reading, and one of the edits had removed all those backslashes; I caught that at the time but ironically I got it wrong when putting them back in.)

The other minor correction I only realized after Arne Blanket's column in the same magazine issue! I had written: "...an IP address after that at sign, such as darren@10.0.0.1, which, while unusual, is technically valid". In fact "darren@[10.0.0.1]" is the technically valid form. Which is doubly annoying because my article's regex would reject those square brackets and I hadn't explicitly pointed that out. (No harm done, though, as this email form is highly discouraged.)

Arne's column also points out that top-level domains are no longer just two or three characters. So my '\.[a-zA-Z]{2,3}$' suggestion should really have been '\.[a-zA-Z]{2,}$'. Luckily, that suggestion was just in a list of other ideas, not part of the article's main regex.

(By the way I enjoyed, and will blog about real soon, some other articles in this particular issue; if you are not a subscriber, and your work involves data and PHP, then this is the back issue you should get!)

Storing website translations in SQL

In php|a magazine, April 2009, there was an article called "Storing Multilingual Records in the MySQL Database". As the title says, it has some mysql-specific elements, but the concepts are quite general. The author introduced and compared three alternatives; as I have used a fourth way I thought I'd write about it here.

The situation was a product database, where the name, url and description have to be localized. But some entries will be untranslated and should fall back to using the default language entry.

1. Translations in a separate table

Here is the database schema:

product: product_id, group_id, price.
product_translation: product_id (links to product), language_id (links to language), name, url, description.
language: language_id, collation.

This requires an SQL join which is hard for mysql to execute efficiently. Fulltext indexes are also a problem, as the name and description fields contain text from different languages. The only good point of this approach is that it is easy to add a new language.

2. Data Copy

Here the group_id and price fields have been moved into the product_translation table. That makes the SQL queries a bit cleaner, as it saves one join, but doesn't really solve the other problems. And the data redundancy is an accident waiting to happen (I'm having a physical reaction just thinking about it).

3. Translation Directly In The Database

This is like 2 above, but each language gets its own field. For instance if we have three languages, English (en), German (de) and Japanese (ja) then it looks like:

product: product_id, language_id (links to language), name_en, url_en, description_en, name_de, url_de, description_de, name_ja, url_ja, description_ja, price, group_id.

The advantage this brings is using the default language is easy; you can either just always select the default language field, or use mysql's IFNULL function. E.g. either:
SELECT name_de,name_en WHERE product_id=123;
(and then check in your PHP script to see if name_de is blank, and if so use name_en instead)
Or:
SELECT IFNULL(name_de,name_en) as name WHERE product_id=123;

The other advantage is FULLTEXT indices work well now, as only one language is kept in each text column. Disadvantages are the space-wasting if actual translations are sparse, the work required to add a new language, and possibly hitting the mysql maximum row size (64K).

4. One table per language

This is the approach I've used in MLSN, among others:

product: product_id, group_id, price.
product_translation_en: product_id, name, url, description.
product_translation_de: product_id, name, url, description.
product_translation_ja: product_id, name, url, description.

Fulltext indices work well, joins are relatively easy, no data duplication. Adding a language feels cleaner than solution three above, as it doesn't require modifying an existing table, just adding a new one. If the site takes off then different languages can be split to different servers easily. And the language table can disappear as collation will now be part of the field definition when the table is created. (I think the same can be said of solution three, but the php|a article kept the language table in its schema - was that an oversight, or am I missing something?)

The dark side? Getting the default language requires two queries. Or would something like this (untested) SQL work?
SELECT ISNULL(de.name,en.name) FROM product_translation_de de,product_translation_en en WHERE de.product_id=123 AND en.product_id=123

Thoughts?

The 4th option works well for me, but I'd be interested to here arguments against it. Perhaps you use something different again?

Excel and PDF from PHP

Been meaning to post this for ages, but PHP Architect magazine for November 2008 had a useful article on libraries and code for alternatives to HTML.

For PDF html2pdf was introduced. Unlike other PHP PDF libraries (e.g. PDFLib, FPDF) this one has you make your content in HTML then it gets converted. For those of us comfortable with HTML, that means no learning curve.

For Excel, PHPExcel is introduced. This has classes to read and write excel files in a variety of formats. I get this request a lot but so far we have always decided outputting good old CSV is best (simple to make, compact, easy to import into Excel). But if complex layout, or built-in functions, are required this looks worth a look. It can also output into PDF.

Also very interesting is in-memory spreadsheets, with lots of mathematical functions supported. I've not tried it, and neither the article or the website mention explicitly if this will work on Linux (or a Windows machine without Excel installed), but it is described as standalone so it should. It might be an interesting way to port someones spreadsheet to a PHP app!

The article also briefly introduces Google charts, and how to include the produced image in your excel file.

PHP comments from PHP!

php|architect December 2008 edition's most interesting article was on phpdoc. I'm a fanatic when it comes to the javadoc style of commenting. When I create a new function the first thing I do is write function somename(){}, then the second thing I do is write /** */ above it.

If I'm refactoring then I might copy and paste in some code straightaway, but usually the third thing I do is document. I describe what the function does, er, will do, and usually define the @return tag (if I've not worked out what data, if any, it should return then I've got ahead of myself - how can I even name a function if I don't know what it produces??). The @param tags I usually leave until after I've written and tested the function, as they can change a lot (i.e. I need an algorithm before I know what inputs my algorithm takes).

During the writing of the function body I will usually jump up to the comment block and add an @internal comment (describing the implementation), or a @todo comment ("Need to add error-checking on call to somefunc()" is a very common one!).

To quote from the php|a article: "What's really exciting,though, is that since PHP 5.1, the Reflection classes can also supply the subject's "doc comment".

See http://php.net/reflection

The php|a article introduced a base class for automatically creating setters for class vars. My own interest is in somehow using it for contract based programming (also called DBC: Design By Contract).

I believe the Java implementation of DBC uses docblock entries, but with a pre-compiler. Pre-compilers suck: being able to edit and run the same source code file is a big bonus, and maybe reflection of docblocks enables that?

Of course using comments is poor cousin of DBC built into the language itself (especially if it can do compile-time analysis). But, for some reason I've not yet grasped, language designers just don't understand how important it is. They limp along with lint and asserts instead.

Back to the docblocks, I've not thought out the details; does PHP provide some hook that can be called when each function is entered and exited? If not I guess you'd have to write VALIDATE() as the first line of each function, and a RETURN() function everywhere you use return, which would be rather intrusive.

Either way, it could then look for @paramvalidate tags which could describe valid data to be found in parameters. It could look for @variant tags in the class doc block to check the object is valid on each of entry and exit.

Did I mention DBC finds loads of bugs, without you having to write a single unit test; all you have to do is document properly. Which, of course, you are going to do anyway if you are a professional programmer.

But did you realize DBC can also be used for optimization, by prioviding hints to the optimizer? As a concrete example (I'm thinking about C++ here) you have two input variables both described as being in the range 1..100. When they are multiplied together the range is 1..10000. Therefore there is no need to consider overflow. If you then divide by that variable then a divide by zero error is impossible. I'm not a compiler writer, and haven't written assembly in about 15 years, but I'm sure the above information, or similar, can help.

As another example, that also applies to PHP, static analysis of possible values can tell you that for an if/elseif/elseif/else block only the second elseif block can ever be reached. This can be both reported as a compiler warning and all the conditional stuff thrown away.

But, that is all a pipe dream, it requires DBC being implemented in the language itself, and I've drifted from the point of this article. Which was: a PHP script can read its own comments! Cool!