Unit testing FLOW3 with NetBeans

FLOW3 comes with a lot of unit tests, but to run those is only possible with the FLOW3 test runners from the Testing package. This has various reasons:

  • The code under test uses some constants defined by FLOW3 during it's bootstrap. Those are not defined when the tests are run standalone.
  • The code under test as well as the tests themselves rely on a working autoloader, which is not the case for plain PHPUnit runs.
  • Some of the tests do not properly mock their subject's dependencies and thus need the FLOW3 instance provided by out test runners to work. This is clearly a bug in the tests :)

Now that I am using NetBeans I was longing to use it's PHPUnit integration and did a little timebox work on the topic. The result is that one can now run the FLOW3 unit tests in NetBeans with a little preparation...

  1. Make sure to upgrade the Testing package to it's latest version (so it has the Bootstrap.php file).
  2. Edit your NetBeans project configuration to instruct it to use mentioned Bootstrap.php as, well, bootstrap for PHPUnit.
  3. Now comes the tricky part (due to the way NetBeans thinks tests should be organized):
    1. Create a folder (basically somewhere, I put it into my project root) that will contain all tests from NetBeans point of view.
    2. In that folder create symlinks to all Unit folders of the individual packages. One way to do this is a little shell script:
      for i in `ls ../Packages/Framework/` ;
      do ln -s ../Packages/Framework/$i/Tests/Unit $i ;
    3. Configure that folder as the tests folder for your NetBeans project.
If all is well you can now test a class file with Cmd-F6, run a test file with Shift-F6, switch between test and class file using Cmd-Shift-T and adore all the glory of pretty test results:


Code works, the unit test fails...

In this case it turned out to be simple: the test failed because it was making wrong assumptions after a code change. But I never got to know, as PHPUnit ran out of memory... This is the end of the XDebug trace:
 42.4811  96481024 -6824    -> PHPUnit_Framework_Constraint_IsEqual->fail(class F3\Fluid\Core\Parser\SyntaxTree\RootNode, string(36), ???)
42.4812 96481136 +112 -> PHPUnit_Framework_Constraint->failureDescription(class F3\Fluid\Core\Parser\SyntaxTree\RootNode, string(36), bool)
42.4812 96481216 +80 -> PHPUnit_Framework_Constraint->customFailureDescription(class F3\Fluid\Core\Parser\SyntaxTree\RootNode, string(36), bool)
42.4812 96481440 +224 -> PHPUnit_Util_Type::toString(class F3\Fluid\Core\Parser\SyntaxTree\RootNode, ???)
42.4813 96481616 +176 -> is_array(class F3\Fluid\Core\Parser\SyntaxTree\RootNode)
42.4813 96481696 +80 -> is_object(class F3\Fluid\Core\Parser\SyntaxTree\RootNode)
42.4813 96481832 +136 -> print_r(class F3\Fluid\Core\Parser\SyntaxTree\RootNode, bool)
51.5357 441897744
TRACE END [2009-11-21 09:59:22]
Note print_r line? The second field is the memory used. PHPUnit trying to get a diff of that turns out is too much. Well, around 450 MByte of variable dump - I cannot blame anyone. :)


PHP Standards Working Group - what's the point?

Recently Robert Lemke joined the PHP Standards Working Group as a representative for TYPO3. Since we are working on the FLOW3 framework as a by-product of our complete TYPO3 rewrite, we are of course interested in interoperability and standards to ease our lives.

The group is currently discussing a first proposal on "autoloader interoperability". While the proposal is short and makes a lot of sense, it deals with a little technical detail that I think is less important to interoperability than other issues: class autoloading. While a common scheme for namespace usage helps avoid name clashes and makes programming less guessing and more knowing (and as such is a very good thing!), having a standard that de facto forces a single autoloader implementation (let's face it, a sample in a standard is prone to become a fact due to laziness...) might get into people's way.

I would be happy accepting the code changes needed to comply with the standard's notion of a namespace beginning with a "vendor name". But moving all of our code out of our Classes folder we have in packages used within FLOW3? The separation between code, documentation, resources and metadata in those packages is too precious.

Thus I would be happy if the proposal focused on a common scheme for namespaces only and left an autoloader implementation out of the picture. Maybe suggest a common name for a vendor-specific autoloader so you know where to find it. Because, let's face it, adding another autoloader is the least of your problems when integrating a library.

Which brings us to the more important issues the group should look into in the future: How to foster technical interoperability in the sense of component decoupling, reliance on certain PHP settings (I'm sure most of you ran into some sort of magic_quotes_gpc issue in the past) and so forth.

Anyway, having the most prominent (framework) projects inside that standards group is already something worth celebrating! Keep the ball rolling!


Avoiding a temporary array not worth it

Recently I changed some code in PHPCR and thus also TYPO3CR that made it necessary to change some client code. Where I used to create an iterator and then append() items to it in a loop, I was now forced to prepare an array to hand over to the iterator's constructor.

So I created code like this:
foreach ($object->getData() as $item) {
$stuff[] = $item->getInfo();
$iterator = new \Foo\BarIterator($stuff);
That seemed ugly, and today I stumbled across a post in the php-internals mailing list that was asking for performance issues on a solution that avoids loops like the one above by using an anonymous function.
$iterator = new \Foo\BarIterator(
array_map(function($item) {return $item->getInfo(); }, $object->getData())

It seemed elegant, so I created some tests to check for time end memory impact of that solution. The result was - sadly - not what I had hoped. That way does not save on memory but takes longer. While elegance and readability are values in itself, this is (at least for me at that part of my code) not worth the compromise.

Here is the test code I used, for three variants of creating the iterator:
// test.php
function microtime_float() {
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);

$data = array();
for ($i=0; $i < 100000; $i++) {
$data[] = str_repeat('123-', 100);

$time_start = microtime_float();
$copy = array();
foreach ($data as $value) {
$copy[] = $value;
$a = new ArrayIterator($copy);

echo (microtime_float() - $time_start) . " seconds used\n";
echo memory_get_peak_usage() . " bytes used\n";

// test1.php
function microtime_float(){
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);

$data = array();
for ($i=0; $i < 100000; $i++) {
$data[] = str_repeat('123-', 100);

$time_start = microtime_float();
$copy = array_map(function($value) { return $value; }, $data);
$a = new ArrayIterator($copy);

echo (microtime_float() - $time_start) . " seconds used\n";
echo memory_get_peak_usage() . " bytes used\n";

// test2.php
function microtime_float() {
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);

$data = array();
for ($i=0; $i < 100000; $i++) {
$data[] = str_repeat('123-', 100);

$time_start = microtime_float();
$a = new ArrayIterator(array_map(function($value) { return $value; }, $data));

echo (microtime_float() - $time_start) . " seconds used\n";
echo memory_get_peak_usage() . " bytes used\n";

And these are the results (I ran the tests mutliple times, the results where almost identical across runs):
kmac:Sites karsten$ php test.php
0.081659078598022 seconds used
77379144 bytes used
kmac:Sites karsten$ php test1.php
0.12520813941956 seconds used
77379344 bytes used
kmac:Sites karsten$ php test2.php
0.10409712791443 seconds used
77379048 bytes used


Blogging with FLOW3? Almost there!

The last days we had Andreas Förthner up in Lübeck to work on FLOW3 together, and it was indeed very successful!

We split work, so while Robert was working on the Blog package so it can be proudly used for the upcoming FLOW3 tutorial, Andreas worked on the needed - but missing - security, session and login functionality while I dug deep into the persistence code.

As a result we now have a blog with almost complete functionality (multiple blogs, posts, tags, related posts done - comments, categories in the pipeline) whose development pushed the implementation of sorting, limit and offset in the object persistence and made support for easy XML feed output in Fluid reality.

This and next week we'll continue with those efforts and already look forward to put the new stuff to use when coming back to TYPO3v5 development after that!


Migrations in FLOW3

Migrations in Rails are quite nice. Basically what they give you is timestamped files containing steps needed to adjust the database and the data when going up and down in migration history.

I'd keep the timestamped files idea, that leaves the question where do we store what migrations have been applied already? A cache seems too fragile. Something like Migrations.yaml in Data/Persistent?

Then we don't have tables, but classes. No problem, we can map that easily. While we could try to diff class schemas and detect added and removed member variables. We don't need added ones, they will be just stored for any new instances and initialized to NULL (or rather not touched at all) during reconstitution. Removed ones don't do harm, aside from taking up space.

But isn't renaming of class members and changing the data inside what is most problematic? There is no way to automatically find out if "zip" used to be "postcode" and to change "year" from DateTime to integer using wild guesses.

So, Rails got quite some things right there with it's migration concept and making developers write down what needs to be changed.

Since a migration has up and down parts for going forth and back in time, it's easy to revert most things (see the ChangeAlbumYearToInteger migration example in the article linked above). Anyway, some stuff is irreversible, they say: really deleting stuff with drop_table and remove_column. Is it?

This is where TYPO3 was right for quite some time, renaming things to zzz_deleted_foobar instead of dropping right away. Now, if we would rename them and include the timestamp from the migration file, we could even restore such stuff.

I'll look into a nice syntax for this the next days...


Git instead of Subversion?

We are currently looking into Git as an alternative to Subversion. Interested in coming along?

A very nice, although fast-paced, introduction to git is http://gitcasts.com/posts/railsconf-git-talk

This is about git on Windows, from the same guy: http://gitcasts.com/posts/git-on-windows

Now, practical stuff. As long as everything is still in Subversion for us, using git's built-in Subversion bridging is cool. Just read this to get started: http://www.viget.com/extend/effectively-using-git-with-subversion/

I did that for PHPCR and TYPO3CR last week to be able to branch for some experimental stuff I'm doing, and that works beautifully. Creating a branch, committing to that branch, switching back to master, changing something there, committing that to Subversion. All worked well.

Most impressive is the speed with which all that happens (aside from the svn commit). Looking at the history? Diff? All lighting fast.

Granted for the Mac there is GitX, a nice frontend. But git comes with built-in GUI tools as well, git gui and gitk. TextMate has very complete Git bundle, and there is EGit for Eclipse (still rough). And TortoiseGit.

That might be helpful. I hope. :)