H5mag: Tested by the Borg

Borg cubes in the office.

One of the most remarkable sights in our office is the large number of Borg cubes (cube shaped space ships from Star Trek television show) glimmering on a white table. It doesn't have a very pretentious name, we just call it 'test system' but it's actually very important for H5mag development.

As long as software gets written by neural computers, i.e. humans, bugs will trickle in. The reason for this is the fact that humans aren't capable of perfect reasoning, whereas computers are. The test system is one of our tools to prevent our production code from assimilating bugs.

How it works

Once a developer has completed a new feature, he or she sends a so-called 'pull request'. The request is meant for the lead developer but is actually sent to the test system. The test system will then test the feature the developer just created and pass the request to the lead developer if there haven't been any regressions, otherwise the author will get an email describing the regression found by the testing system. This system has saved us many hours and also prevented tons of issues. Without the test system, there would be no H5mag.

Here's how a developer sends a pull request:

php pullrequest

That's it! After those magic words a request will be filed with the test system and the borg cubes on the table will light up. The test system will checkout the master branch and pull the code from the developer into that code. Next, the system will start to run all kinds of tests. If there is a regression, the developer will receive a test report from the system by e-mail. Until all tests passed and we can be reasonably sure there are no regressions, the lead developer will not be bothered by pull requests. As such, the test system also acts as a filter.

The email report contains a lot of information, a log file containing all log messages from the client, an HTML file describing the current state of the program at the time the regression occurred and even a screenshot. There's also an animated gif to balance out the bad news with some entertainment.

A mail from the test system informing the developer that a test has failed.

What do we test?

Unfortunately, tests don't write themselves. Our developers have to write the tests themselves. There are several categories of tests, we call these A-G.

• A: Server-side unit tests
• B: JS Linting and Compile tests
• C: Setup test (register a magazine)
• D: Client-side unit tests
• E: Editor tests
• F: Front-end tests
• G: Device tests

During test category 'E', the test system will fire up the editor and attempt to create a full-blown magazine. It will click on buttons, drag images and areas and construct the magazine just like a user would do. Here is a funny comilation of our test system working hard:

Screen capture of the automated testing process, video is sped up 10 times from actual speed.

Once the magazine has been created, the system will switch to the front-end and start 'viewing' the magazine, checking if it has been created correctly. Then, it will connect to all the other devices on the table and they will also view and check the magazine (category 'G'). Unit tests

Although we refer to some categories as unit tests, we actually don't really believe in the effectiveness of unit testing in the 'narrow sense'. Our tests tend to use rather 'wide' units. We try to focus on testing 'brittle' parts. If we consider a piece of code to be 'trivial' we don't bother write a test. We decide together which parts of our code will be tested and which parts won't be. However we try to write our code with testing in mind. For instance, we put as much logic in models as we can, because we consider models to be easily testable. Thus, we keep our Controllers rather thin, because testing a controller is a bit more painful.

Test statistics

Besides the pull request command, developer can also issue the 'prinfo' command. This tool shows the current activity on the testing table and it shows how active developers are currently testing their code. Here is some sample output:

=== TEST TAFEL OVERVIEW ===
Pull requests in queue : 0
No. of devices : 13
Lucky dog of the week : Abby (0.8000)
Poor bastard of the week : Gabor (0.2727)
Fanatic of the week : Gabor (22)

Here you see I (Gabor) have send 22 pull request but only 27% of my attempts have been succesful, so I am considered the poor bastard of the week. Abby on the other hand has a success rate of 80%. You can also see the length of the queue and the number of tablets and phones connected to the testing system (13).

Saved by the Borg cubes

Our test system has grown organically. What started with a simple test script has grown into a complete testing platform for H5mag. As developers we love and hate our testing system, but one thing is for sure, if it hadn't been around, there would not be an H5mag. Many, many issues have been traced and prevented from being released by those borg cubes on the white table. So, the Borg are not so bad after all :)