×
×

Drupal Testing Methodologies Are Broken - Here's Why

That's right! I said it. Currently available Drupal testing methodologies are broken. They are nowhere close to being enterprise-ready. All Drupal developers who have worked on long-term projects have felt this pain. Yes, Drupal 7 ships with Simpletest module. But there are two big problems with it:

Drupal 7 is not unit-test friendly

Although Simpletest supports unit testing, Drupal, especially custom code, which we really need to test, is not unit-test friendly. That's because unit testing requires that database not be intialized and all the input conditions be supplied as arguments to the function (what we call Dependency Injection in Symfony and Drupal 8). Since Drupal stores its configuration in the database, code assumes that the DB is available whenever it needs it. Drupal code also uses global variables such as $user and $base_url. Purists will say that we should write the code such that the DB connection or global variables are provided as arguments to the function as required. But then you are needlessly complicating the code. The other problem with unit testing Drupal's custom code is that almost everything happens in hooks. The input arguments to these hooks, such as hook_form_alter(), are deeply nested arrays and the hook changes one or two things in those arrays. If you want to unit test the hooks, you need to create such deeply nested arrays that you provide as arguments which the hooks can modify and that is really cumbersome.

As an example, consider a simple hook_form_alter() where we are changing the default value of the Greeting text field to "Hello" if the Article node we are editing has the highest nid among all the nodes in the system.

/**
 * Implements hook_form_FORM_ID_alter() for node_article_form().
 */
function mymodule_form_node_article_form_alter(&$form, &$form_state) {
  if ($form['#node_edit_form'] == FALSE) {
    return;
  }

  $nid = $form['#node']->nid;

  // Check if there is any other node in the system with a higher nid.
  $query = new EntityFieldQuery();
  $query->entityCondition('entity_type', 'node')
    ->entityCondition('entity_id', $nid, '>');
  $result = $query->execute();

  if (!isset($result['node'])) {
    // There is no node with a higher node id.
    $form['greeting'][LANGUAGE_NONE][0]['#default_value'] = t('Hello');
  }
}

It's nightmare to unit test this simple function. In fact, we can not unit test it because it uses a database query. We first need to refactor the function to the following:

/**
 * Implements hook_form_FORM_ID_alter() for node_article_form().
 */
function mymodule_form_node_article_form_alter(&$form, &$form_state) {
  if (!is_edit_form($form)) {
    return;
  }

  if (node_has_highest_nid($form['#node']->nid)) {
    modify_greeting_field($form);
  }
}

/**
 * Check whether the provided form is a node edit form.
 */
function is_edit_form($form) {
  return $form['#node_edit_form'];
}

/**
 * Check whether the node id provided is the highest node id in the system.
 */
function node_has_highest_nid($nid) {
  // Check if there is any other node in the system with a higher nid.
  $query = new EntityFieldQuery();
  $query->entityCondition('entity_type', 'node')
    ->entityCondition('entity_id', $nid, '>');
  $result = $query->execute();

  return !isset($result['node']);
}

/**
 * Set the default value of the Greeting field.
 */
function modify_greeting_field(&$form) {
  $form['greeting'][LANGUAGE_NONE][0]['#default_value'] = t('Hello');
}

You will notice that the number of lines of codes has increased from 21 to 39. Now we can unit test is_edit_form() and modify_greeting_field() functions like this:

/**
 * Test is_edit_form() function.
 */
function testIsEditForm() {
  $form = array();
  $form['#node_edit_form'] = TRUE;
  $this->assertTrue(is_edit_form($form), "A node edit form is misinterpreted as not not an edit form.");

  $form['#node_edit_form'] = FALSE;
  $this->assertFalse(is_edit_form($form), "A form which is not an edit form is misinterpreted as edit form.");

  unset($form['#node_edit_form']);
  $this->assertFalse(is_edit_form($form), "A form which is not an edit form is misinterpreted as edit form.");
}

/**
 * Test modify_greeting_field() function.
 */
function testModifyGreetingField() {
  $form = array();
  modify_greeting_field($form);
  $this->assertEqual("Hello", $form[LANGUAGE_NONE][0]['#default_value'], 'Greeting field value is not set to "Hello".');

  $form[LANGUAGE_NONE][0]['#default_value'] = t('Random');
  $this->assertEqual("Hello", $form[LANGUAGE_NONE][0]['#default_value'], 'Greeting field value is not set to "Hello".');
}

Even after all these extra lines of code, we are essentially testing less than half of the original code. Even in that, we are just testing array manipulation logic. The EntityFieldQuery() logic, which is the most important logic in this function, is not being tested at all. No wonder that most Drupal developers don't write automated tests and most clients don't want to pay for it.

Simpletest module requires reconfiguring the site from scratch

While doing functional tests, Drupal's simpletest module creates separate environment and database tables for each test and doesn't use the existing site configuration. In addition to being slow, this approach requires that before each functional test, you configure the site in code to replicate the existing site that is in production. This is nuts!!!! As an example, if you have a custom content type "Student" with 20 fields, you will have to create the content type and all those 20 fields in code before you start your functional tests. That's a lot of extra code! Yes, you can use Features for creating the content types with fields and you can enable that feature module before you start the functional test. But then you are managing a lot of feature modules for your site. You should try resolving conflicts in features modules if multiple developers are working on your code. It's a lot of fun! :) And to top it all off, not all the configurations work with Features so at some point you will have to write code to configure the site before you start the functional tests. This is not the most difficult problem to solve though. Some developers have tried modifying Simpletest so that it uses either the existing dev database or creates a new test database by uploading an SQL dump. I have not tried them but even if this issue is resolved, the issue of difficulty of writing unit tests for Drupal 7 still remains.

So what's the solution?

Forget about writing tests. Just kidding! :) As explained earlier, unit-testing won't cut it for Drupal because of how Drupal is written. We need to consider integration and functional tests. There are some good functional testing tools that work with Drupal such as Behat, Sahi and Selenium. We actually use these and they work pretty well. The problem is that functional tests are very slow because they use the browser. Even PhantomJS, a headless browser without the GUI, is much slower than unit tests or integration tests. This may not be because PhantomJS is slow, but because every single request, Drupal needs to do a complete bootstrap. For a medium complexity site, if you write all your tests as functional tests, it can easily take between 2 to 3 hours to complete one round of automated testing. That's agonizing! There has to be something better.

That's where integration tests come in. Integration tests use the database but not the browser. Although integration tests do need to bootstrap Drupal, they can be made to do it only once for the complete test suite. As a result, they are slower than unit tests but much faster than functional tests. Since integration tests have access to the database, developers don't have to modify their code to make it unit-testable. Here are some characteristics of an integration testing framework that will interest Drupal developers:

  1. It needs to work with the existing site and database. Developers shouldn't need to configure their site in code before they start the tests.
  2. It should help in reducing the time it takes for developers to write and maintain tests. This implicitly means that the test code should be readable and not too long.

Agreed that integration tests won't be able to test the site layout or JS. For these tests, we have to use functional tests but for other tests where we are testing business logic or site structure, integration tests are a much better solution that functional tests.

Since little over a year, we have been creating such a framework and using it with our large Drupal development projects. It is created specifically for Drupal and it understands Drupal so a lot of mundane background tasks of setting up the tests and the content types with fields are done automatically. Developer just has to concentrate on writing the business logic for the test. We are already on the fourth iteration and we feel that it's good enough and useful enough for general public. We have decided to make it open source so that the community can benefit from it and the brand of Drupal as a quality web development platform increases. Follow us on Twitter and LinkedIn or sign up below for our newsletter to know as soon it's released as Open Source!

EDIT: The integration test framework, named Red Test, is open-source now. If you have 15 minutes, start by writing your first integration test!

Services: 
Drupal Development

Sign up for our weekly newsletter


Comments

  • by Kevin (not verified)
  • Fri, 03/20/2015 - 09:48

You can use PHPUnit with Drupal 7, it just takes a little setup. You require bootstrap.inc and bootstrap Drupal, then you can use PHPUnit barebones to test custom code and have Drupal functions at your fingertips.

Then again, PHPUnit is baked into D8, so.

neerav.mehta's picture
  • by neerav.mehta
  • Fri, 03/20/2015 - 10:55

Yes, that's exactly what we have done for D7. We used PHPUnit specifically because it's baked into D8. In addition to PHPUnit, this framework has a lot of helper functions to automate the setup of the tests and create test content.

  • by chx (not verified)
  • Fri, 03/20/2015 - 15:18

people only write about a well researched topics.

  • by Albert Albala (not verified)
  • Sat, 03/21/2015 - 09:18

Thanks for the article, a good diagnosis of what's wrong with Drupal testing in my experience. However I would not agree that it's a good idea to test what's in the dev database, simply because what's in the dev database can change from instant to the next and it's not under version control. Basically, I like tests to work with a known good starting point. If I run a test today on a specific commit of my code, I want to have the exact same results as I did last month. This is not the case if we test an unversioned database. I few approaches I have tried have been the site deployment module; and caching simpletest setup with Simpletest Turbo. I admit there's no perfect solution and I very much appreciated your article.

neerav.mehta's picture
  • by neerav.mehta
  • Sat, 03/21/2015 - 14:58

Hi Albert,

Thanks a lot for your suggestions! In fact, I actually ran across your site deployment article a few months ago to see on what existing options are available to improve testing experience within Drupal. Your article about site deployment did stand out and we were going to give it a try but haven't yet. This is the first time I am looking at Simpletest Turbo module. It seems like a good idea for executing Simpletest.

Coming back to the gist of your comment: yes, testing an unversioned database or a database with no known starting data does not make sense. This framework is just one piece of the puzzle in making sure the site is stable. For the projects where we use this test framework, we also have continuous integration and continuous delivery in place. Each developer gets a new copy of the production database every day. Before developer starts a task, he takes this new database and  creates a new git branch from the prod branch. As soon as he commits this branch to git, our continuous integration server automatically runs all the tests in its own Docker container. A task is deployed in production only if all the tests pass. As soon as a task is pushed to prod branch, all the other feature branches get updated with the new code. As you can see, a developer never has old code and a database that is older than a few days from production and he always has access to a fresh copy. As soon as a big task is deployed on production, we inform all the developers to specifically update their database. As a result, we haven't encountered the problem of testing a stale database.

I'll contact you once we make our framework open-source so that we can collaborate on this and make Drupal testing rock! :)

Regards,Neerav.

neerav.mehta's picture
  • by neerav.mehta
  • Sat, 03/21/2015 - 23:54

Yes, we did look at Upal. From what I can tell, it is a functional testing framework. This means that Drupal gets bootstrapped for each test so it still will be slow. Also it doesn't provide many helper functions to make developing and maintaining tests easier.

  • by rakesh.james
  • Mon, 03/23/2015 - 05:20

Well said Neerav!. Thank you for the wonderful article which expalins from all the view points. 

  • by alex.designworks (not verified)
  • Mon, 04/13/2015 - 16:18

https://www.drupal.org/project/site_test

SiteTest allows for each test to run in 3 modes:
* Site - use current database tables.
* Clone - create copy of site tables.
* Core - create new tables for blank Drupal installation (core SimpleTest module implementation).

neerav.mehta's picture
  • by neerav.mehta
  • Sun, 05/03/2015 - 07:44

I got the "Site" mode, which doesn't create a new test environment, to work. For a very barebones test with 100 loops, it takes 1 min 1 seconds. The same test without "Site" mode, which creates a new test environment, takes about 1 min 6 seconds. So the "site" mode shaves off about 5 seconds and that too with a very basic setup. I am sure it will be able to reduce the time quite a lot if the setup is more complicated. Good work!

  • by Emiliano (not verified)
  • Mon, 01/04/2016 - 09:46

Thanks a loooot alex.designworks!!! and also to the neerav.mehta
you don't have any idea how much time am I looking for a solution and the plugin
https://www.drupal.org/project/site_test

solve all my problems in 5 minutes

THANKS SO MUCH!
emiliano

Add new comment