Getting to know functional tests.
Part of a series
This post is the fourth one in a series of posts I’m publishing to chronicle my personal test-driven development flow to develop a WordPress plugin.
The first post in the series, the second and the third one could be better starting points than this post.
##The starting code and a the tools I’m pushing the code to GitHub as I work tagging it when a relevant change happens.
The code I’m starting from for this post is tagged post-04 on GitHub.
As tools of the trade I’m relying on wp-browser and Codeception for the testing, DI52 as dependency injection container and xrstf PHP 5.2 compatible autoloader for Composer to autoload the classes.
While the plugin code is PHP 5.2 compatible the tests code won’t.
Are these acceptance tests?
So far I’ve written two “acceptance” tests, one in Cept and the other in Cest format, the code below comes from one of those tests:
/**
* I can see a plugin provided configuration on the options page
*/
public function test_i_can_see_a_plugin_provided_configuration_on_the_options_page( AcceptanceTester $I ) {
$I->am('QA person');
$I->wantTo('see a plugin provided configuration');
$I->cli('scaffold plugin acme');
$pluginPath = $I->cli('plugin path acme --dir');
$qaFolder = $pluginPath . '/qa';
$I->copyDir(codecept_data_dir('configurations/one-configuration'), $qaFolder);
$I->seeFileFound($qaFolder . '/qa-config.json');
$I->loginAsAdmin();
$I->amOnAdminPage('/admin.php?page=qa-options');
$I->canSeeNumberOfElements('#qa-configuration option', 4);}
While Codeception ablative methods syntax makes it easy to understand what is happening it’s clear I’m using the WPCLI module to set up the testing fixture and fiddle with the filesystem.
Acceptance tests are usually described as black box tests where functional tests are usually described as white box tests.
The former are supposed to interact with with the application using its API (or UI) only without knowledge of its inner workings; the latter are instead tests built with knowledge of the behaviour implementation where that knowledge is used to set up pre-conditions, interact with the object and assert post-conditions.
There is always confusion between the two kind of tests and it only gets worse when I rewrite the test above as a functional test:
use function tad\WPBrowser\Tests\Support\rrmdir;
class PluginConfigurationReadCest {
/**
* @var string
*/
protected $acmePlugin;
/**
* @var string
*/
protected $qaFolder;
public function _before(FunctionalTester $I) {
$this->acmePlugin = $I->getWpRootFolder() . '/wp-content/plugins/acme';
rrmdir($this->acmePlugin);
$I->cli('scaffold plugin acme');
$this->qaFolder = $this->acmePlugin . '/qa';
}
public function _after(FunctionalTester $I) {
rrmdir($this->acmePlugin);
}
/**
* I can see a plugin provided configuration on the options page
*/
public function test_i_can_see_a_plugin_provided_configuration_on_the_options_page(FunctionalTester $I) {
$I->copyDir(codecept_data_dir('configurations/one-configuration'), $this->qaFolder);
$I->seeFileFound($this->qaFolder . '/qa-config.json');
$I->loginAsAdmin();
$I->amOnAdminPage('/admin.php?page=qa-options');
$I->canSeeNumberOfElements('#qa-configuration option', 4);
}
}
The lack of difference is striking: beside changing the tester from AcceptanceTester to FunctionalTester and dealing with the filesystem directly thanks to the WordPress::getWpRootFolder() method I’m doing exactly what I was doing before.
Understanding the difference
Acceptance tests are meant to simulate a user interaction with the system through its UI.
“Simulating a user” does not mean simulating a “dumb” user: it means simulating someone that cannot access and manipulate the system internals during the tests beyond what means the system provides to do so.
The “QA Person” I’m simulating in the tests is capable of pulling an acme plugin from his/her team repository and install the plugin in his/her own WordPress local installation: keeping in mind the kind of user I’m simulating so far there is no violation of the “black box” approach.
The difference is more evident writing one more functional test:
/**
* @test
* it should not show example configurations if disabled
*/
public function it_should_not_show_example_configurations_if_disabled(FunctionalTester $I) {
$I->copyDir(codecept_data_dir('configurations/one-configuration'), $this->qaFolder);
$I->seeFileFound($this->qaFolder . '/qa-config.json');
$I->haveOptionInDatabase('qa-thing', ['disable-examples' => true]);
$I->loginAsAdmin();
$I->amOnAdminPage('/admin.php?page=qa-options');
$I->canSeeNumberOfElements('#qa-configuration option', 1);
}
I want to make sure the plugin will provide a way to deactivate the example configurations provided; specifically that way is a toggle option (disable-examples key in the qa-thing option). I’m testing the expected behaviour, seeing just the configuration provided by the plugin, setting up the test fixture with a direct database access and setting the disable-examples sub-option in the qa-thing one.
A user should not care or know such an option exists: this makes the test a “functional” one.
The “acceptance” counterpart of this is one where the developer provides some UI the user can interact with, say a checkbox, to disable the example configurations, the user checks that and asserts that example configurations will not show up.
So a new suite?
I’ve shown and used the functional suite to run the test above and the “switch” deserves some explanation.
I’ve updated the acceptance suite configuration file (acceptance.suite.yml) to only use the WPBrowser and WPCLI modules removing the WPDb module as violating a “black box approach”:
class_name: AcceptanceTester
modules:
enabled:
- \Helper\Acceptance
- WPBrowser
- WPCLI
config:
WPBrowser:
url: 'http://wp.dev'
adminUsername: admin
adminPassword: admin
adminPath: /wp-admin
WPCLI:
path: '/Users/Luca/Sites/wp'
and updated the functional.suite.yml file to use the WPCLI and Filesystem modules too.
wp-browser scaffolds the functional suite to use the WordPress and WPDb modules so the final version of the functional.suite.yml configuration file is:
class_name: FunctionalTester
modules:
enabled:
- \Helper\Functional
- WPDb
- WordPress
- WPCLI
- Filesystem
config:
WPDb:
dsn: 'mysql:host=localhost;dbname=wp'
user: root
password: root
dump: tests/_data/dump.sql
populate: true
cleanup: true
url: 'http://wp.dev'
tablePrefix: wp_
WordPress:
depends: WPDb
wpRootFolder: /Users/Luca/Sites/wp
adminUsername: admin
adminPassword: admin
WPCLI:
path: '/Users/Luca/Sites/wp'
After the updates rebuilding the Tester classes requires running codecept build.
The biggest difference between the two “core” modules, WPBrowser for the acceptance suite and WordPress for the functional suite, is how they “hit” the code base: WPBrowser is a PHP-based browser that will go through all the steps of a real browser to fetch the response from the server while the WordPress module will hit the “route handlers” directly.
To make a simple example writing this line of test will lead to different actions in the two modules:
$I->amOnPage('/');
WPBrowser will send a GET request to http://wp.dev while WordPress will include the /Users/Luca/Sites/wp/index.php file directly and buffer its output.
Other frameworks that support long-running processes (handling more than one HTTP request in a life cycle) allow the test code to share the same variable scope as the tested code but WordPress, with its heavy use of globals and constants, will not allow for that; in stricter terms the “functional” testing the WordPres module offers is not the same offered by modules like the Laravel one but it’s a step forward nonetheless.
A failing test!
The test I’ve just written, be it functional or not, is failing.
Time to make it pass again writing as little code as possible; the simplest way to do it is to add a class in charge of managing the plugin options, qa_Options_Repository, require it as a dependency in the options page class and call it in the render method:
public function render() {
?>
<select name="qa-configuration" id="qa-configuration">
<?php if (!$this->options->disableExamples()) : ?>
<option value="foo" class="qa-configuration-option">Foo</option>
<option value="baz" class="qa-configuration-option">Baz</option>
<option value="bar" class="qa-configuration-option">Bar</option>
<?php endif; ?>
<?php /** @var qa_Configurations_ConfigurationI $configuration */
foreach ($this->scanner->configurations() as $configuration) : ?>
<option value="<?php echo $configuration->id() ?>"
class="qa-configuration-option"><?php echo $configuration->name() ?></option>
<?php endforeach; ?>
</select>
<?php
}
The new test is now a success:
Time to write a new functional test to make sure the example configuration scripts will provide some real code.
Or not?
Time for some lower level tests maybe?
Knowing what has to be tested, how and how much is not an exact science.
The blanket approach of testing “everything” works to a point: knowing that smaller changes in minor details like output strings or CSS classes could break 20 tests does not help rapid refactoring and iteration.
Only testing code I own is the first rule I use to avoid testing code but others apply to the code I own:
- test any code that’s in contact with user input
- test any code that “forks” (contains conditional statements)
- test any code that “scales” (contains loops on variable number of objects)
Applying these rules to the current code it’s evident I need to write some tests, and I’m not committing to the type of tests I need to write yet, for the following classes:
- the plugins scanner, class
qa_Configurations_Scanner - the options repository, class
qa_Options_Repository - the options page, class
qa_Options_Page
Current code
The code current to this article can be found on GitHub tagged post-05.
Next
In the next post I will move from the higher level functional and acceptance tests I’ve written so far to unit and integration testing.