Thinking about an easier Codeception setup for every party involved.
The problem with sharing test fixtures
One of the main issues I’ve found pushing the TDD cause forward is the perennial initial difficulty. Even inside the seemingly homogeneous population of developers local setups, machines and preferences vary so much to make that one-time-only initial step the hardest and, often, last one. The problem I’ve tried to solve with wp-browser was to “make WordPress acceptance, functional and unit testing easier”. The Codeception suite the package builds on has roughly the same objective. But “easier” is not the same as “easy”; with an hyperbolic metaphor going to the moon is now easier than it was in 1969 yet it’s not easy at all. I do not think “easy” is an attainable point any time soon but “easier” is a result always worth prosecuting. Delving into a context I know here is an example of how “sharing a test fixture” might get overly complicated while developing a WordPress plugin:
- The team is made of 3 developers (Sheila, John and Luca) and a QA person (Alex)
- Sheila uses Docker to manage her local development environment
- John uses WAMP on Windows
- Luca uses valet
- Alex uses MAMP
Excluding the “they all should use X because X is better than Y” solution that leaves the team with quite a challenge when it comes to synchronizing the development environment. But what does it mean to “synchronize the development environment”?
Different set ups for different roles
The three developers (Sheila, John and Luca) should be able to seamlessly write and run wp-browser and Codeception powered unit, functional and acceptance tests. Alex should be able to write and run acceptance tests written using Gherkin (because Codeception is adding support for the Gherkin syntax). Sheila, John and Luca can find their way around Codeception configuration files (unit.suite.yml, functional.suite.yml and so on) and, furthermore, distribution version of those files could be included in the plugin repository to have a solid starting point (e.g. acceptance.suite.dist.yml). Alex might be in a different league when it comes to modifying Yaml configuration files and debugging why and where something went wrong. He gets help along the way from a developer though and is ready to roll in about half an hour. Alex will have forgotten it all 20 minutes later of course. One of first challenges hits immediately: the four need to share a starting database dump to run functional and acceptance tests: ideally it should be a clean and fresh WordPress installation with just the plugin installed and activated. Sheila leads the way with some CLI magic using wp-cli:
cd ~/Containers/WP/our-plugin
wp site empty --yes
wp plugin activate our-plugin
wp db export ./wp-content/plugins/our-plugin/tests/_data/start.sql
git add ./wp-content/plugins/our-plugin/tests/_data/start.sql
git commit -m "added start db dump, just activated our-plugin"
git push
anyone else in the team git pulls and… hits the wall. WordPress hardcodes full URLs in the database so the dump will contain Sheila’s setup local development address: wp.container. John uses wordpress.dev, Luca uses wp.dev and Alex uses our-plugin.dev. The devs find again their way out; Luca could write:
wp search-replace http://wp.container http://wp.dev --recurse-objects --url=wp.container
And one of them could do this to help Alex:
wp search-replace http://wp.container http://our-plugin.dev --recurse-objects --url=wp.container
wp db export ~/Desktop/stuff-that-will-be-sent-to-alex-using-slack/local-start.sql
All of them now on track they are ready to roll.
Another fixture, another form of tragedy
One database dump fixture covers them for a week of development until Alex finds out that the plugin will malfunction when two post categories will share the same name (not slug, just name). Alex cautiously sets up the context to reproduce and writes an acceptance test in Gherkin:
Feature: our plugin should deal with categories with the same name
Scenario: the plugin should handle the case where two categories have the same name
Given there are categories "one, two"
And ...
Scenario: the plugin should handle the case where three categories have the same name
Given there are categories "one, two, three"
And ...
Scenario: the plugin should handle the case where two tags have the same name
Given there are tags "one, two"
And ...
Scenario: the plugin should handle the case where three tags have the same name
Given there are tags "one, two, three"
And ...
Alex pushes the acceptance test to the repository, anyone pulls and the issue gets solved. Then another issue: when the count of posts categorized as “Uncategorized” reaches 447 and there is just another category with an higher count and at least 15 posts have titles longer than 60 words and the third-party “Foo Commerce” plugin is active the plugin will malfunction. As “sudo” as this example might be the intricacies of WordPress inner workings might very well work like this. In place of requiring the developers on the team to write new Gherkin step methods to set up an intricate starting point the team quickly votes the issue deserves its own database dump to be used in its own Codeception environment. Alex is hand-held through the process of exporting his database dump from MAMP that’s then added to the plugin test/_data folder named 447issue-start.sql and the developers go through the usual search and replace with wp-cli drill. Again the issue is solved and life goes on. It’s very easy to understand how and why this flow has the scaling capacity of a deep-fried rubber band: sharing fixtures is a pain, onboarding people with different skill sets and setups is a pain, remembering the simple act of a setting up a local testing environment is a pain.
The possible solution
What if the team could ease the pain a little? Hassle-free is not an option: we are still talking, in different roles, of people developing software and the gain comes with some pain. But what if the least CLI-savvy of the four, Alex, could have an easier life? Say run a shell script to have his local testing set up or run another script to snapshot his database to share with the team. While that’s still “something” to learn it’s not “everything” and the procedure does not lends itself to error as much. I’m writing an imaginary Gherkin syntax feature to explain:
Feature: setting up and contributing to the local testing environment should require just the skill to run a CLI command
Scenario: the local set up of the testing environment requires running a CLI script and answering some questions
Given I have pulled the latest repository version
When I execute the "./setup-local.sh" script
And I answer "wp" to the question "Acceptance db name?"
And I answer "localhost" to the question "Db host?"
And I answer "root" to the question "Db username?"
And I answer "root" to the question "Db password?"
And I answer "admin" to the question "Admin username?"
And I answer "admin" to the question "Admin password?"
And I answer "http://wp.dev" to the question "WP acceptance tests domain?"
And I answer "~/Sites/wp" to the question "WP location?"
Then I am able to run the tests executing "./vendor/codecept run"
And I am able to run the acceptance tests executing "./vendor/codecept/run acceptance"
Scenario: the current database state can be dumped and generalized running a CLI script and answering some questions
Given I have the database in the state I want to dump
When I execute the "./db-snapshot.sh" script
And I answer "issue-2233" to the question "Dump file name?"
Then I can see the "./tests/_data/issue-2233.dist.sql" file
What it all comes down to
The end being to make setting up a local testing environment across team members easier the mean to do it could be a combination of custom to the project solutions, the shell scripts in the Gherkin examples above, and some bundled Codeception commands. I’d lean on the platform-agnostic second solution as much as possible. Modern frameworks like Laravel come with their own CLI tool and I’m working to add some commands to wp-browser, and possibly Codeception, to be able to run commands like this:
codecept search-replace http://wp.local http://wp.dev ./tests/_data/start.dist.sql ./tests/_data/start.sql
wpcept snapshot-db ./tests/_data/issue443.dist.sql
On top of those still very much CLI friendly commands I’m working to build a Yaml configuration based codecept setup command to allow teams to write platform independent local set up configuration scripts. A “sudo” setup-local.yml file would look like this:
acceptance:
var:
name: dbname
question: What's the name of the database?
default: wp
var:
name: dbUser
question: What's the database user name?
default: ""
var:
name: dbPass
question: What's the database user password?
default: root
var:
name: dbHost
question: What's the database host?
default: localhost
var:
name: adminUsername
question: What's the administrator username?
default: admin
var:
name: adminPassword
question: What's the administrator password?
default: admin
var:
name: domain
question: What's the local website domain?
default: wp.local
command: search-replace wp $dbName ./tests/acceptance.suite.dist.yml ./tests/acceptance.suite.yml
command: search-replace root $dbUser ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
command: search-replace root $dbPass ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
command: search-replace localhost $dbHost ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
command: search-replace " adminUsername " $adminUsername ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
command: search-replace " adminPassword " $adminPassword ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
command: search-replace http://wp.local http://$domain ./tests/acceptance.suite.yml ./tests/acceptance.suite.yml
message: Configured accceptance suite (tests/acceptance.suite.yml)
message: Localizing database dumps...
command: search-replace http://wp.local http://$domain ./tests/_data/start.dist.sql ./tests/_data/start.sql
command: search-replace http://wp.local http://$domain ./tests/_data/issue3322.dist.sql ./tests/_data/issue3322.sql
command: search-replace http://wp.local http://$domain ./tests/_data/issue3344.dist.sql ./tests/_data/issue3344.sql
message: All done!
message: Use "./vendor/codecept run acceptance" to run acceptance tests.
Anb be consumed using the codecept setup command this way:
codecept setup local
With Codeception CLI tool relying on Symfony Console Component that’s not an impossible task.