Introduction

I've been using LLMs for code assistance, in the form of completion/copilot suggestions or through a CLI like Claude Code, for some time now, and while I've got a running list of things I like and do not like, I can see value in the idea of LLM-assisted code writing.

Some things keep nagging at me:

• the tools do not work on a plane i.e., when there is no internet connection or when I'm not otherwise connected
• I've got a powerful laptop: why do I have to use it like a terminal in the original sense of the word? The compute power is not local to my machine, it's in a server somewhere I have to connect to, and all my RAM and CPU/GPU is wasted
• I have only a superficial understanding of how these tools work in the context of the IDE I use for work all-day long

The last point, in particular, triggers me into wanting to build my own copilot plugin for my IDE of choice: an IntelliJ one and, more specifically, PHPStorm.

Previous attempts and art

I've tried to switch away from PHPStorm to other Visual-Studio-Code-like forks and variations, but the muscle memory, the tooling and the code inspection capabilities are not where I would like them to be.
I've tried setting up the VS-Code-likes using all the recommended plugins and tools (including licensed ones), I've tried other IDEs as well (e.g. Zed or neovim) and all of them are great.
But they are not great for me, my idea of IDE perfection that I get out of the box with PHPStorm always another plugin/extension away.

The one thing that really surprised me is the llama.vim plugins for vim/neovim.

With a relatively small model (Qwen2.5-Coder-3B) running on my laptop, it provides excellent suggestion with a " look-around" quality that other tools lack.
The lack of an IntelliJ version is the inspiration for this project of mine.

Can I build an IntelliJ plugin providing auto-completions inspired by the llama.vim plugin?

It should be noted I know nothing of Kotlin, the language I will use to build the plugin.

Level 0

I've done some research, and if I'm building a plugin for an IntelliJ-based IDE, I should use the plugin template.
Once cloned and configured I can run build the plugin using the gradlew buildPlugin task and install it in my IDE of choice (PHPStorm) from the Plugins section of the IDE using the Install Plugin from Disk option.

I've customized the project and set its name: "completamente". It means "completely" in Italian and, again in Italian is a composition of the words "complete" and "mind".
It's meant to be a word play around the concept of completion and mind.
Naming things is hard; I'm not a poet.

Since building the plugin and installing it in my work IDE (i.e. the IDE I work with all-day) is not ideal, I've set up a run configuration to run the plugin in PHPStorm to debug it.

The CLI command to start a different copy of PHPStorm with my plugin installed in it would be:

./gradlew runIdeForUiTests -PplatformType=PS -PplatformVersion=2025.1.3

This is convenient, but I'm still not getting a step debugging environment with this.

To start a debug version of PHPStorm including my plugin I've added a run configuration to the project in the .run/Run Plugin in PHPStorm.run.xml file:

<component name="ProjectRunConfigurationManager">
    <configuration default="false" name="Run Plugin in PHPStorm" type="GradleRunConfiguration" factoryName="Gradle">
        <log_file alias="IDE logs" path="$PROJECT_DIR$/build/idea-sandbox/*/log/idea.log" show_all="true"/>
        <ExternalSystemSettings>
            <option name="executionName"/>
            <option name="externalProjectPath" value="$PROJECT_DIR$"/>
            <option name="externalSystemIdString" value="GRADLE"/>
            <option name="scriptParameters" value=""/>
            <option name="taskDescriptions">
                <list/>
            </option>
            <option name="taskNames">
                <list>
                    <option value="runIde"/>
                </list>
            </option>
            <option name="vmOptions" value=""/>
        </ExternalSystemSettings>
        <ExternalSystemDebugServerProcess>false</ExternalSystemDebugServerProcess>
        <ExternalSystemReattachDebugProcess>true</ExternalSystemReattachDebugProcess>
        <DebugAllEnabled>false</DebugAllEnabled>
        <RunAsTest>false</RunAsTest>
        <method v="2"/>
    </configuration>
</component>

Now I can run or debug a copy of PHPStorm with my plugin installed:

Tapping into the completion API

The first task is connecting my plugin to the reccomended IDE completion API through the plugin configuration file ( src/main/resources/META-INF/plugin.xml):

<!-- Plugin Configuration File. Read more: https://plugins.jetbrains.com/docs/intellij/plugin-configuration-file.html -->
<idea-plugin>
    <id>com.github.lucatume.completamente</id>
    <name>completamente</name>
    <vendor>lucatume</vendor>

    <depends>com.intellij.modules.platform</depends>

    <resource-bundle>messages.MyBundle</resource-bundle>

    <extensions defaultExtensionNs="com.intellij">
        <inline.completion.provider implementation="com.github.lucatume.completamente.completion.Service"/>
    </extensions>
</idea-plugin>

The Service class is, at this stage, not doing much if not providing the same completion, "Hello World!" over and over:

package com.github.lucatume.completamente.completion

import com.intellij.codeInsight.inline.completion.InlineCompletionEvent
import com.intellij.codeInsight.inline.completion.InlineCompletionProvider
import com.intellij.codeInsight.inline.completion.InlineCompletionProviderID
import com.intellij.codeInsight.inline.completion.InlineCompletionRequest
import com.intellij.codeInsight.inline.completion.elements.InlineCompletionGrayTextElement
import com.intellij.codeInsight.inline.completion.suggestion.InlineCompletionSingleSuggestion
import com.intellij.codeInsight.inline.completion.suggestion.InlineCompletionSuggestion
import com.intellij.codeInsight.inline.completion.suggestion.InlineCompletionVariant

class Service : InlineCompletionProvider {
    // A class to represent a single string suggestion.
    class StringSuggestion(private val suggestion: String) : InlineCompletionSingleSuggestion {
        override suspend fun getVariant() = InlineCompletionVariant.build {
            emit(InlineCompletionGrayTextElement(suggestion))
        }
    }

    override val id: InlineCompletionProviderID
        get() = InlineCompletionProviderID("completamente")

    override suspend fun getSuggestion(request: InlineCompletionRequest): InlineCompletionSuggestion {
        // No computation going on here, the same string every time.
        return StringSuggestion("Hello World!")
    }

    override fun isEnabled(event: InlineCompletionEvent): Boolean {
        // It's always enabled.
        return true
    }
}

I've started the plugin in debug mode and I can see the completion in action in the context of my di52 project:

It works. Badly and poorly, but it compiles and it works.

Tests

I'm doing this project to learn.
The easiest way for me to learn things is to test them.

I've started simple with a test to cover the really basic StringSuggestion class:

// src/test/kotlin/com/github/lucatume/completamente/completion/StringSuggestionTest.kt
package com.github.lucatume.completamente.completion

import com.intellij.testFramework.fixtures.BasePlatformTestCase
import kotlinx.coroutines.flow.count
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking

class StringSuggestionTest : BasePlatformTestCase() {
    fun testItReturnsTheSameString() {
        val stringSuggestion = StringSuggestion("hello")
        val variant = runBlocking { stringSuggestion.getVariant() }
        val size = runBlocking { variant.elements.count() }
        val suggestedString = runBlocking { variant.elements.first().text }

        assertEquals(1, size)
        assertEquals("hello", suggestedString)
    }
}

Things got complicated as I entered the world of Kotlin and found myself immediately staring down the abyss of async, multiprocess code.

The StringSuggestion::getVariant function is marked with the suspend keyword, which means it's going to be executed in a different thread; asynchronously, that is.

From my meager understanding of Kotlin coroutines, I've worked out that suspend is like JavaScript async and runBlocking is like await.
If I was to write the code in JavaScript, I'd write something like this:

const variant = await stringSuggestion.getVariant();

// The `count` function is a coroutine that returns the number of elements in the flow.
// That too is asynchronous.
const size = await variant.elements.count();

// The `first` function is a coroutine that returns the first element in the flow.
// That too is ... asynchronous.
const suggestedString = await variant.elements.first().text;

Running the tests with gradlew test shows the test passing.

Testing the service itself required me to dig a bit into the IntelliJ API, but I managed to put together a testing approach that works using real files and with limited mocking.
In the test below, the file src/test/testData/completion/text-file.txt is a real file that I've created for testing purposes and the makeInlineeCompletionRequest function provides a convenient API to create a InlineCompletionRequest object from a file path and offsets:

class ServiceTest : BasePlatformTestCase() {
    // Where I'm getting tests files from, allow me to use relative paths in the tests.
    override fun getTestDataPath() = "src/test/testData/completion"

    // Create an inline completion for tests.
    fun makeInlineCompletionEvent(): InlineCompletionEvent {
        return object : InlineCompletionEvent {
            override fun toRequest(): InlineCompletionRequest {
                throw UnsupportedOperationException("event.toRequest() is not supported in tests")
            }
        }
    }

    // Create an inline completion request for tests.
    fun makeInlineCompletionRequest(filePath: String, startOffset: Int, endOffset: Int): InlineCompletionRequest {
        val event = makeInlineCompletionEvent()
        val file = myFixture.configureByFile(filePath)
        val editor = myFixture.editor
        val document = myFixture.getDocument(file)

        return InlineCompletionRequest(event, file, editor, document, startOffset, endOffset)
    }

    fun testId() {
        val service = Service()

        assertEquals("completamente", service.id.id)
    }

    fun testIsEnabled() {
        val service = Service()

        assertTrue(service.isEnabled(makeInlineCompletionEvent()))
    }

    // This function is more about knowing I can write and run tests than about testing the service.
    fun testGetSuggestion() {
        val request = makeInlineCompletionRequest("text-file.txt", 0, 0)
        val service = Service()

        val suggestion: InlineCompletionSuggestion = runBlocking { service.getSuggestion(request) }
        val variants: List<InlineCompletionVariant> = runBlocking { suggestion.getVariants() }

        assertEquals(1, variants.size)

        assertEquals(1, runBlocking { variants.first().elements.count() })
        assertEquals("Hello World!", runBlocking { variants.first().elements.first().text })
    }
}

The tests are not currently testing much, but I'm getting acquainted with the IntelliJ API, and I need to nail the basics before moving on.

The tests are currently passing. Progress.

Next

Keeping the llama.vim code to the side, I will incrementally reproduce the implementation approach in my plugin.
The next step will be connecting to an external HTTP completion API to obtain my first real completion.

Embracing bare metal

During the rewrite of version 4 of wp-browser, I've used the chance to remove old code that was either required to support older versions of PHP and Codeception, or to support deprecated or little-used features.

Among the "victims" of this cleaning spree was the Symlinker extension: a Codeception extension provided by wp-browser that would create symbolic links from a source to a destination before tests ran.

I was rarely using the extension, always had very little feedback about it (and no data since I've never collected user and usage information in any way), and had seen it rarely used in the wild, so I removed it.

Until recently, I've always dealt with the intricacies of the file and directory structure required by WordPress (e.g., plugins in wp-content/plugins, themes in wp-content/themes) by using containers.
The nature of container bind mounts makes it easy to have plugins and themes live anywhere and just bind them in place.
If a plugin lives in /home/lucatume/vendor/some-plugin on my machine, I can "bind it in place" using a bind mount in /var/www/html/wp-content/plugins/some-plugin and call it a day.

Support for that functionality and setup is not gone from the latest versions of wp-browser, but I've started using "bare metal" solutions more and more where allowed by the nature of the project.

Nginx/Apache on Docker? No, plain PHP built-in web-server with 5 workers.
Chrome in a container? No, Chrome or Chromium already installed on my machine.
MySQL or MariaDB from a container? No, SQLite from a local file, if I can manage it.

Turns out that CI support is pretty solid as well, and wp-browser CI setup itself is much simplified by just using what is available in most CI environments now

There are situations where all the complexities of more complicated, container-based, setups are required, but that is frequently not the case.

Along with the cleaning, I've reworked wp-browser setup template, the one used when running vendor/bin/codecept init wpbrowser, to use a PHP Built-in server, SQLite and Chromium stack by default, allowing plugin, theme and site developers to be up and running in no time.

![Bare metal setup][images/bare-metal-setup.png]

Placing things

In that setup, I'm symbolically linking the plugin or theme under development in the correct location in the WordPress directory.

I wanted to make sure that symbolic link location would work both in integration and end-to-end tests, the two types of tests scaffolded by default by the template, and could not think of a better way to do it than to bring it back the Symlinker extension.

The latest version of the setup makes it clear setting up the main Codeception configuration file, codeception.yml, like this:

namespace: Tests
support_namespace: Support
paths:
    tests: tests
    output: tests/_output
    data: tests/Support/Data
    support: tests/Support
    envs: tests/_envs
actor_suffix: Tester
params:
    - tests/.env
extensions:
    enabled:
        - Codeception\Extension\RunFailed
        - lucatume\WPBrowser\Extension\ChromeDriverController
        - lucatume\WPBrowser\Extension\BuiltInServerController
        - lucatume\WPBrowser\Extension\Symlinker
    config:
        lucatume\WPBrowser\Extension\ChromeDriverController:
            port: '%CHROMEDRIVER_PORT%'
        lucatume\WPBrowser\Extension\BuiltInServerController:
            workers: 5
            port: '%BUILTIN_SERVER_PORT%'
            docroot: '%WORDPRESS_ROOT_DIR%'
            env:
                DATABASE_TYPE: sqlite
                DB_ENGINE: sqlite
                DB_DIR: '%codecept_root_dir%/tests/Support/Data'
                DB_FILE: db.sqlite
        lucatume\WPBrowser\Extension\Symlinker:
            wpRootFolder: '%WORDPRESS_ROOT_DIR%'
            plugins:
                - .
            themes: []
    commands:
        - lucatume\WPBrowser\Command\RunOriginal
        - lucatume\WPBrowser\Command\RunAll
        - lucatume\WPBrowser\Command\GenerateWPUnit
        - lucatume\WPBrowser\Command\DbExport
        - lucatume\WPBrowser\Command\DbImport
        - lucatume\WPBrowser\Command\MonkeyCachePath
        - lucatume\WPBrowser\Command\MonkeyCacheClear
        - lucatume\WPBrowser\Command\DevStart
        - lucatume\WPBrowser\Command\DevStop
        - lucatume\WPBrowser\Command\DevInfo
        - lucatume\WPBrowser\Command\DevRestart
        - lucatume\WPBrowser\Command\ChromedriverUpdate

So, there and back again; the Symlinker extension is back along with an improved support for "metal" set ups and more to come in wp-browser future.

You can read more about wp-browser in its documentation page.

When I started working on the branch called version-4/minimum-compat-pass in May last year, I thought updating wp-browser to be compatible with Codeception version 5 in a minimal way would have been a matter of a few weeks.
The idea was to fix the most glaring issues of the WordPress testing framework and then move on to a more thorough refactoring of the codebase.

I was wrong.

The first pass took a year and a month and it's now available as a release candidate in the branch v4.

Not because of Codeception, but because of 9 years of code I had to Mary-Kondo through. My love for the craft of code, which to this day is still raging, made it very difficult for me to overlook the many issues I found in the codebase and I ended up rewriting most of it.

My first commit was on June 16, 2014; I knew so little about PHP I believed all the following to be true:

1. I cannot write code without PHPUnit tests or the plugin repository will reject my plugin.
2. No company would work with a PHP developer that does not test her code.
3. Every one uses XDebug.

There is no judgement here, I was just naive and I had no idea what I was doing.

Being a first pass, it's still not yet all the things I want it to be, but it's a good start and it is, above all, compatible with Codeception version 5, PHP 8.0, 8.1 and 8.2.

Where possible, I tried to keep back compatibility with existing tests: I'm an avid user of my own framework and I didn't want to have to rewrite all my tests to use the new version. Some tests will need rewriting, though, and I'll try to document them as I go and find out.

Next steps are, in an order I wish I will respect but likely won't:

1. Documentation update to reflect the changes in the new version.
2. A new first setup experience leveraging containers for a no-hassle setup for theme, plugins and site projects.
3. A new test case, provisionally called SuperWordPressTestCase (no jokes) to leverage the new features of the framework.

I'm also planning to write a series of posts on the new features of the framework and what they bring to the table in terms of new possible use cases.

Previously

This post is part of a series of posts where I'm exploring how to use TLA+ to specify, check and then implement a PHP project that deals with concurrent processes; here are the links to the first post, the second, the third one, and the fourth one.

Fast-failure support

I'm using TLA+ to model a Loop that will be the core of a yet-to-be-implemented testing framework.
One of the features I find very useful is support for fast failure: as soon as a test fails, break out of the Loop, gracefully shut down the other running tests, and exit with an error code.

Since each test "job", whatever that will end up representing, could be running when the first failure comes is, the challenges are the graceful shutdown of the Loop and correct closing of the currently running jobs.

One of the assumptions I had coded in the specification before this step is that all worker processes would run, thus making the worker processes fair; in TLA+ terms, each process is "weakly fair".
Fast failure introduces a drastic change in that paradigm: a worker process could not run if the Loop never starts it because a previous worker's job failed.
This means I should update the worker process by removing the fair keyword from it: while it will run most of the time, it might not run at all if a previous worker job failed triggering the fast-failure handling.
After the update, though, the model checking will fail with Stuttering:

In the image above, without expanding the details of the states, this happened:

1. The Loop started with two workers
2. When the first worker completed the Loop started a new one (i.e., set the _startedRegister[J3] value to TRUE).
3. The Loop waits for updates from the workers.
4. The last worker process started, J3 stutters: it simply never executes because it is an "unfair" (i.e., not fair) process.

An unfair process simulates a process that might never run: it crashes or takes too much time to produce any noticeable side-effect.

This little experiment shows that making the worker processes not fair is not the right solution.

How should I translate the following from plain English to something that TLA+ would understand?

Beyond PlusCal

The language I've used to write and update the specification is called PlusCal. What I write in PlusCal is then translated into TLA+ specification language proper.
PlusCal is not the specification language; it's merely a comment when it comes to TLA+.
While PlusCal is powerful, it does not allow the kind of access I need to solve my problem: it's time to get my hands dirty.

Before applying my solution to the actual specification, I would like to experiment with a simpler and smaller specification with all the elements required by the real one, but in a more uncomplicated form.
There is a Loop process that will start worker processes, each process should run when activated, and there is the concept of the Loop completing before all worker processes have a chance to start.

Here is the heavily commented version of the PlusCal code, \* starts a comment:

----------------------------- MODULE test_goto -----------------------------
EXTENDS TLC, Integers, FiniteSets
CONSTANTS Loop, Workers, NULL

(*--algorithm test_goto
variables
    started = [x \in Workers |-> FALSE],
    startedCount = 0,
    workTally = 0;

define
NotStartedWorkers == {x \in Workers: started[x] = FALSE} 
end define;

fair process loop = Loop
variables proc = NULL;
begin
    Loop_StartWorker:
        proc := CHOOSE x \in NotStartedWorkers: TRUE; \* Pick the first not started worker.
        startedCount := startedCount + 1;
        started[proc] := TRUE; \* Mark this process started, to power NotStartedWorkers.
        if startedCount /= 2 then \* Purposefully not starting the 3rd worker.
            goto Loop_StartWorker;
        end if;
    Loop_WaitForWorkDone:
        await workTally = 2; \* Two workers are done, the 3rd one wil never start.
end process;

fair process worker \in Workers begin \* If not blocked, workers should execute.
    Worker_AutostartGuard: \* Always blocked action: workers cannot start on their own.
        await FALSE = TRUE;
    Worker_Work: \* The Loop will move workers directly here to start them.
        workTally := workTally + 1;
end process;

end algorithm;*)
=============================================================================

The model I'm testing the specification with, after translation, is set up as follows:

• What is the behavior spec? - 'Temporal formula'

• What to check?

  • Deadlock
  • Termination

• What is the model?

  • Loop <- [ model value ]
  • Workers <- [ model value ] {W1, W2, W3}
  • NULL <- [ model value ]

Translating and running the model will result in a deadlock:

The Loop will start two workers out of 3; both of them blocked by a guard in the following form:

await FALSE = TRUE;

That await will permanently block the worker processes on the Worker_AutostartGuard action.
Instead of setting a flag in a register shared by Loop and workers to coordinate and stimulate the Loop starting the processes, the _startedRegister variable in the specification, I want the Loop to start the workers by moving them to the Worker_Work action directly.

Before I show the code, I would like to make a note about how PlusCal code is translated.
In the TLA+ translation, there is no concept of "processes"; there are simply states the system will transit to that, for convenience of writing, PlusCal will wrap in processes. The TLA+ specification will be one giant state machine, and the model checker will move from state to state, respecting the available nodes.
When translated from PlusCal to the TLA+ language, the translation will add a pc variable that will map each process to its next state; it can be seen in the image above, initially set to this:

<Initial Predicate>
    pc
        W1 -> Worker_AutostartGuard
        W2 -> Worker_AutostartGuard
        W3 -> Worker_AutostartGuard
        Loop -> Loop_StartWorker

My thinking is that I could move a worker to the Worker_Work phase by setting its entry in the pc variable to Worker_Work to make it look something like this:

<Initial Predicate>
    pc
        W1 -> Worker_Work
        W2 -> Worker_AutostartGuard
        W3 -> Worker_AutostartGuard
        Loop -> Loop_StartWorker

I've updated my PlusCal code to do that:

----------------------------- MODULE test_goto -----------------------------
EXTENDS TLC, Integers, FiniteSets
CONSTANTS Loop, Workers, NULL

(*--algorithm test_goto
variables
    started = [x \in Workers |-> FALSE],
    startedCount = 0,
    workTally = 0;

define
NotStartedWorkers == {x \in Workers: started[x] = FALSE} 
end define;

fair process loop = Loop
variables proc = NULL;
begin
    Loop_StartWorker:
        proc := CHOOSE x \in NotStartedWorkers: TRUE; \* Pick the first not started worker.
        startedCount := startedCount + 1;
        started[proc] := TRUE; \* Mark this process started, to power NotStartedWorkers.
        pc[proc] := "Worker_Work"; \* Move the worker process to the "Worker_Work" action directly.
        if startedCount /= 2 then \* Purposefully not starting the 3rd worker.
            goto Loop_StartWorker;
        end if;
    Loop_WaitForWorkDone:
        await workTally = 2; \* Two workers are done, the 3rd one wil never start.
end process;


fair process worker \in Workers begin \* If not blocked, workers should execute.
    Worker_AutostartGuard: \* Always blocked action: workers cannot start on their own.
        await FALSE = TRUE;
    Worker_Work: \* The Loop will move workers directly here to start them.
        workTally := workTally + 1;
end process;

end algorithm;*)
=============================================================================

Except that is not allowed, and the translation will fail with the message:

Unrecoverable error:
-- Multiple assignments to pc

I've not done multiple assignments to pc, but the translation did.
Looking at the translated code, I can see the pc variable is modified in each state, its next value (indicated by pc') updated using EXCEPT. As an example, this is the translation of the Worker_Work action:

Worker_Work(self) == /\ pc[self] = "Worker_Work"
                     /\ workTally' = workTally + 1
                     /\ pc' = [pc EXCEPT ![self] = "Done"]
                     /\ UNCHANGED << started, startedCount, proc >>

The syntax means "when the model checker will run the Worker_Work action, then the next state will be this."; what the model checker will do next is check all the available actions and pick one whose pre-conditions are met and execute it.

In plain English, the above means:

if the action I (a worker) should execute is the "Worker_Work" one
then the next value of workTally will be workTally + 1
and the next value of pc (the program counter) for myself will be Done
and started, startedCount and proc will not change.

Remember /\ is a logic AND, var' = <value> is an assignment applied to the next state, and self means the process itself, used as a key.
The string pc' = [pc EXCEPT ![self] = "Done"] means "the next state of pc is like pc except for the key self that will have a value of Done"; the syntax is a bit weird.

The words "action" and "state" should ring a bell from store-pattern implementations like Redux. It's an excellent concept to keep in mind and wrap my head around the fact that each PlusCal label (e.g., the Worker_Work one) will become an action (an operator, a function in TLA+ terms) that will take a state as an input and return the next state as an output.

Since each translation of a PlusCal label will update the next state of the pc variable, I cannot do that in the context of PlusCal.
Why not allow updates of the variables more than once? I would not know for sure, I did not write the TLA+ model checker, but I can guess to avoid the next state being a function of itself.

Messing with the translation

I've commented the problematic parts of the PlusCal code and translated it to get the bulk of it set up.
In the translation, I've commented the original code out and added the updated version right after it.

----------------------------- MODULE test_goto -----------------------------
EXTENDS TLC, Integers, FiniteSets
CONSTANTS Loop, Workers, NULL

(*--algorithm test_goto
variables
    started = [x \in Workers |-> FALSE],
    startedCount = 0,
    workTally = 0;

define
NotStartedWorkers == {x \in Workers: started[x] = FALSE} 
end define;

fair process loop = Loop
variables proc = NULL;
begin
    Loop_StartWorker:
        proc := CHOOSE x \in NotStartedWorkers: TRUE; \* Pick the first not started worker.
        startedCount := startedCount + 1;
        started[proc] := TRUE; \* Mark this process started, to power NotStartedWorkers.
        \* pc[proc] := "Worker_Work"; >>> TODO IN THE TRANSLATION.
        if startedCount /= 2 then \* Purposefully not starting the 3rd worker.
            goto Loop_StartWorker;
        end if;
    Loop_WaitForWorkDone:
        await workTally = 2; \* Two workers are done, the 3rd one wil never start.
        \* pc := [x \in DOMAIN pc |-> "Done"] >>> TODO IN THE TRANSLATION.
end process;

fair process worker \in Workers begin \* If not blocked, workers should execute.
    Worker_AutostartGuard: \* Always blocked action: workers cannot start on their own.
        await FALSE = TRUE;
    Worker_Work: \* The Loop will move workers directly here to start them.
        workTally := workTally + 1;
end process;

end algorithm;*)
\* BEGIN TRANSLATION (chksum(pcal) \in STRING /\ chksum(tla) \in STRING)
VARIABLES started, startedCount, workTally, pc

(* define statement *)
NotStartedWorkers == {x \in Workers: started[x] = FALSE}

VARIABLE proc

vars == << started, startedCount, workTally, pc, proc >>

ProcSet == {Loop} \cup (Workers)

Init == (* Global variables *)
        /\ started = [x \in Workers |-> FALSE]
        /\ startedCount = 0
        /\ workTally = 0
        (* Process loop *)
        /\ proc = NULL
        /\ pc = [self \in ProcSet |-> CASE self = Loop -> "Loop_StartWorker"
                                        [] self \in Workers -> "Worker_AutostartGuard"]

Loop_StartWorker == /\ pc[Loop] = "Loop_StartWorker"
                    /\ proc' = (CHOOSE x \in NotStartedWorkers: TRUE)
                    /\ startedCount' = startedCount + 1
                    /\ started' = [started EXCEPT ![proc'] = TRUE]
                    /\ IF startedCount' /= 2
                        \*   THEN /\ pc' = [pc EXCEPT ![Loop] = "Loop_StartWorker"]
                        \*   ELSE /\ pc' = [pc EXCEPT ![Loop] = "Loop_WaitForWorkDone"]
                          THEN /\ pc' = [pc EXCEPT ![Loop] = "Loop_StartWorker", ![proc'] = "Worker_Work"]
                          ELSE /\ pc' = [pc EXCEPT ![Loop] = "Loop_WaitForWorkDone", ![proc'] = "Worker_Work"]
                    /\ UNCHANGED workTally

Loop_WaitForWorkDone == /\ pc[Loop] = "Loop_WaitForWorkDone"
                        /\ workTally = 2
                        \* /\ pc' = [pc EXCEPT ![Loop] = "Done"]
                        /\ pc' = [x \in DOMAIN pc |-> "Done"]
                        /\ UNCHANGED << started, startedCount, workTally, proc >>

loop == Loop_StartWorker \/ Loop_WaitForWorkDone

Worker_AutostartGuard(self) == /\ pc[self] = "Worker_AutostartGuard"
                               /\ FALSE = TRUE
                               /\ pc' = [pc EXCEPT ![self] = "Worker_Work"]
                               /\ UNCHANGED << started, startedCount, 
                                               workTally, proc >>

Worker_Work(self) == /\ pc[self] = "Worker_Work"
                     /\ workTally' = workTally + 1
                     /\ pc' = [pc EXCEPT ![self] = "Done"]
                     /\ UNCHANGED << started, startedCount, proc >>

worker(self) == Worker_AutostartGuard(self) \/ Worker_Work(self)

(* Allow infinite stuttering to prevent deadlock on termination. *)
Terminating == /\ \A self \in ProcSet: pc[self] = "Done"
               /\ UNCHANGED vars

Next == loop
           \/ (\E self \in Workers: worker(self))
           \/ Terminating

Spec == /\ Init /\ [][Next]_vars
        /\ WF_vars(loop)
        /\ \A self \in Workers : WF_vars(worker(self))

Termination == <>(\A self \in ProcSet: pc[self] = "Done")

\* END TRANSLATION 
=============================================================================

In the Loop_StartWorker and Loop_WaitForWorkDone actions, I'm updating more than one entry of the pc' map, modeling the idea that the Loop will "start" the worker processes correctly without using a register and avoiding unterminated worker processes by setting the next stage of any process to "Done" in the Loop_WaitForWorkDone phase.

Checking the model for Termination and Deadlock will pass this time:

Next

In my next post, I will apply this newfound "tool" to the actual specification I care about and keep working on it to move it toward its final version.

Previously

This post is part of a series of posts where I'm exploring how to use TLA+ to specify, check and then implement a PHP project that deals with concurrent processes; here are the links to the first post, the second, and the third one.

Checking for output

In the previous post, I have completed a specification of the Loop that would pass model checking in different scenarios with more jobs than parallelism, more parallelism than jobs, and, finally, more of a "serial" case with a parallelism of 1.
I've added two more models to check:

• 3 jobs, parallelism 3
• 1 job, parallelism 1

The specification modified as outlined in the third post passes all model checks without any change.

There is something fundamental to the Loop correct execution that I'm currently not checking: the Loop should collect all the output emitted by the Workers.

In the specification, I've represented the output emitted by the workers as a * char appended to a sequence. I'm keeping that over-simplification in place and leveraging that by ensuring the amount of * chars collected by the Loop is the same emitted by the workers.
Since the workers use an Inter-process Communication pipe to share their output with the Loop, I want to make another assertion that all worker pipes are drained by the end of the Loop.

I've done the first update to the specification to support this new check:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers, FiniteSets, Sequences
CONSTANTS Loop, JobSet, Parallelism, NULL

(*--algorithm loop
variables
	jobsCount = Cardinality(JobSet),
	_startedRegister = [x \in JobSet |-> FALSE],
	_processStatusRegister = [x \in JobSet |-> NULL],
	_processPipesRegister = [x \in JobSet |-> <<>>],
	_emittedOutputLen = 0,
	_collectedOutputLen = 0;

define
	StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE})
	NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE
	CompletedCount == Cardinality({x \in JobSet: _processStatusRegister[x] /= NULL})
	Running == StartedCount - CompletedCount
	ParallelismRespected == Running <= Parallelism
	StreamSelectUpdates == {x \in JobSet: _processPipesRegister[x] /= <<>>}
end define;

fair process loop = Loop
variables
	streamSelectUpdates = {},
	updatedProcess = NULL,
	processStatus = NULL,
	processToOutputMap = [x \in JobSet |-> <<>>],
	processToExitStatusMap = [x \in JobSet |-> NULL];

	begin
		StartInitialBatch:
			while StartedCount < Parallelism /\ StartedCount < jobsCount do
				with p = NextNotStarted do
					_startedRegister[p] := TRUE;
				end with;
			end while;
		WaitForStreamUpdates:
			streamSelectUpdates := StreamSelectUpdates;
			await Cardinality(StreamSelectUpdates) > 0;
		HandleStreamUpdates:
			while Cardinality(streamSelectUpdates) > 0 do
				updatedProcess := CHOOSE x \in streamSelectUpdates: TRUE;
				streamSelectUpdates := streamSelectUpdates \ {updatedProcess};
				GetProcessOutput:
					with processOutput = _processPipesRegister[updatedProcess] do
						processToOutputMap[updatedProcess] := Append(processToOutputMap[updatedProcess], processOutput);
						_collectedOutputLen := _collectedOutputLen + Len(processOutput);
					end with;
					_processPipesRegister[updatedProcess] := <<>>;
				GetProcessStatus:
					processStatus := _processStatusRegister[updatedProcess];
				UpdateTrackedProcessPipes:
					if processStatus /= NULL then
						processToExitStatusMap[updatedProcess] := processStatus;
					end if;
				CheckLoopStatus:
					if processToExitStatusMap[updatedProcess] /= NULL then
						MaybeStartOneMore:
							if StartedCount < jobsCount /\ Running < Parallelism then
								with p = NextNotStarted do
								_startedRegister[p] := TRUE;
								end with;
								goto WaitForStreamUpdates;
							end if;
					end if;
					CheckAllDone:
						if Cardinality({x \in JobSet: processToExitStatusMap[x] /= NULL}) = jobsCount then
							assert(_collectedOutputLen = _emittedOutputLen);
							goto Done;
						else
							goto WaitForStreamUpdates;
						end if;
			end while;
end process

fair process worker \in JobSet
begin
	WaitToStart:
		await _startedRegister[self] = TRUE;
	Work:
		either
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
			_emittedOutputLen := _emittedOutputLen + 1
		or
			skip;
		end either;
	ExitStatus:
		either
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
			_emittedOutputLen := _emittedOutputLen + 1;
			_processStatusRegister[self] := 0;
			goto Done;
		or
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
			_emittedOutputLen := _emittedOutputLen + 1;
			_processStatusRegister[self] := 1;
			goto Done;
		end either;
end process;

end algorithm;*)
\* BEGIN TRANSLATION
\* [...]
\* END TRANSLATION 

OneWorkerPerJobStarted == <>[](StartedCount = Cardinality(JobSet))
AllWorkersCompleted == <>[](CompletedCount = Cardinality(JobSet))
=============================================================================

To note in the specification:

• I'm using the _emittedOutputLen to store the length of the output generated by the workers; only the worker processes will update it.
• The _collectedOutputLen global variable stores the length of the output collected by the Loop process.
• I've not added any Invariant Property to my model

The specification is failing when checked against a model:

The error message reads as follows (I'm omitting the lines):

The first argument of Assert evaluated to FALSE; the second argument was:
"Failure of assertion at line 69, column 57."
The error occurred when TLC was evaluating the nested
expressions at the following positions: ...

The failing assertion is the one I've added in the Loop CheckAllDone phase; before moving to the Done phase of the Loop, I want to make sure the loop collected all the emitted output.
Instead of using a temporal property, I'm using an assertion. The reason for this approach is that a temporal property would be an "eventually, then always" one.
But the length of the emitted and collected output would be 0 at the start of the specification, and it would be invalidated now and then as the Loop and the workers alternate emitting and collecting it. Changing the temporal condition to just "eventually" would not guarantee me that is true at the end, when the Loop is completed.
And "when the loop is done" is precisely the only moment I want to make this check, so an assertion makes sense.

That assertion fails, telling me the current specification is not correctly collecting all the output emitted by the workers.

Another pass on the specification adds a step in the CheckLoopStatus phase to get the output the process might have emitted before exiting. This update will make the specification pass across all models:

"`tla ----------------------------- MODULE spec_loop ----------------------------- EXTENDS TLC, Integers, FiniteSets, Sequences CONSTANTS Loop, JobSet, Parallelism, NULL

(*--algorithm loop variables jobsCount = Cardinality(JobSet), _startedRegister = [x \in JobSet |-> FALSE], _processStatusRegister = [x \in JobSet |-> NULL], _processPipesRegister = [x \in JobSet |-> <<>>], _emittedOutputLen = 0, _collectedOutputLen = 0;

define StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE}) NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE CompletedCount == Cardinality({x \in JobSet: _processStatusRegister[x] /= NULL}) Running == StartedCount - CompletedCount ParallelismRespected == Running <= Parallelism StreamSelectUpdates == {x \in JobSet: _processPipesRegister[x] /= <<>>} end define;

macro startProcess() begin with p = NextNotStarted do _startedRegister[NextNotStarted] := TRUE; end with; end macro;

macro collectProcessOutput(p, processToOutputMap) begin with processOutput = _processPipesRegister[updatedProcess] do processToOutputMap[updatedProcess] := Append(processToOutputMap[updatedProcess], processOutput); _collectedOutputLen := _collectedOutputLen + Len(processOutput); end with; _processPipesRegister[updatedProcess] := <<>>; end macro;

macro emitOutput(p, output) begin _processPipesRegister[p] := Append(_processPipesRegister[p], output); _emittedOutputLen := _emittedOutputLen + 1; end macro;

macro exitWithStatus(p, status) begin _processStatusRegister[p] := status; end macro;

fair process loop = Loop variables streamSelectUpdates = {}, updatedProcess = NULL, processStatus = NULL, processToOutputMap = [x \in JobSet |-> <<>>], processToExitStatusMap = [x \in JobSet |-> NULL];

begin
	StartInitialBatch:
		while StartedCount < Parallelism /\ StartedCount < jobsCount do
		    startProcess();
		end while;
	WaitForStreamUpdates:
		streamSelectUpdates := StreamSelectUpdates;
		await Cardinality(StreamSelectUpdates) > 0;
	HandleStreamUpdates:
		while Cardinality(streamSelectUpdates) > 0 do
			updatedProcess := CHOOSE x \in streamSelectUpdates: TRUE;
			streamSelectUpdates := streamSelectUpdates \ {updatedProcess};
			GetProcessOutput:
			    collectProcessOutput(updatedProcess, processToOutputMap);
			GetProcessStatus:
				processStatus := _processStatusRegister[updatedProcess];
			UpdateTrackedProcessPipes:
				if processStatus /= NULL then
					processToExitStatusMap[updatedProcess] := processStatus;
				end if;
			CheckLoopStatus:
				if processToExitStatusMap[updatedProcess] /= NULL then
					GetProcessExitOutput:	
					    collectProcessOutput(updatedProcess, processToOutputMap);
					MaybeStartOneMore:
						if StartedCount < jobsCount /\ Running < Parallelism then
		                    startProcess();
							goto WaitForStreamUpdates;
						end if;
				end if;
				CheckAllDone:
					if Cardinality({x \in JobSet: processToExitStatusMap[x] /= NULL}) = jobsCount then
						assert(_collectedOutputLen = _emittedOutputLen);
						goto Done;
					else
						goto WaitForStreamUpdates;
					end if;
		end while;

end process

fair process worker \in JobSet begin WaitToStart: await _startedRegister[self] = TRUE; Work: either emitOutput(self, ""); or skip; end either; ExitStatus: either emitOutput(self, ""); exitWithStatus(self, 0); goto Done; or _processPipesRegister[self] := Append(_processPipesRegister[self], "*"); _emittedOutputLen := _emittedOutputLen + 1; _processStatusRegister[self] := 1; goto Done; end either; end process;

end algorithm;*) * BEGIN TRANSLATION * [...] * END TRANSLATION

OneWorkerPerJobStarted == <>[](StartedCount = Cardinality(JobSet))

AllWorkersCompleted == <>[](CompletedCount = Cardinality(JobSet))

Besides putting in place the fix to make sure the specification will pass all models check, I've also refactored the code to extract duplicated code into `macro's: `startProcess`, `collectProcessOutput`, `emitOutput`, and `exitWithStatus`.  
What each of them does is pretty easy to understand, and macros work like functions. For the most: the one exception is macros will only be able to change the value of variables that are either local to the macro (but this is pretty common in any programming language) or that are passed to the macros. Once this is taken care of, macros help reduce the verbosity and duplication of the code a bit.

### PHP translation
There are more features I want my Loop-based code to support, but it's worth trying to translate that into PHP code before I move on and find myself having to write too complicated code.

```php
<?php

class Loop
{
    private $jobs;
    private $jobsCount;
    protected $parallelism;
    private $procs = [];
    private $startedCount = 0;
    private $runningCount = 0;
    private $readStdoutStreams = [];
    private $readStderrStreams = [];
    private $jobToExitStatusMap = [];
    private $jobToStdoutContents = [];
    private $jobToStderrContents = [];

    public function __construct($jobs, $parallelism)
    {
        $this->jobs = $jobs;
        $this->jobsCount = count($jobs);
        $this->parallelism = $parallelism;
        reset($this->jobs);
    }

    private function collectProcessOutput(stdClass $proc)
    {
        if (!isset($this->jobToStdoutContents[$proc->job])) {
            $this->jobToStdoutContents[$proc->job] = '';
        }
        if (!isset($this->jobToStderrContents[$proc->job])) {
            $this->jobToStderrContents[$proc->job] = '';
        }
        $this->jobToStdoutContents[$proc->job] .= stream_get_contents($proc->stdoutStream);
        $this->jobToStderrContents[$proc->job] .= stream_get_contents($proc->stderrStream);
    }

    private function startWorker()
    {
        $job = current($this->jobs);
        next($this->jobs);
        $command = sprintf("%s %s %s", PHP_BINARY, escapeshellarg(__FILE__), $job);
        $desc = [
            0 => ['pipe', 'r'], // Proc STDIN.
            1 => ['pipe', 'w'], // Proc STDOUT.
            2 => ['pipe', 'w'], // Proc STDERR.
        ];
        $procHandle = proc_open($command, $desc, $pipes, null, null, ['bypass_shell']);
        $procStdout = $pipes[1];
        $procStderr = $pipes[2];
        $this->procs[] = (object) [
            'procHandle' => $procHandle, 'stdoutStream' => $procStdout, 'stderrStream' => $procStderr, 'job' => $job
        ];
        $this->startedCount++;
        $this->runningCount++;
        $this->readStdoutStreams[] = $procStdout;
        $this->readStderrStreams[] = $procStderr;
    }

    private function getProcFromStream($stream)
    {
        foreach ($this->procs as $key => $proc) {
            if ($proc->stdoutStream === $stream || $proc->stderrStream === $stream) {
                return $proc;
            }
        }

        return null;
    }

    public function run()
    {
        // StartInitialBatch
        while ($this->startedCount < $this->parallelism && $this->startedCount < $this->jobsCount) {
            $this->startWorker();
        }

        while (true) {
            //WaitForStreamUpdates

            // Reset this on each run to make sure we only wait for updates from active process streams.
            $readStreams = array_merge($this->readStdoutStreams, $this->readStderrStreams);
            $read = $readStreams;
            $write = [];
            $except = [];
            $streamUpdates = stream_select($read, $write, $except, 10);

            if (!$streamUpdates) {
                continue;
            }

            // HandleStreamUpdates
            $skipRead = [];
            foreach ($read as $index => $stream) {
                if (isset($skipRead[$index])) {
                    // This stream was coupled with an already read one, it's been read already.
                    continue;
                }

                // GetProcessOutput
                $proc = $this->getProcFromStream($stream);
                $this->collectProcessOutput($proc);
                // GetProcessStatus
                $procStatus = proc_get_status($proc->procHandle);
                if (!$procStatus['running']) {
                    $this->jobToExitStatusMap[$proc->job] = $procStatus['exitcode'];

                    // UpdateTrackedProcessPipes
                    // We'll empty the other stream now, skip that.
                    $otherStreamIndex = in_array($proc->stdoutStream, $this->readStdoutStreams, true) ?
                        array_search($proc->stderrStream, $readStreams, true)
                        : array_search($proc->stdoutStream, $readStreams, true);
                    $skipRead[$otherStreamIndex] = true;
                    --$this->runningCount;
                    $this->readStdoutStreams = array_diff($this->readStdoutStreams, [$proc->stdoutStream]);
                    $this->readStderrStreams = array_diff($this->readStderrStreams, [$proc->stderrStream]);

                    // GetProcessExitOutput
                    $this->collectProcessOutput($proc);
                    // MaybeStartOneMore
                    if ($this->startedCount < $this->jobsCount && $this->runningCount < $this->parallelism) {
                        $this->startWorker();
                    }
                }
                //CheckAllDone
                if (count($this->jobToExitStatusMap) === $this->jobsCount) {
                    break 2;
                }
            }
        }

        return array_map(function ($job) {
            return [
                'job' => $job,
                'stdout' => $this->jobToStdoutContents[$job],
                'sterr' => $this->jobToStderrContents[$job],
                'exitStatus' => $this->jobToExitStatusMap[$job]
            ];
        }, $this->jobs);
    }
}

function worker()
{
    do {
        $stdoutOrStderr = mt_rand(0, 1);
        if ($stdoutOrStderr === 0) {
            fwrite(STDOUT, "*", 1);
        } else {
            fwrite(STDERR, "*", 1);
        }
        $action = mt_rand(0, 3);
    } while ($action > 0);

    $exitStatus = mt_rand(0, 1);
    exit($exitStatus);
}

if (!isset($argv[1])) {
    // Start the loop.
    $loop = new Loop(range(1, 5), 2);
    echo json_encode($loop->run(), JSON_PRETTY_PRINT);
} else {
    // Handle a worker request.
    worker();
}

I've tried to port over, as comment blocks, the labels that would be in the specification.
Running the script will print output similar to this on the screen:

» php test-loop-02.php
[
    {
        "job": 1,
        "stdout": "**",
        "sterr": "******",
        "exitStatus": 1
    },
    {
        "job": 2,
        "stdout": "",
        "sterr": "*",
        "exitStatus": 1
    },
    {
        "job": 3,
        "stdout": "***",
        "sterr": "*****",
        "exitStatus": 1
    },
    {
        "job": 4,
        "stdout": "**",
        "sterr": "",
        "exitStatus": 0
    },
    {
        "job": 5,
        "stdout": "*",
        "sterr": "***",
        "exitStatus": 0
    }

Nothing to be too excited about, but enough to demonstrate the Loop component works as intended.
I've gone for clarity and verbosity over cleverness; it could be polished further; I'm not doing that now as there are more features I want to add to the specification that will need at least a further iteration over the code.
The most significant differences between the PHP code and the specification are:

• In PHP, `resource's cannot be used as keys to arrays, so I've added some indirect maps to deal with that.
• Again, in PHP, processes will emit output on the STDOUT and the STDERR streams. To account for that, the Loop will read from both streams. I've taken some additional care to avoid calling proc_get_status twice on a terminated process, as the second call will return a -1 and not the actual exit code.

Besides the implementation differences, the structure is the same.
I've not gone as far as to use goto in the PHP code, though, and relied on regular loops.
The code lacks several security features like error handling and closing of the streams; it conveys the idea good enough as it is.

Next

In the next post, I will be adding fast-failure support to the Loop and start on a shared resources lock sharing system.

Previously

In the first post of the series, I've introduced the project I'm trying to complete using TLA+ and its specification verification power.
In the second post, I've started moving on with the specification first and the PHP implementation second.

This prior art contains insight into my thinking process and flow and provides a longer description of what I'm trying to build.

Pipes

My last iteration on the specification and implementation did nail the part where the main PHP thread, the Loop, dispatches the processing of each test block (whatever that will end up being) to separate PHP processes.
The issue with the last implementation was CPU-locking, all the possible mitigations of it (see the first post), and the lack of communication between the Loop thread and the worker threads.

Both issues can be solved using PHP stream_select function: the function, when provided a set of streams to observe, will block the execution of the PHP script until at least one of them is updated.

Why is this important? Because translating TLC's await keyword into PHP code that will actually wait, not poll on a CPU-locked cycle, cannot be achieved with many functions in PHP. Furthermore, the stream_select function will deal with the problem of the Inter-Process Communication (IPC from now on), allowing the Loop to get "push updates" from the workers.

Pipes specification

I've modeled this new approach in the specification below:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers, FiniteSets, Sequences
CONSTANTS Loop, JobSet, Parallelism, NULL

(*--algorithm loop
variables
	jobsCount = Cardinality(JobSet),
	_startedRegister = [x \in JobSet |-> FALSE],
	_processStatusRegister = [x \in JobSet |-> NULL],
	_processPipesRegister = [x \in JobSet |-> <<>>];

define
	StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE})
	NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE
	CompletedCount == Cardinality({x \in JobSet: _processStatusRegister[x] /= NULL})
	Running == StartedCount - CompletedCount
	ParallelismRespected == Running <= Parallelism
	StreamSelectUpdates == {x \in JobSet: _processPipesRegister[x] /= <<>>}
end define;

fair process loop = Loop
variables
	streamSelectUpdates = {},
	updatedProcess = NULL,
	processStatus = NULL,
	processToOutputMap = [x \in JobSet |-> <<>>],
	processToExitStatusMap = [x \in JobSet |-> NULL];

	begin
		StartInitialBatch:
			while StartedCount < Parallelism do
				with p = NextNotStarted do
					_startedRegister[p] := TRUE;
				end with;
			end while;

		WaitForStreamUpdates:
			\* Collect the live value into a copy.
			streamSelectUpdates := StreamSelectUpdates;
			await Cardinality(StreamSelectUpdates) > 0;

		HandleStreamUpdates:
			while Cardinality(streamSelectUpdates) > 0 do
				\* A hack to pop the Jobs from the updates one by one.
				updatedProcess := CHOOSE x \in streamSelectUpdates: 1 > 0;
				streamSelectUpdates := streamSelectUpdates \ {updatedProcess};

				GetProcessOutput:
					with processOutput = _processPipesRegister[updatedProcess] do
						processToOutputMap[updatedProcess] := Append(processToOutputMap[updatedProcess], processOutput);
					end with;
					_processPipesRegister[updatedProcess] := <<>>;

				GetProcessStatus:
					processStatus := _processStatusRegister[updatedProcess];

				UpdateTrackedProcessPipes:
					if processStatus /= NULL then
						processToExitStatusMap[updatedProcess] := processStatus;
					end if;

				CheckLoopStatus:

					if processToExitStatusMap[updatedProcess] /= NULL then

						MaybeStartOneMore:
							if StartedCount < jobsCount then
								with p = NextNotStarted do
								_startedRegister[p] := TRUE;
								end with;
								goto WaitForStreamUpdates;
							end if;

					end if;

				CheckAllDone:
					if Cardinality({x \in JobSet: processToExitStatusMap[x] /= NULL}) = jobsCount then
						goto Done;
					else
						goto WaitForStreamUpdates;
				end if;
			end while;
end process

fair process worker \in JobSet
begin
	WaitToStart:
		await _startedRegister[self] = TRUE;

	Work:
		either
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
		or
			skip;
		end either;

	ExitStatus: \* Either exit 0 or 1, a process MUST produce output.
		either
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
			_processStatusRegister[self] := 0;
			goto Done;
		or
			_processPipesRegister[self] := Append(_processPipesRegister[self], "*");
			_processStatusRegister[self] := 1;
			goto Done;
		end either;
end process;

end algorithm;*)
\* BEGIN TRANSLATION
\* [...]
\* END TRANSLATION 

OneWorkerPerJobStarted == <>[](StartedCount = Cardinality(JobSet))
AllWorkersCompleted == <>[](CompletedCount = Cardinality(JobSet))
=============================================================================

Before unpacking the code, a successful run with the following settings:

• What to check? - Temporal formula
• Deadlock - checked
• Properties - something that should eventually be true.

* `Termination` - all processes will always eventually terminate.
* `OneWorkerPerJobStarted` - each job will, eventually, have a worker assigned to it.
* `AllWorkersCompleted` - all workers should eventually complete.

• Invariants - citing the IDE "Formulas true in every reachable state".

* `ParallelismRespected` - There should never be a state with more workers than parallelism would allow.

The constants are set up as follows:

• JobSet <- [ model value ] {J1, J2, J3}
• Parallelism <- 2

Registers

The registers, prefixed with a _, are artifacts of the specification that will not make it into code: they represent, utilizing a global variable, the inter-process communication between the Loop and the workers.

To "start" the processes, I'm using the same approach used in the previous post: a _startedRegister variable that maps Jobs to their having a Worker started for them.
This specification global variable is the equivalent of the proc_open PHP function.
Where the proc_open function returns a process resource handle of the resource type, the specification will set the flag indicating a worker started for a job in the _startedRegister variable to TRUE.

The _processPipesRegister variable keeps track of the output emitted by the workers on the write end of the IPC pipe assigned to them.
The proc_open function will create, contextually with starting the process in a separate thread, "pipes" for inter-process communication. Typically one to communicate with the process, the STDIN one, and two to get output from the process: the STDOUT and STDERR ones. For the sake of the specification, I conflated the two pipes into one, and the IPC is represented by the worker process writing the _processPipesRegister global variable, and the Loop process reading from it.

The last register is the _processStatusRegister and represents what a call to the proc_get_status function would return in PHP: the exit status of the worker processes.

More cases mean more models

The specification passes when I use 3 jobs and parallelism of 2, but what about other cases?
I found out a "data-provider-like" approach to TLA+ is not how I should use the tools. To test different cases, I should create a new model. Better: clone the one I used so far and change the value of the constants.

The first alternate model is to test if the Loop specification works in "serial" mode: jobs are executed one after the other, with a parallelism of 1.

Nothing changes from previous checks, if not the constants:

• JobSet <- [ model value ] {J1, J2, J3}
• Parallelism <- 1

And it fails, the ParallelismRespected invariant violated at some point.

To debug the specification failure, I find it extremely useful getting, first, a view of the run, using the UI control to collapse all I can get an idea of where the violation happened (the MaybeStartOneMore phase of the Loop process) and what sequence led the run to that.

The first guiding light to find the cause of the model checking failure is that values changed by the state are highlighted in red; the second is that invariants and (temporal) properties will run after the state has changed the values.
After the MaybeStartOneMore state ran, the ParallelismRespected property is checked and will find 3 started processes in the _startedRegister variable, only 1 of which had its exit status collected and thus was deemed as completed. So, at this stage, there are 2 processes running which is more than the allowed parallelism of 1.

The fix is easy enough: in the MaybeStartOneMore phase, I have to check two conditions:

• There are still jobs that do not have workers assigned; the specification was already checking this.
• And the number of running workers is less than, or equal to, the parallelism; this check is the missing one.

Translated in PlusCal, the MaybeStartOneMore phase will change as follows:

MaybeStartOneMore:
-	if StartedCount < jobsCount then
+	if StartedCount < jobsCount /\ Running < Parallelism then
		with p = NextNotStarted do
		_startedRegister[p] := TRUE;
		end with;
		goto WaitForStreamUpdates;
	end if;

Now both models will pass.

More parallelism than jobs

The last model I want to check against the specification is the one where the parallelism is more than the number of jobs.

Clone another model and set:

• JobSet <- [ model value ] {J1, J2}
• Parallelism <- 3

And this one fails again; this time, it's a TLC syntax violation.

The message:

TLC threw an unexpected exception.
This was probably caused by an error in the spec or model.
See the User Output or TLC Console for clues to what happened.
The exception was a java.lang.RuntimeException
: Attempted to compute the value of an expression of form
CHOOSE x \in S: P, but no element of S satisfied P.
line 102, col 19 to line 102, col 83 of module spec_loop
The error occurred when TLC was evaluating the nested
expressions at the following positions:
0. Line 131, column 22 to line 142, column 62 in spec_loop
1. Line 131, column 25 to line 131, column 54 in spec_loop
2. Line 132, column 25 to line 137, column 61 in spec_loop
3. Line 133, column 33 to line 135, column 82 in spec_loop
4. Line 133, column 36 to line 134, column 94 in spec_loop
5. Line 134, column 38 to line 134, column 94 in spec_loop
6. Line 134, column 58 to line 134, column 94 in spec_loop
7. Line 134, column 85 to line 134, column 85 in spec_loop
8. Line 133, column 45 to line 133, column 58 in spec_loop
9. Line 102, column 19 to line 102, column 83 in spec_loop

Three things to note:

• The line and columns number refer to the translation, not the PlusCal code. So, better learn to read that.
• The message is pretty clear about the failure but will use symbols.
• TLA+ does not handle the concept of returning anything.

When I say that the error report will use symbols, I mean that I did not write CHOOSE x \in S: P, but no element of S satisfied P anywhere in my PlusCal code, and that is nowhere to be found in the translation.
That error is about this line of the PlusCal code:

NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE

And when I say that TLA+ does not handle returning anything, I mean that it does not work as PHP would.
If I had to write the same code in PHP I would translate the code above into the following:

$_startedRegister = ['J1' => true, 'J2' => true];
$nextNotStarted = array_filter($_startedRegister, function($started){ return !$started; });

Next

In the next post, I will make the specification more robust adding more invariants and properties to, finally, move into the PHP translation of it.

Previously

The previous post on the subject introduced the project I'm working on and why I've decided to use TLA+.
This post will take off from the end of that to complicate my specification to include something closer to what I need to implement.

Atomic labels

Closing the first post, I've said that a "step" in TLA+ is marked by a "label" and is "atomic". In the context of a process block, a label is anything that looks like This: (alpha and then a :). A label has a body of statements that happen inside it that will occur atomically, in the order they appear.
The best way I can explain it is with an example:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers

(*--algorithm loop
variables
	accumulator = 0; \* Start with a value of 0.

define
	AccumlatorLessThanEqualOne == accumulator <= 1 \* We'll use this as invariant.
end define;

fair process p \in {1,2} \* 2 processes will run at the same time.
	begin
		Increase_Accumulator:
			accumulator := accumulator + 1;
		Decrease_Accumulator:
			accumulator := accumulator - 1;
end process

end algorithm;*)
=============================================================================

I've omitted the translation of the PlusCal code to TLC as it's machine-generated and not that interesting.
I've set up my model to this:

• What to check? - Temporal formula
• Deadlock - checked
• Properties - something that should eventually be true.

* `Termination`, both processes should terminate.

• Invariants - citing the IDE "Formulas true in every reachable state".

* `AccumlatorLessThanEqualOne` - the value of the `accumulator` variable should be <= 1

The last part, the one where I set the AccumlatorLessThanEqualOne invariant, is the one that will make the model fail.
The model will fail because there is at least one state where the value of accumulator is not <= 1.
The TLA+ will report on such state:

The UI requires highlights in red what changed between a step and the next:

• At the Initial predicate, the start of the model checking, the accumulator value is 0, both processes will execute the Increase_Accumulator step next, the AccumlatorLessThanEqualOne invariant is not violated.
• The model checking picks process 2 to execute its Increase_Accumulator step: that will increase accumulator by 1. All fine and dandy.
• The model checking will, then, pick process 1 to execute, and that too will complete the Increase_Accumulator step, adding 1 to the accumulator and violating, thus, the AccumlatorLessThanEqualOne invariant.

The model checker output reads Invariant AccumlatorLessThanEqualOne is violated..

There is, indeed, a flow in which one process executes both steps, the Increase_Accumulator and Decrease_Accumulator ones, in sequence, and accumulator is never > 1.
But the model checker just verified that our logic will not hold under the pressure of concurrency.

There is a fix for this: before increasing the accumulator value, wait for it to be precisely 0:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers
CONSTANTS Loop, NULL

(*--algorithm loop
variables
	accumulator = 0;

define
	AccumlatorLessThanEqualOne == accumulator <= 1
end define;

fair process p \in {1,2}
	begin
		Increase_Accumulator:
			await accumulator = 0; \* If the accumulator value is not 0, wait.
			accumulator := accumulator + 1;
		Decrease_Accumulator:
			accumulator := accumulator - 1;
end process

end algorithm;*)
=============================================================================

This solution passes, and it shows the use of a powerful feature of TLA+: the await keyword.
The keyword will stop executing that process step until the condition after await is met.

Atomic PHP

My purpose in using TLA+ is to develop good specifications for ideas that I will then have to translate into code.
That code happens to be PHP, and I need to understand what "atomic" means in the context of PHP.

I wrote a small script to test out what atomic execution means in PHP:

<?php
$isWorker = isset($argv[1]);

if($isWorker){
		$id = $argv[1];
		foreach(range(1,200) as $k){
				echo $id; // Print the process name.
		}
		exit(0);
}

foreach(range(1,2) as $n){
		$command = PHP_BINARY . ' ' . escapeshellarg(__FILE__) . ' ' . $n;
		$specs = [STDIN, STDOUT, STDERR];
		proc_open($command, $specs, $pipes);
}

sleep(1); // Dumb wait for all processes to be done.
echo "\n";
exit(0);

Running the script will yield a variation of this output:

22222222222222222111111111111111111112111212221212121212121212121212121212121212
12121212121212121212121212121212121212121212121212121212121212121212121212121212
12121212121212121212121212121212121212121212121212121212121212121212121212121212
12121212121212121212121212121212121212121212121212121212121212121212121212121212
12121212121212121212121212121212121212121212121212121212121212121212121212121222

Note: I've used proc_open as it's a "fire and forget" function that will not block the execution of the main script waiting for the processes to finish. Using passthru, as an example would stop the main script to wait for the started script to complete.

The processes get their turn to print in a non-deterministic order.
A note: using smaller numbers will incur in some execution caching that will unwrap the execute the foreach a first time and will then store and just print a string; that is why there is a 200 max in there.
The processes will print when they are allocated CPU time; the allocation of that CPU time cannot be known before the execution, which means the atomicity of PHP is a single instruction.

The takeaway is: after each PHP instruction, the CPU might be allocated to another PHP thread.
There is no guarantee another PHP thread will not get the CPU and do something between two adjacent PHP instructions, like an echo following another in the example code.

This information will come in handy later when it is time to translate the specification to code.

A first Loop specification

After exploring some key concepts, it's time to get back to the Loop specification.
The first version of the specification leaves much to be desired but is helpful to illustrate more logic and TLA+ constructs I will use.

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers, FiniteSets
CONSTANTS Loop, NULL

(*--algorithm loop
variables
	jobsCount = 5,
	maxParallelism = 2,
	_startedRegister = [x \in 1..jobsCount |-> FALSE];

define
	StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE})
	NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE
end define;

fair process loop = Loop
	begin
		StartInitialBatch:
			while StartedCount < maxParallelism do
				with p = NextNotStarted do
					_startedRegister[p] := TRUE;
				end with;
			end while;
end process

fair process worker \in 1..5
begin
	WaitToStart:
		await _startedRegister[self] = TRUE;
	Work:
		goto Done;
end process;

end algorithm;*)
=============================================================================

I've imported the FiniteSets module into my specification. This module contains functions, like the Cardinality one, that allows me to operate on sets.
Sets are lists of elements of the same type: {1, 2, 3} is a set of integers with a cardinality of 3, 1..3 is a quick way of defining the set {1, 2, 3}.

This version of the specification defines two types of processes: one is the Loop, the other is the worker.
There will be one Loop and 0 or more workers at any given moment.

The first construct I'm introducing is the _startedRegister variable.
I'm sure someone with more experience would develop a better solution, but I'm stuck with my knowledge of the subject.
Any _variableName prefixed with _ indicates a "technical" variable that I've defined in the specification, but that will most likely not be translated to PHP code. To understand the purpose of the _startedRegister variable, look at the worker process: the process worker will start in the WaitToStart phase and will move to the next phase, the Work one, only when it is started from the Loop.
The definition of the _startedRegister variable is _startedRegister = [x \in 1..jobsCount |-> FALSE] and it can be read like "a map from jobs to the boolean false value". It's better understood in PHP array terms:

$_startedRegister = [1 => false, 2 => false, 3 => false, 4 => false, 5 => false];

So, when the Loop sets the flag for process 2 to TRUE, the await condition of worker 2 will be satisfied, and it will start.

The StartedCount and NextNotStarted definitions are operators. They are like user-defined functions in PHP.
It's worth reiterating TLA+ is not an implementation language but a specification one; it expresses operations, for the most, using mathematical definitions and logic operations.

StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE}) means "StartedCount is the number of keys of _startedRegister for which the value for that key is true".
Translate that to PHP:

global $_startedRegister;

function StartedCount(){
	return array_count(array_filter($_startedRegister));
}

On the same note, NextNotStarted picks the key first element in _startedRegister the value of which is false; DOMAIN _startedRegister is like array_keys($_startedRegister. To end the syntax tour, with is used to define a local variable.
Read more about TLA+ syntax through Leslie Lamport's video course, on the Learn TLA+ site and from Hillel Wayne's book.

Running the specification now, checking for Deadlock and Termination will fail due to a deadlock.
I've defined five workers to run, but only 2 will be started in the StartInitialBatch phase of the Loop; TLA+ is telling me 3 workers are deadlocked: they will await forever for something that will never happen.

Maybe start all the processes?

After some work, I came up with a second iteration of the specification that is starting all the processes:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC, Integers, FiniteSets, Sequences
CONSTANTS Loop, JobSet, Parallelism, NULL

(*--algorithm loop
variables
	jobsCount = Cardinality(JobSet),
	_startedRegister = [x \in JobSet |-> FALSE],
	_exitStatusRegister = <<>>,
	exitStati = <<>>;

define
	StartedCount == Cardinality({x \in DOMAIN _startedRegister: _startedRegister[x] = TRUE})
	NextNotStarted == CHOOSE x \in DOMAIN _startedRegister: _startedRegister[x] = FALSE
	Running == StartedCount - Len(exitStati)
	ParallelismRespected == Running <= Parallelism
	CompletedCount == Len(exitStati)
end define;

fair process loop = Loop
	begin
		StartInitialBatch:
			while StartedCount < Parallelism do
				with p = NextNotStarted do
					_startedRegister[p] := TRUE;
				end with;
			end while;
		CheckExits:
			if Len(_exitStatusRegister) > 0 then
				CollectExitStatus:
					with status = Head(_exitStatusRegister) do
						exitStati := Append(exitStati, status);
					end with;
					_exitStatusRegister := Tail(_exitStatusRegister);
				CheckExitedCount:
					if Len(exitStati) = jobsCount then
						goto Done;
					end if;
				MaybeStartOneMore:
					if StartedCount < jobsCount then
						with p = NextNotStarted do
							_startedRegister[p] := TRUE;
						end with;
					end if;
					goto CheckExits;
			else
				goto CheckExits;
			end if;
end process

fair process worker \in JobSet
begin
	WaitToStart:
		await _startedRegister[self] = TRUE;
	Work:
		skip;
	Exit:
		_exitStatusRegister := Append(_exitStatusRegister, 0);
end process;

end algorithm;*)
\* BEGIN TRANSLATION
\* [...]
\* END TRANSLATION 

OneWorderPerJobStarted == <>[](StartedCount = Cardinality(JobSet))
AllWorkersCompleted == <>[](CompletedCount = Cardinality(JobSet))
=============================================================================

I will not go through the specification line by line and concentrate on important parts.

At the bottom, after the translation, I've defined a temporal formula: it looks like an operator between parentheses and temporal "symbols" before it.
Specifically:

• <> means "Eventually"; there is at least one state where the condition between the parentheses is true.
• [] means "Always"; what is between the parentheses is always true.

"Eventually always" means "When it happens for the first time, then it should keep being true".
Eventually, a flower pot will fall and break apart; it will then always be broken. If someone came and put the pot back together, then it would violate the "always" part, and the model verification would fail.

The rest of the code is operators to make the code more readable.
I've added one more technical variable, the _exitStatusRegister one. This is a sequence, denoted with <<>>. In sequences, order matters; in sets, type matters.
That represents a worker process ending and returning its exit status to the Loop process.

This specification is not doing all I need it to do, but will pass verification with the following settings:

• What to check? - Temporal formula
• Deadlock - checked
• Properties

* `Termination`
* `OneWorderPerJobStarted`
* `AllWorkersCompleted`

• Invariants

* `ParallelismRespected`

• Constants

* `JobSet <- [ model value ] {J1, J2, J3, J4, J5}`
* `Parallelism <- 2`

Five jobs and a parallelism of two.

This is good enough to try and translate it into code.

Translating the specification

<?php

function worker()
{
    exit(0); // Just exit, like skip.
}

function startWorker($key)
{
    $command = sprintf("%s %s %s", PHP_BINARY, escapeshellarg(__FILE__), $key);
    return proc_open($command, [STDIN, STDOUT, STDERR], $pipes);
}

function loop(array $jobSet, $parallelism)
{
    $batchSize = min(count($jobSet), $parallelism);
    $started = [];
    $exitStati = [];

    foreach (range(1, $batchSize) as $k) {
        $started[] = startWorker($k);
    }

    while (true) {
        $procKeysToRemove = [];
        foreach ($started as $process) {
            $procStatus = proc_get_status($process);

            if ($procStatus['running']) {
                continue;
            }

            $exitStati[] = $procStatus['exitcode'];
            $procKeysToRemove[] = array_search($process, $started, true);

            if (count($exitStati) === count($jobSet)) {
                break 2;
            }

            if (count($started) < count($jobSet)) {
                $started[] = startWorker(++$k);
            }
        }

        // Reduce the checked set as they complete.
        foreach ($procKeysToRemove as $key) {
            unset($started[$key]);
            $started = array_values($started);
        }
    }

    return $exitStati;
}

if (!isset($argv[1])) {
    $exitStati = loop(range(1, 5), 2);
    echo "\n", print_r($exitStati, true);
} else {
    worker();
}

The code should work on any machine, should one want to try it out.
Just one note: calling proc_get_status a second time on an already exited process will yield an exit value of -1. To cope with that and to get a smaller optimization to the loop logic, I remove the processes from the $started array when done.
The non-technical variables from the specification are all there, and the loop works, but with two significant issues:

1. That while( true ) is a CPU-locking loop. If the workers were running long enough, it would hoard CPU time running continuously, bringing the CPU load to 100% with wasted energy consumption: most of those cycles would not update a state.
2. The workers do something, but what? There is no communication from them back to the loop. Did they fail? Did they explode?

As pipes for the workers, I'm setting the STD ones, but that is just to make the code shorter. Letting the workers print to the same output stream as the loop is a bad idea as it does not allow to control the print order, and thus formatting, and does not allow the Loop to get any information beyond an exit status.

Next

In my next post, I will add pipes and Inter-Process Communication (IPC) to the mix to see how the specification and the PHP code will evolve.

What is TLA+?

In my personal understanding of it: a specification language that helps me explore fallacies in my PHP software architecture and test that architecture under the pressure of concurring threads. To better explain what TLA+ is, one should refer to the site of Leslie Lamport, the mind behind TLA+.
I've found Hillel Wayne's site and book to be the best way for me to learn the basics.

The pitch for this latest exploration of mine was Wayne's video "Tackling Concurrency Bugs with TLA+": it struck a chord. The concurrency part especially. Then I found out there was testing involved and could not let go of the thought.

What am I using TLA+ for?

Given enough hot metal under it, TLA+ could really be used to model any concurring process my mind could think of, from high-level distributed systems to low-level algorithms.
Yet I hate answers that sound like a variation of the "it depends on your situation and context", and I will avoid that by providing the particular context I'm applying TLA+ to.

I'm using TLA+ to model parts of a process-based PHP testing framework I'm writing.
I'm still pretty foggy about how this testing framework will fit in the grand scheme of things, but I've spent some time working with Jest and found it to be really delightful.
I'm writing something from scratch and, without the pressure of having to ship a product, I can take my time to do it right and in a way that satisfies me.

Let's start from an example file, a "spec files" in Jest jargon, this imaginary test framework would consume:

<?php
/**
 * @env WordPress
 */

describe('get_foos_function', function () {
    // Create the test users.
    $admin      = wp_insert_user(['user_login' => 'a', 'user_pass' => 'a', 'role' => 'administrator']);
    $subscriber = wp_insert_user(['user_login' => 's', 'user_pass' => 's', 'role' => 'editor']);
    // Create the test foos: one public, one draft.
    foo_create(['status' => 'publish']);
    foo_create(['status' => 'draft']);

    describe('Admin context', function () use ($admin, $subscriber) {
        // Simulate Admin context.
        define('WP_ADMIN', true);

        it('should return all foos to admin user', function () use ($admin) {
            wp_set_current_user($admin);
            $foos = get_foos();
            expect(count($foos))->toBe(2);
        });

        it('should return only public foos to non-admin', function () use ($subscriber) {
            wp_set_current_user($subscriber);
            $foos = get_foos();
            expect(count($foos))->toBe(1);
        });
    });

    describe('Front-end context', function () use ($admin) {
        it('should return public foos to admin user', function () use ($admin) {
            wp_set_current_user($admin);
            $foos = get_foos();
            expect(count($foos))->toBe(1);
        });

        it('should return public foos to non-admin', function () {
            wp_set_current_user(0); // Visitor.
            $foos = get_foos();
            expect(count($foos))->toBe(1);
        });
    });

    describe('REST context', function () use ($admin) {
        // Simulate a REST request.
        define('REST_REQUEST', true);

        it('should return all foos to REST admin user', function () use ($admin) {
            wp_set_current_user($admin);
            $foos = get_foos();
            expect(count($foos))->toBe(2);
        });

        it('should show no foos in REST to visitors', function () {
            wp_set_current_user(0); // Visitor.
            $foos = get_foos();
            expect(count($foos))->toBe(0);
        });
    });
});

To note in the above code example:

• The @env WordPress annotation will tell the (again: still hypothetical) testing framework that WordPress should be loaded before the tests. This will allow the following code to call WordPress functions.
• The block at the top creating the users and the "foos" would be the equivalent of Jest beforeAll function, or PHPUnit setUpBeforeClass method call: it will do something once, before any test runs.
• In the inner describe blocks, I'm defining constants like WP_ADMIN and REST_REQUEST to simulate, respectively, the admin and the REST request contexts.
• Was this code to run sequentially, the second describe block, the Front-end context one, would inherit a defined WP_ADMIN constant that would make it fail.
• Being "smart" about the order of the describe blocks here, then, would not change much as at least one describe block would end up inheriting the constant defined by one of the describe blocks that ran before.

The "simple" solution is to run each describe block in isolation in a dedicated, separate PHP process.

Let's say I'm willing to do it: there will be a main tread in charge of starting, and managing the state of, multiple dedicated PHP processes running the describe blocks.

Loop concerns

A loop waiting on processes to be done is a pretty well-explored pattern.
Yet there are challenges deriving from the specific application and PHP limits.

A quick list:

1. Processes might succeed or fail and will need to communicate that back to the main PHP process for it to be able to orderly print the output: this means Inter-Process Communication (IPC) is required.
2. On Windows, one of the possible environments where this testing framework might run, IPC is not that easy given the many IPC pipes and streams inconsistencies.
3. A loop should not be implemented with a logic that sounds like, "Every now and then check on the started processes, sleep a bit, repeat". Checking more often would make the Loop more responsive but hoard CPU cycles. Checking less often would reduce resource consumption, making the Loop less responsive.

Some prototypes later, I concluded I should use the stream_select function to "watch" the pool of running processes from the main PHP thread, get prompt updates, and avoid a CPU lock.
When used on blocking streams, the function will wait for some streams part of the pool (a stream could be opened over a file or an IPC pipe) to update to move on.

All of this long and winded introduction to get to the first TLA+ model.

CLI and TLA Toolbox

To verify my specification, I'm using the TLA+ toolbox; I'm not fond of the IDE part, though, and prefer using vim and Hillel Wayne's vim plugin to write my specifications.

With that out of the way, let's start with my model.

I would like to reiterate I'm not a TLA+ expert. I am sharing my discovery process and findings. Go back to the first section of this article to find the references and literature to provide expert insight into it.

I will be dealing, to model the Loop and ancillary PHP processes, with two types of process:

1. The Loop, the PHP process the user starts that will, in turn, start and manage one PHP process per describe block.
2. The Worker, a PHP process started by the Loop to handle a describe block.

I've created a new spec_loop specification in the TLA+ Toolbox application and wrote the first version of my specification, just to check if things work.

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC
CONSTANTS Parallelism, Jobs, Loop, NULL

(*--algorithm loop

process loop = Loop
	begin
		StartInitialBatch:
			goto Done;

end process

end algorithm;*)
=============================================================================

The TLA+ Toolbox supports two languages:

• TLC - the somewhat difficult to read and write specification language initially used for TLA+.
• PlusCal - the higher-level language that will transpile to TLC.

PlusCal is what I put between (*--algorithm loop and end algorithm;*).
When I save the file and click "Translate PlusCal Algorithm", I get the following output:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC
CONSTANTS Loop, NULL

(*--algorithm loop

process loop = Loop \* There is process called Loop
	begin
		StartInitialBatch: \* That process starts...
			goto Done; \* ...and directly goes to Done, a state that means it's terminated.

end process

end algorithm;*)
\* BEGIN TRANSLATION (chksum(pcal) = "f8a984b0" /\ chksum(tla) = "83520898")
VARIABLE pc

vars == << pc >>

ProcSet == {Loop}

Init == /\ pc = [self \in ProcSet |-> "StartInitialBatch"]

StartInitialBatch == /\ pc[Loop] = "StartInitialBatch"
                     /\ pc' = [pc EXCEPT ![Loop] = "Done"]

loop == StartInitialBatch

(* Allow infinite stuttering to prevent deadlock on termination. *)
Terminating == /\ \A self \in ProcSet: pc[self] = "Done"
               /\ UNCHANGED vars

Next == loop
           \/ Terminating

Spec == Init /\ [][Next]_vars

Termination == <>(\A self \in ProcSet: pc[self] = "Done")

\* END TRANSLATION 

=============================================================================
\* Modification History
\* Last modified Wed Mar 16 10:46:22 CET 2022 by lucatume
\* Created Mon Mar 14 08:40:43 CET 2022 by lucatume

The \* BEGIN TRANSLATION and \* END TRANSLATION comments fence the translation operated by the IDE.
In the following code examples, I will not report the translation as that is a direct output of the PlusCal algorithm and not that interesting to this post.
After \*, you see comments in PlusCal and TLC syntax.

Notes:

• EXTENDS is like import in node: import one or more modules I will need.
• CONSTANTS defines the inputs my specification will have. They are the parametric part of my specification, the values Models will use to customize a check.

The specification is easy enough, I create a Model called "MC" in TLA+ Toolbox and set it up like this:

• What is the behavior spec? -- Temporal formula.
• What to check?

* `Deadlock`
* `Properties`
	* `Termination`

• What is the model?

* `NULL <- [ model value ]`
* `Loop <- [ model value ]`

Here's a screenshot of the settings in the TLA+ toolbox:

What does all this mean?

Temporal formula means I want to check if my specification works over time. More on that later.
Deadlock means I want to check if there's any state, a combination of execution steps, that would put the model in a deadlock.
Termination means I want to check that, in any state, all the processes will get to the Done phase, the last one. NULL and Loop are custom types I will use in my specification.

I run the model and... it fails: Temporal properties were violated., Stuttering.
That means the one temporal property I've set, the one where all processes should eventually terminate, is violated.
Here comes the first great power of TLA+: it will model states where some, or all, processes will not run.
If the Loop process never runs, it will not terminate. As such, the Termination temporal property is violated.
I'm not interested in modeling the case where the Loop does not run. That means the PHP code never got there, which implies some error happened before, the user is shown some exception or message: I do not need a specification for that.

Please, run the Loop process

How do I tell the model checker I'm not interested in a state where the Loop process does not run?
I tell it the process is fair. It means any state where the Loop process does not run should not be part of the model check.

I update the specification to make the Loop process a fair one:

----------------------------- MODULE spec_loop -----------------------------
EXTENDS TLC
CONSTANTS Loop, NULL

(*--algorithm loop

fair process loop = Loop
	begin
		StartInitialBatch:
			goto Done;

end process

end algorithm;*)
=============================================================================

After re-compiling the code and rerunning it, the model check passes: the Loop process has only one step: the StartInitialBatch one, which will immediately go to the Done one. A "step" in TLA+:

• Is indicated by a Label:
• Is atomic.

This is enough for one post.

Check out the second post.

This post is the second in a series of posts where I will document my building of the WordPress testing framework I would have liked to find when I started working with WordPress.

The name of this framework in-fieri is ork, and you can read more about it in the first post dedicated to it.

Getting the whole testing setup up and running

In my previous post, I've set up some basic tests that got some of the job done. Still, I would like to come back to those to set them up once, while the involved logic is straightforward for me to follow, addressing Continuous Integration and IDE integration to start on the right foot.

For CI, I've picked GitHub Actions: it comes included in my plan and is fast enough and reliable enough for my current needs.
Being the lover or make I am, I've set up the project Makefile to allow me to build and run the tests by sticking to the trusted pattern of make build && make test:

name: CLI App CI
on: [push]
jobs:
  bdd:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          ref: 'main'
      - run: cd app && make build && make test

Not much to see here.
To avoid the back and forth of testing GitHub Actions through pushing and waiting for the worker to run, I'm using nektos/act. The one-file binary will run any workflow in the project .github directory using a container image that is the same as the one used by GitHub actions workers for most intents and purposes.
I have configured my machine ~/.actrc file to correctly alias the ubuntu machines.

-P ubuntu-latest=nektos/act-environments-ubuntu:18.04
-P ubuntu-18.04=nektos/act-environments-ubuntu:18.04

After confirming the tests are passing locally, I've been rewarded with a green light on GitHub Actions too.

Just a note: act will simulate the GitHub Actions environment almost perfectly, but the GITHUB_TOKEN environment variable that GitHub Actions will set up by default will be missing.
Since I'm working in a private repository, I've created a token dedicated to nektos/act in the ~/.github-token file that I export at run time to allow act to pull and test private dependencies the same way GitHub Actions would access them:

GITHUB_TOKEN="$(<${HOME}/.github-token)" act

This token definition is all it takes for act to work correctly.

Laying out the first steps

I made a habit of starting my development work from the end.

This approach helps me thinking of the "what" before thinking about the "how".

The project requirements will, roughly, be:

1. Download ork for your current operating system somewhere the terminal can execute it from your terminal emulator of choice.
2. Use whatever local development webserver solution you want to host your development WordPress server.
3. Navigate to the directory where your testing project will live.
4. Run ork and start testing.

Point 2 above, especially, is something I care about.
One of the things the wp-browser project taught me is I should not make assumptions about what operating system, or development stack, different developers will use to create, maintain and test WordPress projects.

There are many solutions available for WordPress developers, ranging from PHP built-in server to Mamp and other host-based solutions to complex Docker-based stacks or virtual machines.
Each one has its complications from the perspective of a testing framework: some are running "on the metal" (e.g., PHP built-in server, Mamp or valet), some are using containers (e.g., Devilbox or the previous version of Local by Flywheel) and some others are using virtual machines (e.g., the VVV project).
Understanding where the PHP server serving the development WordPress server is, in respect to the machine that is executing the tests, is where several users get lost and give up.

Say I have set up Codeception tests with wp-browser on my host machine. I've installed WordPress, locally, at /Users/lucatume/Sites/wp, but I am running my tests from within a Docker container. The path to the WordPress installation will probably be /var/www/html as that is the path to the WordPress installation in the default WordPress Docker image.

The point is: there are plenty configurations and local development server solutions out there that work for those using them.
I want ork to remove the friction of dealing with each and "just work" as long as one can visit the WordPress site using a browser.
Can you go to https://wordpress.local? Fine, ork should work.
Do you have your site served with PHP built-in server at http://localhost:8888? This setup, too, is fine, ork should work with it.
Have you set up WordPress with docker-compose where Docker runs into a Linux VM running, in turn, on your Windows machine? If you can visit the WordPress site from your browser, then ork should work.

All the different setups above have in common that the user can visit the development site URL. The ork binary will have to rely, at least initially, only on that piece of information to work.

The first step, then, assuming points 1 to 3 are taken care of by the user, is to model the following scenario:

» mkdir ork-tests
» cd ork-tests
» ork

What is the URL of the site you would like to test? ork.local

Checking site URL ... ok
Creating the first spec ... ok
Running specs ...

  Homepage
    √ it works (410ms)

  1 passing (422ms)

Those familiar with JS testing should recognize mocha BDD output.
Since I'm currently using it to power my tests, I've just used the inspiration to draft this imaginary CLI output.

There's a bit to get done before getting to the actual running of tests, in any case, so I better get to it.

Testing a binary and the command line

The first test I would like to write, in Gherkin syntax, would read out like this:

Given I have changed the directory to one that does not contain an `ork` configuration file
When I run `ork`
Then I should be asked to provide the URL of the site I would like to run tests for
And I should see an `ork.config.json` file created containing the URL I provided

This seems a simple enough test to translate into a mocha test.

It turned out not to be that simple for me, though.
Being, in essence, a novice when it comes to NodeJs development, it took me some time to come to terms with how the loop of execution is structured and how my code will execute.

Starting from the end here is the first test I wrote that is, currently, passing:

// test/FirstSetup.js

const { existsSync } = require('fs')

// I've created a test support library of functions to allow me to write tests the way I want. 
const { invokeOrk, mkTmpDir, forFile, toReadFile } = require('./_support.js')

// I'm using `chai` to write assertions in spec/BDD format.
const expect = require('chai').expect 

describe('First setup', function () {
  it('should prompt the user to create config file if not found', async function () {
    // As a first step, create a test directory in the test/_output directory. Yes, Codeception habit.
    const tmpDir = mkTmpDir()
    
    // This is where the configuration file should be, if everything goes according to plan.
    const configFilePath = tmpDir + '/ork.json'
  
    // To begin with, the file should not exist.
    expect(existsSync(configFilePath)).to.be.false
  
    // More on this later, I invoke the compiled binary in the test directory.
    let event = await invokeOrk([], { cwd: tmpDir })

    // The first event should be output coming from the binary, asking the site URL.
    expect(event.type).to.equal('stdout')
    expect(event.data).to.match(/url.*site/i)
    
    // Reply to the request with `ork.local`.
    event.proc.stdin.write('ork.local\n')
    
    // Wait up to 5s for the configuration file to appear.
    const exists = await forFile(configFilePath, 5000)
    expect(exists).to.be.true
    
    // Read the configuration file contents and make sure they are correct.
    const configFileContents = await toReadFile(configFilePath, 'utf-8')
    const config = JSON.parse(configFileContents)
    expect(config.url).to.equal('http://ork.local')
  })
})

There are some "gotchas" I collected along the way.
They might be obvious to more experienced Node developers, but this is all pretty new to me:

1. Due to Node single-thread nature, blocking that only thread on one operation cannot be done. The single thread nature of Node is not that different from PHP, what makes it very different is that thread is also the one in charge of handling the user input, and blocking it would mean freezing any UI; there is no UI in PHP. The issue is not that pressing in a CLI application, but most Node primitives are not synchronous, and they will yield control immediately. Appreciating and understanding this required some adjustment on my side.
2. All this async code can be made sync-ish using async\await; "ish" as what will happen is that Node will move to another task at the same scope level.
3. Most existing primitives can be "promisified" to make them return promises when they would, instead, fire callbacks when something happens.

Grasping all the above did take me some time, but I'm pretty satisfied with two solutions in particular in thetest/_support.js file.

The first is the one that allows me to run the sub-process that is running the ork binary in "events".
Possible events are stdout, error, stderr and exit.
Each event will return a Promise to get the next event, and this is what allows me to write event-by-event tests in my first spec and writing the test above in a way that closely resembles the interaction the user would have with the CLI:

// Build a Promise for the process next event.
function processPromise (proc) {
   return new Promise(function (resolve, reject) {
      for (let eventName of ['error', 'data', 'exit']) {
         for (listener of proc.listeners(eventName)) {
            proc.off(eventName, listener)
         }
      }

      proc.on('error', function (error) {
         reject(error)
      })

      proc.stdout.on('data', function (chunk) {
         resolve({ type: 'stdout', data: chunk, proc: proc, next: processPromise(proc) })
      })

      proc.stderr.on('data', function (chunk) {
         resolve({ type: 'stderr', data: chunk, proc: proc, next: processPromise(proc) })
      })

      proc.on('exit', function (code) {
         resolve({ type: 'exit', data: code, proc: proc, next: processPromise(proc) })
      })
   })
}

async function invokeOrk (args = [], options = {}) {
   const mergedOptions = {
      ...{
         env: { ...process.env, ...{ NODE_OPTIONS: '' } },
         stdio: ['pipe', 'pipe', 'pipe'],
      }, ...options
   }
  
   // Spawn the process, an async task, and set up some defaults.
   const proc = spawn(locateBin(), args, mergedOptions)
   proc.stdout.setEncoding('utf8')
   proc.stderr.setEncoding('utf8')
   
   // Return the Promise that will yield the process first event; probably output or an error.
   return processPromise(proc)
}

The next piece of code I'm proud of is the one that allows me to wait for the file to come into existence.
In the tests, I use it to wait for the configuration file to appear, but I'm sure it will come in handy again:

// The name is to follow its most likely use of `await forFile`.
async function forFile (file, timeout = 5000) {
   return new Promise(function (resolve, reject) {
      const basename = file.replace(dirname(file) + '/', '')

      if (fs.existsSync(file)) {
         // If the file already exists when we start, resolve now.
         resolve(true)
      }
  
      // Set a timeout: either the files appears before it runs out, or fail.
      const timeoutRef = setTimeout(() => {
         watcher.close()
         resolve(false)
      }, timeout)
  
      // Start watching the directory that should contain the file.
      const watcher = fs.watch(dirname(file), async (eventType, filename) => {
         if (eventType === 'rename' && filename === basename) {
            if (fs.existsSync(file)) {
               // The file came into existence, block the timer and resolve.
               watcher.close()
               timeoutRef.unref()
               resolve(true)
            }
         }
      })
   })
}

Finally, the test output at the end of all the tribulation:

» make test


  First setup
    ✓ should prompt the user to create a config file if not found (354ms)


  1 passing (360ms)

» tree -L 3 test/_output
test/_output
└── dirs
    └── KJvSkAw4Nj
        └── ork.json

2 directories, 1 file
» cat test/_output/dirs/KJvSkAw4Nj/ork.json
{
  "url": "http://ork.local"
}

Next

Getting to this first test to pass took some effort, but I'm glad I did it.
Next time I will get the CLI output, right now pretty spartan, into shape and extend the functionalities using BDD.

The backstory or "The part you skip because you do not care about it"

What's a good introduction without a backstory?
When I started working with PHP and WordPress about eight years ago, my first line of PHP was a PHPUnit test.

I only wrote code using TDD in my previous career, and really thought one could not write PHP code without testing it.
I look back at that time smiling: there are, indeed, people out there somewhere that do write code without writing tests for it.

As I spent time working on more WordPress projects and with more teams, I appreciated some factors that have not changed but softened my stance on testing.

First, writing automated tests for WordPress is not that widely documented, and the wider community is not that "into it". I've seen some talks at WordCamps about testing, but not that many. Mind: there are developers that do that, just not so many to tip the scale of the typical WordPress conversation; if you're doing it, you've got all my appreciation.

Second, there are tools and frameworks out there that will make testing WordPress easier; wp-browser objective is precisely that. Still, there are wide cracks in how they allow different types of testing to be covered, and if a developer's need falls in one of those cracks, then that developer will have to do a lot of research and problem solving to get anything done. When one masters the instrument, then the payoff will be immense, but that's a steep price to pay to start.

Third: most testing tools will work well on new projects and either impose a "refactoring tax" on the existing code to provide any meaningful return or outright not allow it.

With a touch of gatekeeping and elitism, all of the above pushes well-intentioned, but inexpert, people away from adopting tests far too often by conflating adoption and maintenance price into, simply, too high a price to pay to "just test code".

One test is better than none

When consulting with companies trying to adopt testing approaches, my first step is usually to demystify testing.

Sure, 100% coverage would be excellent; automated builds and flawless documentation of all the possible features and scenarios would be great to have, and blazing fast regression coverage would be even better.

But a developer, or a team of developers, has built something that does solve people problems; how can this be so bad?
Would we be having this conversation if we could run a command like magic-test-all-my-code without any further input, with complete test coverage and checks? Probably not, in most cases.
Ease of setup, adoption, and use, I think, is fundamental to frame this kind of conversations; if I could write magic testing frameworks, I would. I cannot, sadly.

So, in short, the ork project is my attempt at creating the testing solution I would have liked to find when I first got into PHP and WordPress development.

The first PHP and WordPress code I wrote was a messy blob in the theme functions.php file; it took me some time to be able to run tests for it. Like most, the first code they write will not be "new" code but will rely on differing levels of legacy code.

And I've run into a lot of that in the WordPress universe.

What is ork then?

This post is not by chance titled "Building ork": I have not built it yet.
Not all of it, at least.

How would ork compare to wp-browser and Codeception?
That much is clear: ork is not a power tool.
Its objective is not to cover all the possible testing requirements but to allow someone that "just wants to start testing" to do that with as little knowledge required as possible.

It's my take on the idea of getting more (testing) bang for the buck.

And then I'm building it from scratch. In Node. Cross-compiling binaries for different operating systems. And it will support PHP and JS syntax to write tests.

What could go wrong?

Getting down to business

All this boring introduction to get to the first line of code.

The first decision I took, as anticipated above, is to ship ork as a cross-compiled executable for Windows, Linux, and Mac.

This decision comes from my experience in maintaining wp-browser and from the pain caused by keeping compatibility with a sizable matrix including different PHP, WordPress, Codeception, and Composer versions. I love the project, but I would like to avoid this particular pain.

After some research, I've settled on using Node and pkg to write once and compile something that should work on all operating systems.
I forgot what using something that is just one file feels like in years of package management with PHP and JS.

After some project scaffolding, here is the src/bin.ts file that will act as the entry point of my application; a very humble "Hello World" type of message:

console.log('Hello ork!');

I've decided to use Typescript to develop the project to leverage its compile-time type-checking.
As it's been the norm for my development projects for some time now, I've set up a Docker image to create a portable development system.

This is the Dockerfile of the node image I will be using over and over to run and compile the code:

## Use LTS version of node.
ARG NODE_VERSION="14.16.1"
FROM node:${NODE_VERSION}
## Make npm cache, the node modules and /usr/bin/lib accessible by all.
RUN mkdir /.npm \
    && mkdir -p /usr/local/lib/node_modules \
    && chown -R 501:0 /.npm \
    && chmod -R 0777 /usr/local/
## Change the user to my user to avoid file mode issues.
ARG UID=0
ARG GID=0
USER ${UID}:${GID}
## Install the required build dependencies.
ARG NODE_PKG_CACHE_PATH
RUN npm install -g typescript pkg dts-gen prettier
## pkg will require caching to not take forever.
ENV PKG_CACHE_PATH="${NODE_PKG_CACHE_PATH}"
## No default entrypoint or command, overwrite the default ones.
ENTRYPOINT [""]
CMD [""]
## Expose debugging port.
EXPOSE 9229

To streamline the build process as much as I can, I've set up a Makefile to be used with the make binary, with a collection of solutions I've accumulated over time:

## Use bash as shell.
SHELL := /bin/bash
## If you see pwd_unknown showing up, this is why. Re-calibrate your system.
PWD ?= pwd_unknown
## PROJECT_NAME defaults to name of the current directory.
PROJECT_NAME = $(notdir $(PWD))
## Suppress `make` own output.
.SILENT:
## Make `build` the default target to make sure it will display when make is called without a target.
.DEFAULT_GOAL := build test
## Create a script to support command line arguments for targets.
## The specified targets will be callable like this `make target_w_args_1 -- foo bar 23`.
## In the target, use the `$(TARGET_ARGS)` var to get the arguments.
## To get the nth argument, use `export TARGET_ARG_2="$(word 2,$(TARGET_ARGS))";`.
SUPPORTED_COMMANDS := node_machine node npm
SUPPORTS_MAKE_ARGS := $(findstring $(firstword $(MAKECMDGOALS)), $(SUPPORTED_COMMANDS))
ifneq "$(SUPPORTS_MAKE_ARGS)" ""
  TARGET_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
  $(eval $(TARGET_ARGS):;@:)
endif
## Set up some defaults.
NODE_VERSION = 14.16.1
## Any target commented with `## <description>` will show on `make help`.
help: ## Show this help message.
   @grep -E '^[a-zA-Z0-9\._-]+:.*?## .*$$' $(MAKEFILE_LIST) \
      | sort \
      | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.PHONY: help

build: ts pkg ## Builds the ork application.

pkg: ## Packages the application.
   docker run --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node pkg package.json

_docker/node/uuid: _docker/node/Dockerfile
   [ -d "$(PWD)/.cache/pkg" ] || mkdir -p "$(PWD)/.cache/pkg"
   docker build \
      --build-arg NODE_VERSION="$(NODE_VERSION)" \
      --build-arg UID="$${UID}" \
      --build-arg GID="$${GID}" \
      --build-arg NODE_PKG_CACHE_PATH="$(PWD)/.cache/pkg" \
      --tag ork-app/node:latest \
      --iidfile _docker/node/uuid \
      _docker/node

### Build the node image, if not built already.
ts: _docker/node/uuid
   docker run --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node tsc
.PHONY: ts

node_machine: _docker/node/uuid ## Runs a generic command in the node container.
   docker run --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node $(TARGET_ARGS)

node: _docker/node/uuid ## Runs a node command, e.g. `make node -- <command>`.
   docker run --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node node $(TARGET_ARGS)

npm: _docker/node/uuid ## Runs a npm command, e.g. `make npm -- <command>`.
   docker run --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node npm "$(TARGET_ARGS)"

Whit these files in place, building the cross-system application is as easy as running make build.

» make build
> pkg@5.1.0

Following the configuration in the package.json file, the executables for Windows, Linux, and macOS have been created in the dist directory.
Running the command on macOS and Linux, I will get the following output:

» ./dist/ork-macos
Hello ork

The same will happen in the Windows terminal:

C:\Users\lucatume\Desktop>.\ork-win.exe
Hello ork

This might not look like much, but it lays the foundation for the flow I would like to adopt to develop the application.

Adding tests

There is almost nothing in the current CLI application that will warrant testing.
That is precisely why adding tests at this stage will be so cheap and convenient.

The tests I'm setting up for the CLI application, let's call it "acceptance" testing or "end-to-end" testing for the sake of analogy of what I would do while developing a Web application, are tests that will run the compiled ork binary in a sub-process and check its output and side effects.

For that, I've chosen to use the command-line-test package and the mocha testing framework.

To keep with the portability and reproducibility theme, I've set up two additional make targets to execute the tests in the node container with, and without, remote Node debugger support:

test_debug: _docker/node/uuid ## Runs the tests in debug mode, port
   docker run -i --volume "$(PWD):$(PWD)" -w "$(PWD)" -p "9229:9229" ork-app/node node --inspect-brk=0.0.0.0 \
      node_modules/mocha/bin/mocha \ --timeout 0 --ui bdd
.PHONY: test_debug

test: _docker/node/uuid ## Runs the tests.
   docker run -i --volume "$(PWD):$(PWD)" -w "$(PWD)" ork-app/node node \
      node_modules/mocha/bin/mocha --timeout 0 --ui bdd
.PHONY: test

I wrote a first test just to confirm the setup works correctly:

const {orkBin} = require('./_support.js');
const CliTest = require('command-line-test');
const assert = require('assert');

describe('hello ork', function () {
    it('should display hello ork', async function () {
        const cliTest = new CliTest();
        const {error, stdout, stderr} = await cliTest.spawn(orkBin);

        assert(error === null, error);
        assert(stdout !== null);
        assert(stdout === 'Hello ork\n', stdout);
    });
});

In case you're wondering, the _support.js file will only provide me with a portable solution to find the compiled ork binary depending on the system the tests are running on:

const os = require('os');
const osName = require('os-name')(os.platform(), os.release()).substr(0, 3);
const locateBin = function () {
    let binPostfix;
    switch (osName) {
        case 'mac':
            binPostfix = 'macos';
            break;
        case 'win':
            binPostfix = 'win.exe';
            break;
        default:
            binPostfix = 'linux';
            break;
    }

    return __dirname + `/../dist/ork-${binPostfix}`;
};

module.exports = {
    orkBin: locateBin()
};

I'm closing this first post on the passing test output:

» make test


  hello ork
    ✓ should display hello ork (408ms)


  1 passing (414ms)

Next

In my next post, I will start working on the actual code laying out the pieces of my application one by one as I develop the application concept.

Making two great solutions work together.

In my last post, I've ˙praised the uncelebrated power of the WP-CLI shell command.

WP-CLI maintainer Alain Schlesser provided a further tip to make the WP-CLI shell even better through the schlessera/wp-cli-psysh package. I strongly suggest you try that out.

Another tool I use for my daily work is the Tinkerwell application.

Tintkerwell is a step up from an interactive PHP shell to an elementary code editor. Tinkerwell UI does not replace a proper PHP IDE, but it provides a UI with some neat features (vim mode, for one) with a clean and straightforward approach.

I've found this article and videos by Ross Wintle enjoyable and easy to follow and suggest you give it a read and look. I could set up Tinkerwell to work on my local WordPress site in little time, and you will be able to do the same if your WordPress application is served on localhost.

Should that not be the case, e.g., if you're using Local by Flywheel or some other "containerized" solution, then setting Tinkerwell up might require either its remote connection function to do some modification to the WordPress installation configuration file.

Setting up a Local site for Tinkerwell

Since the SSH-based approach seems overkill and too complicated for the simple needs of tinkering on a local site, I prefer the second approach and consistently update my Local by Flywheel websites wp-config.php file to correctly handle requests coming from the Tinkerwell application.

First of all, set the Tinkerwell working directory to the app/public directory of your Local by Flywheel WordPress site:

Second, modify the site wp-config.php file to detect an incoming request from the Tinkerwell application and connect to the WordPress database via the correct database host:

/** MySQL hostname */
if ( isset( $_SERVER['SCRIPT_NAME'] ) 
	&& false !== stripos( $_SERVER['SCRIPT_NAME'], 'tinkerwell' ) ) {
	// Host from the Local > Database > "Remote Host" field.
	$db_host = '192.168.95.100';
	// Port from the Local > Database > "Remote Port" field, omit if there's no port.
	$db_port = ':4026';
	define( 'DB_HOST', "{$db_host}{$db_port}" );
} else {
	define( 'DB_HOST', 'localhost' );
}

I use the previous version of Local, the Docker-based one, but the snippet will work with the new, MAMP-like version.

If everything is correct, you should be able to start tinkering with your WordPress site using Tinkerwell.

If you're using more elaborate solutions, e.g, a Docker stack, then just set the $db_host and $db_port variables to the database container IP address and port. Chances are, if you've got no idea what this means, then you're probably not using a container-based solution and the snippet and example above should be fine.

I do not see nearly enough mention, or use, of the wp shell command.

By now, anyone working on WordPress projects for a living should, at the very least, know what WP-CLI is. If you do not, then take a read here.

The short version is that WP-CLI is a tool that allows skipping the WordPress UI to perform mundane tasks or script complex ones. WP-CLI is, as the name suggests, a Command Line Interface that will not run outside of WordPress but in WordPress.

That means that when you type the wp command that, in most situations, is the aliased name of the tool binary, WP-CLI will load WordPress, and with it, its current theme, must-use, and other plugins. While this is not the case for some lower-level commands, it's accurate for most.

WordPress is a CMS (Content Management System); as such, its primary and intended use is the one to take an HTTP request as an input, say a GET request to the /2020-10-15/my-great-posts URI and go, roughly, through the following steps: load the WordPress core components like user management, session management, load the must-use plugins and active plugins load the theme, and plugins, identify the request as one for the site front-end (as is the case for this request example) identify the target area of the request (front-end as is the case for this example request, REST API, Admin UI et cetera) finally, route the request to the components that should handle it Produce output in response, the format depending on the nature of the request (e.g., HTML for a front-end request, JSON for a REST API request, and so on).

If only one could stop the loading process at step 4 from the list above, it would allow someone to tap into a fully up and running WordPress session, with all the core components, plugins, and theme loaded to "do things in the WordPress" installation.

That is precisely what the WP-CLI shell command allows. From my terminal application, I can type wp shell and be greeted by a modest, although powerful, prompt:

Sites/wp-site » wp shell
wp>

That prompt is a PHP interactive shell prompt; you could get something similar by typing php -a and being greeted by the PHP interactive shell prompt. Since this is a PHP interactive shell and not your terminal application, only PHP code and functions will make sense.

The difference, though, is that the PHP interactive shell will not load WordPress for you as the wp shell command does:

Sites/wp-site » php -a
Interactive shell

php > get_option('siteurl');

Warning: Uncaught Error: Call to undefined function get_option() in php shell code:1
Stack trace:
##0 {main}
  thrown in php shell code on line 1
php > exit
Sites/wp-site » wp shell
wp> get_option('siteurl');
=> string(19) "http://wordpress.test"
wp> exit
Sites/wp-site »

As is the case for PHP interactive shell, you can exit the WP-CLI interactive shell by issuing the exit command.

I use the wp shell command daily to debug the output of some functions deeply nested into the code and know their output or effects. Each PHP or WordPress function invoked will affect the underlying WordPress installation, requiring some care. Still, I prefer this method to test out ideas quickly or to avoid filling my code with echo, var_dump, print_r, die, and other debug methods that would entail long browser refresh trial and error approaches.

If you've never tried out WP-CLI, you definitely should, and if you've never used the wp shell command, I invite you to try it out now and get addicted.

Taking inspiration from this blog, I encourage you to make this reading worth it:

If you answer the questions below, you will create new neurons in your head through neurogenesis. But if you don't, you will forget about 80% of what you just read tomorrow.

1. At which step of the WordPress loading process does wp shell put you?
2. What is the difference between wp shell and your terminal emulator?
3. Why should you use care to use the wp shell?

Edit: some Twitter responses later, WP-CLI maintainer Alain Schlesser provided a further tip to make the WP-CLI shell even better with integration of the schlessera/wp-cli-psysh package. Just run wp package install schlessera/wp-cli-psysh and marvel at the much better output you get out of the wp shell command.

Previously, in this series

In the first post in this series, I've covered the basic stuff: what Docker and docker-compose are and how they could enable a portable environment for development, and more interesting to me, testing WordPress projects.
In the second post, I've detailed the behavior of the same docker-compose.yml stack on macOS, Windows, and Linux machines.
I've concluded the post in a situation where files created by the WordPress container, on a Linux host, would not belong to my user, but another user.

In this third article, I will work to mitigate the "portability" issues arising from a stack that will behave differently depending on the host operating system (macOS, Windows or Linux) to move closer to the ideal of a portable testing environment for WordPress projects.

The docker-compose.yml file in question, the one I'm starting this post from, is this one:

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

The difference between host users and container users in Linux

The main difference between how Docker works on macOS and Windows versus how works on Linux is, without delving too deep into technical aspects I'm not expert enough to discuss, there is an additional layer (allow me to call it a "virtual machine") between the macOS or Windows host and the containers that is not present on Linux.

Containers are a Linux technology ported over to macOS and Windows. The ports solve several compatibility problems across different operating systems and mitigate, leveraging the use of "virtual machine" of sorts, some of the key differences between the OSes. Containers ignore the underlying implementation; in a way, Docker (for Windows, for macOS and Linux) acts as Java: create an image, build, publish it, share it, and run wherever Docker runs.

The differences become something to deal with and put some care into when sharing files between the host machine and the container. With some exceptions, most images are based on Linux OSes, where file permissions are serious business and not a small part of why Linux is the de-facto standard when it comes to servers.

When containers run on Linux, file ownership of shared files is maintained.
What this means is that:

If a file belongs to the user luca on the host machine, then that file will belong to luca in the container.

• Vice-versa, if a file belongs to www-data in the container, it will belong to www-data on the host machine.

The second would explain why, in my previous post, I could not delete files created by the container from the host machine: my user on the host, luca is not the owner of the files created by the www-data user in the container.
When I tried to do that I would get the following output:

Repos/wp-docker » whoami
luca
Repos/wp-docker » ls -la _wordpress/wp-content/plugins/hello-dolly
total 16
drwxr-xr-x 2 www-data www-data 4096 Jun  6 16:58 .
drwxr-xr-x 4 www-data www-data 4096 Jun  6 16:58 ..
-rw-r--r-- 1 www-data www-data 2593 Jun  6 16:58 hello.php
-rw-r--r-- 1 www-data www-data  623 Jun  6 16:58 readme.txt
Repos/wp-docker » id -u
1002
Repos/wp-docker » id -u www-data
33

To delete the files from the host I am forced to use sudo:

Repos/wp-docker » sudo rm -rf _wordpress/wp-content/plugins/hello-dolly 
[sudo] password for luca: ***********

Using sudo is a dangerous habit and a requirement I would like to avoid in this instance.

Fixing the file ownership issue in the WordPress container

The official WordPress image published on Dockerhub supports running Apache as a different user.
Looking at the docker-entrypoint.sh file that will run when the container starts, I see I could set the APACHE_RUN_USER and APACHE_RUN_GROUP environment variables to have the Apache web-server process belong to a specific user.

What user? If on my host Linux machine my user is luca, and I want my user to be able to modify and delete the files shared with the container without requiring the use of sudo, then Apache must be started by luca.
I will specify the user by its user ID (uid) and group ID (gid).

I've shown it before, but, as a refresher, I can fetch the current user ID and group ID from the terminal:

Repos/wp-docker » id -u
1002
Repos/wp-docker » id -g
1002

Usually, on Linux, those same values will be available on the UID and GID environment variables, but I prefer to rely on commands I run to make sure I'm getting the values I need.

I modify the docker-compose.yml file to define the APACHE_RUN_USER and APACHE_RUN_GROUP environment variables:

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
      APACHE_RUN_USER: ${APACHE_RUN_USER}
      APACHE_RUN_GROUP: ${APACHE_RUN_GROUP}
    volumes:
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

The FOO: ${BAR} notation means "at runtime set the value of the FOO environment variable in the container to the current value of the BAR environment variable on the host". There is no requirement for the two variables to have the same name; it's just convenient.

I try to spin up the stack again, specifying the two environment variables:

APACHE_RUN_USER=$(id -u) \
APACHE_RUN_GROUP=$(id -g) \
docker-compose up

Recreating wp-docker_wordpress_1 ... done
Attaching to wp-docker_wordpress_1
wordpress_1  | AH00543: apache2: bad user name 1002
wp-docker_wordpress_1 exited with code 1

Ok, this makes sense. The user luca does not exist in the container; the Apache process cannot belong to a user that does not exist.

The docker-compose specification allows specifying a user, this is the equivalent of [using the -u|--user option when using the docker run command.

The documentation says:

When passing a numeric ID, the user does not have to exist in the container.

Which is what I need, I want to have a user somehow automagically created for me and I want to run Apache as that user.

I modify the docker-compose.yml file again:

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    ports:
      - 8080:80
    user: ${WORDPRESS_RUN_USER}:${WORDPRESS_RUN_GROUP}
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
      APACHE_RUN_USER: ${APACHE_RUN_USER}
      APACHE_RUN_GROUP: ${APACHE_RUN_GROUP}
    volumes:
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

I try again setting the two new variables as well:

APACHE_RUN_USER=$(id -u) \
APACHE_RUN_GROUP=$(id -g) \
WORDPRESS_RUN_USER=$(id -u) \
WORDPRESS_RUN_GROUP=$(id -g) \
docker-compose up

wp-docker_wordpress_1 is up-to-date
Creating wp-docker_db_1 ... done
Attaching to wp-docker_wordpress_1, wp-docker_db_1
db_1         | 2020-06-09 11:44:26+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.30-1debian10 started.
wordpress_1  | sed: couldn't open temporary file ./sedEJZmHq: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedLW2kUW: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedbvCEiP: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedEf5PRI: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sed8AOfWS: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedo8VJU5: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedpTR0UV: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sed0jXAY3: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedxBnvZX: Permission denied
wordpress_1  | sed: couldn't open temporary file ./sedgVhJmU: Permission denied

Not what I had expected.

When I run the docker-compose up command, I'm just starting a series of connected containers and running commands in each.

What does "running a command" means?

Containers are a "technology" to run commands in an isolated, self-contained environment.
As an example, the Nginx container will run the nginx -g daemon off command, and the WordPress container I'm using in the current stack will run the apache2-foreground command.
The commands, as is the case with the two I've just listed, might be commands to start a server, but any container image (from which containers are built) is essentially a way to prepare for, and run, a single command. The fact that a command might do something and return in a second or run for weeks does not change how containers work.

It's essential to understand this to appreciate why the user running the command is so important: if the user running the command, or any of the commands required to set up the environment for that command to run, is not authorized, then the container will exit.

By default, Docker containers will run using the root user (id=0).
As is the custom in Linux, the root user is the one who can do everything.

The change, in the docker-compose.yml file, that is causing the issue is the one where I specify a user other than the root one; that user is not authorized to write to /tmp.
In this case, writing to the /tmp directory is required while setting up the WordPress container.

I'm in a bit of a conundrum here: I want to run the final container command, apache2-foreground, to start the Apache web-server as my user (luca) but I cannot run the container as that user (what I just did), and I cannot specify, as APACHE_RUN_USER, a user that does not exist in the container.

Modifying the WordPress container command

What I got from the previous section, is that each container is a way to run a command.
To run the command, in the WordPress container, I can modify its command.

Note: specifying the command parameter in a service description in the docker-compose.yml file is the equivalent of specifying a CMD directive in the image Dockerfile.

On Linux, the user name, e.g. luca is just a local (to the machine) name, and what really has value is the user ID and the ID of the user group.
To make a parallel with WordPress, the user ID is the post ID, and the user name is the post_title: a post will keep being uniquely identifiable as long as the post ID stays the same, the title can change over time.
Knowing this, and keeping the objective of portability in mind, I call my user, the user that has the same user and group ID in the container that I have on my Linux machine (id=1002 gid=1002), docker. The user naming choice follows a common standard.

I modify the docker-compose.yml file section related to the wordpress container to:

1. Remove the user parameter and let the container run using root.
2. Create a docker user with the same uid and gid as my luca user on my Linux machine.

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    ports:
      - "8080:80"
    # The default entypoint is `/usr/local/bin/docker-entrypoint.sh`. Not overridden here.
    # The default command is `apache2-foreground`, to start the Apache web-server.
    # The command is overridden to create the `docker:docker` user and group mapped to the host machine
    # user and group if they do not exist already.
    # After creating the user and group, run the original command.
    # The `$$` is to escape the `$` in YAML and avoid docker-compose trying to replace it.
    command: 
        - /bin/sh
        - -c
        - |
            test $$(getent group docker) || addgroup --gid ${APACHE_RUN_GROUP} docker
            test $$(id -u docker) || adduser --uid ${APACHE_RUN_USER} --ingroup docker \
            --home /home/docker --disabled-password --gecos '' docker
            /usr/local/bin/docker-entrypoint.sh apache2-foreground
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
      # Start the Apache server as the `docker` user.
      # I've created the user in the `command` section, and this will work now.
      APACHE_RUN_USER: "docker"
      APACHE_RUN_GROUP: "docker"
    volumes:
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

When I run the docker-compose up command again, everything will work.

APACHE_RUN_USER=$(id -u) \
APACHE_RUN_GROUP=$(id -g) \
docker-compose up

The WordPress installation files belong to my user:

Repos/wp-docker » ls -la _wordpress
total 224
drwxr-xr-x  5 luca luca  4096 Jun 10 15:39 .
drwxr-xr-x  3 luca luca  4096 Jun 10 15:39 ..
-rw-r--r--  1 luca luca   234 Jun 10 15:39 .htaccess
-rw-r--r--  1 luca luca   420 Dec  1  2017 index.php
-rw-r--r--  1 luca luca 19935 Jan  1  2019 license.txt
-rw-r--r--  1 luca luca  7368 Sep  2  2019 readme.html
-rw-r--r--  1 luca luca  6939 Sep  3  2019 wp-activate.php
drwxr-xr-x  9 luca luca  4096 Dec 18 23:16 wp-admin
-rw-r--r--  1 luca luca   369 Dec  1  2017 wp-blog-header.php
-rw-r--r--  1 luca luca  2283 Jan 21  2019 wp-comments-post.php
-rw-r--r--  1 luca luca  3183 Jun 10 15:39 wp-config.php
-rw-r--r--  1 luca luca  2808 Jun 10 15:39 wp-config-sample.php
drwxr-xr-x  4 luca luca  4096 Jun 10 15:39 wp-content
-rw-r--r--  1 luca luca  3955 Oct 11  2019 wp-cron.php
drwxr-xr-x 20 luca luca 12288 Dec 18 23:16 wp-includes
-rw-r--r--  1 luca luca  2504 Sep  3  2019 wp-links-opml.php
-rw-r--r--  1 luca luca  3326 Sep  3  2019 wp-load.php
-rw-r--r--  1 luca luca 47597 Dec  9  2019 wp-login.php
-rw-r--r--  1 luca luca  8483 Sep  3  2019 wp-mail.php
-rw-r--r--  1 luca luca 19120 Oct 15  2019 wp-settings.php
-rw-r--r--  1 luca luca 31112 Sep  3  2019 wp-signup.php
-rw-r--r--  1 luca luca  4764 Dec  1  2017 wp-trackback.php
-rw-r--r--  1 luca luca  3150 Jul  1  2019 xmlrpc.php

I can now delete the files from my host machine without using sudo:

Repos/wp-docker » rm -rf _wordpress/wp-content/plugins/hello.php 
Repos/wp-docker » ls -la _wordpress/wp-content/plugins 
total 16
drwxr-xr-x 3 luca luca 4096 Jun 10 15:56 .
drwxr-xr-x 4 luca luca 4096 Jun 10 15:39 ..
drwxr-xr-x 4 luca luca 4096 Dec 18 23:16 akismet
-rw-r--r-- 1 luca luca   28 Jun  5  2014 index.php

And, after I've installed WordPress using its UI at http://localhost:8080, if I install a plugin from the WordPress installation, tat plugin files too will be accessible to my user.

The docker-compose.yml file, in its wordpress service section especially, could be reduced by building a dedicated image customized to my needs. Currently, I prefer the compact form of the single docker-compose.yml file and will leave that for later, if required.

Thinning the command to start the stack

It's a bit cumbersome to have to start the stack using this command:

APACHE_RUN_USER=$(id -u) \
APACHE_RUN_GROUP=$(id -g) \
docker-compose up

It's not impossible to remember, but it's error-prone, and, as the stack evolves, it will become more and more complex.

To get back the ease-of-use I've created a PHP script that will take care of running the command correctly every time with awareness of the current operating system:

##! /usr/bin/env php
<?php
// Default to `up` if no arguments are provided.
// $argv[0] is the script name itself, drop it.
$unescapedArgs = isset($argv[1]) ? (array)array_slice($argv,1) : ['up'];
// Escape the command arguments before using them.
$escapedArgs = array_map('escapeshellarg', $unescapedArgs);

// Get the current OS; `dar` is macOS, `lin` is Linux, `win` is Windows
// Other OSes (*nix, BSD...) require, probably, the same setup steps as for Linux. 
$os = substr(strtolower(PHP_OS), 0, 3);

$binary = 'docker-compose';

switch ($os){
        case 'win':
            $binary = 'docker-compose.exe';
            break;
        case 'dar':
            // There is nothing to do here.
            break;
        default:
            // If we cannot get the current user and group ID use `0` (`root` in the container).
            putenv('DOCKER_RUN_USER=' . (int)getmyuid());
            putenv('DOCKER_RUN_GROUP=' . (int)getmygid());
            break;
}

// Build the command line.
$commandLine = sprintf('%s %s', $binary, implode(' ', $escapedArgs));

// Debug the command line.
echo "\nCommand: {$commandLine}\n\n";

// Let's not apply the time limit to the processes launched by the script.
set_time_limit(0);

// Execute the command.
passthru($commandLine, $status);

// Print a blank line after the command output to clear the terminal a bit.
echo "\n";

// Exit the same status as the command.
exit($status);

I've called the script stack, so I can now run the equivalent of the previous command by running:

./stack up

I've modified the stack to work on Linux, will it be the same on Windows And macOS?

In my next post, I will test the current solution out on both to move a step closer to a portable WordPress testing environment and iterate on the CLI wrapper script.

Previously, in this series

In the first post in this series, I've covered the basic stuff: what Docker and docker-compose are and how they could enable a portable environment for development, and more interesting to me, testing WordPress projects.
In this second article, I will highlight a less covered issue that might seriously impede, or stop altogether, the adoption and use of Docker as a testing environment for WordPress projects.

I've concluded the previous article with this docker-compose.yml file:

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      # Not using the volume defined in the volumes section anymore, so I've removed it.
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

The file allows me to run a fully working WordPress installation run locally on my macOS machine. The next step, for this stack to be defined "portable", is to try and spin it up on Windows and Linux machines.

Running the stack on Windows

On Windows, docker and docker-compose can be installed using the Docker for Windows application; there is nothing else to do after installing it but start the Docker for Desktop application.
I can check the version of both commands in a vanilla Windows terminal emulator (Command Prompt):

The structure and content of the directory I'm using are similar to the use I used on macOS:

• The project lives in the C:\Users\lucatume\Repos\wp-docker directory.
• The directory only contains, to start, the docker-compose.yml file.

I run the docker-compose up command and wait for images to download and the containers based on the instances to start.
Following the initial set up, the command prompt will tail the logs of the the db and wordpress services, as it happened on macOS, docker-compose created a _wordpress directory where WordPress files are stored, and made WordPress available, locally, at http://localhost:8080:

After completing the WordPress installation using the UI, I want to make sure file modification will work both ways:

• Adding, modifying, or deleting a file on the Windows host will alter the content of the WordPress installation served at http://localhost:8080.
• Adding, modifying, or deleting a file using the installation UI, say adding a plugin, will reflect on the contents of the _wordpress directory.

First I runt this command from the C:\Users\lucatume\Repos\wp-docker directory:

del /f .\wp-content\plugins\hello.php

As expected the "Hello Dolly" plugin does not appear in the list of available plugins in the WordPress installation:

I, now, try to re-add the plugin using the WordPres installation plugin administration UI:

While I could be more thorough, I know this means that read/write is working on both sides.

Time to move to Linux.

Running the stack on Linux

Depending on the Linux distribution you're using, the Docker and docker-compose installation steps might differ.
In this example, I'm using a Ubuntu host machine; you can find the installation instructions here.

Whatever the distribution, I've followed the post-installation steps to make sure I can run the docker command without requiring sudo. It's not needed, but it's the reason I'm not using sudo in the examples, and it makes the use of docker and docker-compose easier.

I start by checking the version of the two commands:

I've reproduced the same starting directory structure as the macOS example:

• The project lives in the /home/luca/Repos/wp-docker directory.
• The directory only contains, to start, the docker-compose.yml file.

As I did on macOS and Windows before, I spin up the stack with docker-compose up and wait for the WordPress installation to be ready:

As has been the case on macOS and Windows, docker-compose created and filled the wp-docker/_wordpress directory with the WordPress installation contents:

Repos/wp-docker » tree -L 2
.
├── docker-compose.yml
└── _wordpress
    ├── index.php
    ├── license.txt
    ├── readme.html
    ├── wp-activate.php
    ├── wp-admin
    ├── wp-blog-header.php
    ├── wp-comments-post.php
    ├── wp-config.php
    ├── wp-config-sample.php
    ├── wp-content
    ├── wp-cron.php
    ├── wp-includes
    ├── wp-links-opml.php
    ├── wp-load.php
    ├── wp-login.php
    ├── wp-mail.php
    ├── wp-settings.php
    ├── wp-signup.php
    ├── wp-trackback.php
    └── xmlrpc.php

4 directories, 18 files

Again, as I did on macOS and Windows, I check if the read/write operations work correctly between the host and the running WordPress container.

As a first step, I try to delete the "Hello Dolly" plugin, a single file, from the host to see if this has the expected effect on the running WordPress installation.
The expected effect is to not see the "Hello Dolly" plugin among the available plugins.

Repos/wp-docker » rm -rf _wordpress/wp-content/plugins/hello-dolly 
rm: cannot remove '_wordpress/wp-content/plugins/hello-dolly/hello.php': Permission denied
rm: cannot remove '_wordpress/wp-content/plugins/hello-dolly/readme.txt': Permission denied
Repos/wp-docker » 

Differently from what happened on macOS and Windows, I cannot delete a file created by the container from the host.
I check the file modes to understand why:

Repos/wp-docker » whoami
luca
Repos/wp-docker » ls -la _wordpress/wp-content/plugins/hello-dolly
total 16
drwxr-xr-x 2 www-data www-data 4096 Jun  6 16:58 .
drwxr-xr-x 4 www-data www-data 4096 Jun  6 16:58 ..
-rw-r--r-- 1 www-data www-data 2593 Jun  6 16:58 hello.php
-rw-r--r-- 1 www-data www-data  623 Jun  6 16:58 readme.txt
Repos/wp-docker » id -u
1002
Repos/wp-docker » id -u www-data
33

The id -u [<user>] will return the user ID of either the current user, luca in my example, or the ID of the specified user, www-data in the second example.

What the output above means is:

• the _wordpress/wp-content/plugins/hello directory is owned by the www-data user, with user ID 33
• my user is luca, with user ID 1002
• due to how user permission and file ownership work on Linux, a user that has no super-user (su) rights cannot modify or delete other user files.

Ok, but where does the www-data user come from?! In short: Apache web-server.
The official wordpress image available on Dockerhub is based on a Linux machine that serves the WordPress installation using Apache; the Apache user has ID 33.

So, in Linux, the owner of the files created by a running container is the user the container is currently using.
I execute a command on the running wordpress container to make sure of this:

Repos/wp-docker » docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
a19f327780d2        wordpress           "docker-entrypoint.s…"   20 minutes ago      Up 20 minutes       0.0.0.0:8080->80/tcp   wp-docker_wordpress_1
d21785e9fcca        mysql:5.7           "docker-entrypoint.s…"   20 minutes ago      Up 20 minutes       3306/tcp, 33060/tcp    wp-docker_db_1
Repos/wp-docker » docker exec a19f327780d2 whoami
root

The user of the wordpress container is not www-data, as expected, but root.
And not my root, but the root user of the Linux installation. So: where does that www-data user Apache is running for, come from?

Finding where the www-data user comes from takes some searching, but looking at the official WordPress image files I can see the Apache user will be set to the www-data one if the APACHE_RUN_USER and APACHE_RUN_GROUP environment variables are not specified.

From the Ubuntu host I can still modify the files using sudo:

Repos/wp-docker » sudo rm -rf _wordpress/wp-content/plugins/hello-dolly 
[sudo] password for luca: 
Repos/wp-docker » 

But having to use sudo to delete files or any other modifications to the files belonging to the WordPress installation is not a viable way of working.

In my next post, I will focus on the Linux version of the stack to solve this problem.

The Docker promise, in a WordPress developer context

Docker is almost everywhere now, and there are many good reasons for it.
I'm not going into all the reasons that make Docker amazing. Instead, I would focus on the possibilities it offers me, as a WordPress developer sharing production and test code across different teams and operating systems.
"It works on my machine" might be a meme, but it's still frequent feedback I run into when reviewing pull requests with broken tests that are, invariably, passing locally and failing in CI. Supposedly.
It would be great for a team, or even just for a developer using multiple operating systems, to have identical testing environments running on each developer machine, whatever the operating system.

The purpose of this series is to consolidate the knowledge I got along the way of learning "how to docker" into a progressive series of articles moving past base, copy-and-paste examples. Ideally, to set up to an OS-independent, reusable, flexible docker-compose based, WordPress local, and CI (Continuous Integration) testing and development environment.

Up and done

I'm using, to start, a container I use daily: the official WordPress container from Dockerhub.

This container is a constant in my builds and testing environments. I've spent quite a bit of time learning it's ins-and-outs and collecting several "gotchas along the path.

I've created a minimal docker-compose.yml configuration file copying the example from the dockerhub official WordPress image documentation:

cd ~/Repos
mkdir wp-docker
cd wp-docker
touch docker-compose.yml

I've just moved around some lines but changed nothing.

version: '3.1'

volumes:

  wordpress:
  db:

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      # Expose port 80 of the container on port 8080 of localhost.
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress:/var/www/html

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

I run the command to start the stack and wait for the setup to be ready:

docker-compose up

If the images are not already present on my machine, they will be downloaded and extracted first, and this will happen only once the first time I require new images.

After a bit of setup and build processes, I will be left with a terminal tab or window following (tailing) the two containers log files:

db_1         | 2020-05-16T11:24:01.540475Z 0 [Note] Event Scheduler: Loaded 0 events
db_1         | 2020-05-16T11:24:01.540824Z 0 [Note] mysqld: ready for connections.
db_1         | Version: '5.7.29'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.80.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | [Sat May 16 11:24:03.714651 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.16 configured -- resuming normal operations
wordpress_1  | [Sat May 16 11:24:03.714774 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

In the docker-compose.yml file I've specified, in services.wordpress.ports section, that I would like to expose port 80 of the container on port 8080 of localhost.

Up and down and up again

When I open the http://localhost:8080 address in my browser, though, WordPress will not be installed.

I go through the installation and will end up with a working WordPress installation:

If I stop the WordPress and database server closing the terminal tab running the logs or sending it the termination command (Ctrl+c), then the site will not be available anymore at http://localhost:8080, as expected.

How would I restart the WordPress installation to work on it, though? Using the same command I've used before:

docker-compose up

Do I need to install WordPress again? No.
In the docker-compose.yml file I've defined two volumes: wordpress and db.
The wordpress volume will store, in my host machine filesystem, the WordPress container files, the content of the WordPress installation.
The db volume will store the data produced by the database instance used by WordPress, again in my host machine filestystem.

The concept of volumes and "Docker volumes" was tricky for me to grasp until I've understood that it just means "store what files you create or update during your work on my machine".

So where are the files?

If they are on my machine, then I would expect the files to be somewhere I can see them; in the same directory containing the docker-compose.yml file, ideally.

Yet they are not there, the directory contains only, even while the wordpress and db containers are running, the docker-compose.yml file:

Repos/wp-docker » tree -L 1 .
.
└── docker-compose.yml

0 directories, 1 file

Getting the path to the real location where, in the host filesystem, the files live requires a bit of inspection into how Docker stores container information.

First, I list the currently defined WordPress containers with docker volume ls:

Repos/wp-docker » docker volume ls
DRIVER              VOLUME NAME
local               wp-docker_db
local               wp-docker_wordpress

As expected, Docker created two volumes have named after the directory where the docker-compose.yml file lives:

1. wp-docker_wordpress for the wordpress volume
2. wp-docker_db for the db volume

Since I want to see the WordPress container files, the container I'm interested in is the wp-docker_wordpress one.
The command docker volume inspect wp-docker_wordpress will provide me with useful information about the volume:

Repos/wp-docker » docker volume inspect wp-docker_wordpress
[
    {
        "CreatedAt": "2020-05-16T11:46:15Z",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "wp-docker",
            "com.docker.compose.version": "1.25.5",
            "com.docker.compose.volume": "wordpress"
        },
        "Mountpoint": "/var/lib/docker/volumes/wp-docker_wordpress/_data",
        "Name": "wp-docker_wordpress",
        "Options": null,
        "Scope": "local"
    }
]

So, are the wordpress volume files, used by the wordpress container, from the wp-docker project, in the /var/lib/docker/volumes/wp-docker_wordpress/_data directory?
Turns out no, they are not:

Repos/wp-docker » ls -la /var/lib/docker/volumes/wp-docker_wordpress/_data
ls: /var/lib/docker/volumes/wp-docker_wordpress/_data: No such file or directory

I could go into the rabbit hole of trying and finding them, but Docker and docker-compose provide a structured and documented way to store the container data exactly where I need it.

Binding volumes

Binding volumes is container jargon to say:

When I put stuff in this directory on my machine, it should appear in this directory in the container. Possibly the other way too.

I do want the WordPress installation files to exist, on my host machine, somewhere practical.
Specifically in the ~/Repos/wp-docker/_wordpress directory.

I update the docker-compose.yml file to reflect that.

version: '3.1'

volumes:

  db:

services:

  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      # Not using the volume defined in the volumes section anymore, so I've removed it.
      # The `.` means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
      - ./_wordpress:/var/www/html

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql

Two things to notice:

I've changed the services.wordpress.volumes entry to ./_wordpress:/var/www/html. The . in the ./_wordpress first fragment of the wordpress service volume definition means "from the directory that contains this file", where "this file" is the docker-compose.yml file.
In docker-compose.yml files, as in the -v parameter of the docker CLI API, volume bindings are read on_the_host:on_the_container. The same applies to port bindings. The first, the host part of a volume binding can either be a path relative to the docker-compose.yml directory, or an absolute path. I'm using the first option here to make the implementation portable.

Since I will not be using the wordpress volume anymore, I've removed it from the file.

If all goes well, firing up the stack again should create a _wordpress directory beside the docker-compose.yml file and put WordPress root directory in there.

I see WordPress

I fire up the stack again and wait, looking at the logs, for the message from the WordPress container to confirm it's ready.

Repos/wp-docker » docker-compose up -d

I open a new terminal tab and check what's in the project directory:

Repos/wp-docker » tree -L 1 .
.
└── docker-compose.yml

0 directories, 1 file

Nothing happened. Why?

Well: Docker is an efficient worker and, since I ran the stack before, will "recycle", reuse, the containers I was running the last time.
And the wordpress container I used before was never told to bind the /var/www/html directory to the _wordpress one in the project root directory. So it didn't.

How do I take control of this back? By tearing down the stack and firing it up again with the down (short for "tear down") command:

Repos/wp-docker » docker-compose down
Stopping wp-docker_wordpress_1 ... done
Stopping wp-docker_db_1        ... done
Removing wp-docker_wordpress_1 ... done
Removing wp-docker_db_1        ... done
Removing network wp-docker_default

Repos/wp-docker » docker-compose up
Creating network "wp-docker_default" with the default driver
Creating wp-docker_wordpress_1 ... done
Creating wp-docker_db_1        ... done
Attaching to wp-docker_db_1, wp-docker_wordpress_1
db_1         | 2020-05-17 11:16:48+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.29-1debian9 started.
db_1         | 2020-05-17 11:16:49+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1         | 2020-05-17 11:16:49+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.29-1debian9 started.
db_1         | 2020-05-17T11:16:49.337125Z 0 [Warning

...

db_1         | Version: '5.7.29'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
wordpress_1  | WordPress not found in /var/www/html - copying now...
wordpress_1  | Complete! WordPress has been successfully copied to /var/www/html
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
wordpress_1  | [Sun May 17 11:16:59.711493 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.16 configured -- resuming normal operations
wordpress_1  | [Sun May 17 11:16:59.711569 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

Can I see WordPress files in the _wordpress directory now?!

Repos/wp-docker » tree -L 2 .
.
├── _wordpress
│   ├── index.php
│   ├── license.txt
│   ├── readme.html
│   ├── wp-activate.php
│   ├── wp-admin
│   ├── wp-blog-header.php
│   ├── wp-comments-post.php
│   ├── wp-config-sample.php
│   ├── wp-config.php
│   ├── wp-content
│   ├── wp-cron.php
│   ├── wp-includes
│   ├── wp-links-opml.php
│   ├── wp-load.php
│   ├── wp-login.php
│   ├── wp-mail.php
│   ├── wp-settings.php
│   ├── wp-signup.php
│   ├── wp-trackback.php
│   └── xmlrpc.php
└── docker-compose.yml

4 directories, 18 files

I got myself a WordPress copy, ready to run, and in my host machine file system.

Trusting no one, I want to see if doing something on the host would modify the files on the container and vice-versa.

Two-way binding test

The first test will be to delete the "Hello Dolly" plugin from the host machine and see if the "Hello Dolly" plugin does disappear in the container:

rm ./_wordpress/wp-content/plugins/hello.php

A check-in the WordPress installation, at http://localhost:8080/wp-admin/plugins.php confirms my expectation:

To test that changing something in the container would, in turn, change it on the guest I will try to install the Hello Dolly plugin from the WordPress plugin repository again; if all goes as intended, then it should re-appear on the host.

Repos/wp-docker » tree -L 1 ./_wordpress/wp-content/plugins
./_wordpress/wp-content/plugins
├── akismet
├── hello-dolly
└── index.php

2 directories, 1 file

And it does.

This confirms volume binding works as intended.

On macOS. Will it be the same on Linux? And Windows?
Spoiler: no, it requires some leg work and understanding of file modes that I will plunge into in my next post.

This post is the third in a series; find the first one here and the second one here.

In this series, I'm documenting the steps to add parallel, container-based, builds to the wp-browser project.

To set expectations: as a result of this research and effort, I might add a container-dedicated command to wp-browser in the future, but this is not my primary goal now.
My main goal is being able to run wp-browser own CI builds (on Travis CI, at the moment) faster and more reliably.

Each step, reasoning, and assumption in these posts is part of my discovery process; as such, these might be confused, out of order, or outright wrong. Ye be warned.

You can find the final post shown in this post here.

Running make tasks in parallel

After the round-robin information and data collection I've done in the second post, before I spend any time developing further, I want to make sure the docker-compose based approach to parallel execution works.

I'm using make to run the scripts, and I do not know, at this stage, if this is the best way, but it's what I can immediately work with now.
The make utility comes with parallel execution support out of the box, I've used it a couple of times, but never with intent.

I set up a small test script and a new target in the project Makefile to test this out:

## Define the list of scripts to run
SCRIPTS = foo baz bar

## Generate the targets from the list of scripts.
targets = $(addprefix php_script_, $(SCRIPTS))

## Generate the targets.
$(targets): php_script_%:
   @php -r '$$i=0; while( $$i<5 ){ echo "PHP Script $@, run " . $$i++ . PHP_EOL; sleep( 5 / random_int( 1,10 ) ); }'

## Define a target in charge of running the generated targets.
pll: $(targets)

There's a bit of make syntax and constructs here, but the general catch is that I'm generating a list of "targets" ("tasks to run" in make terminology) from a list and running them with the pll (short for "parallel" command).
Each PHP script runs 5 times and sleeps a random number of tenths of a second before resuming.
Please note the double $$ symbol: without going into too much detail into how make works, any single $ symbol would be interpreted, by make as a make variable; since PHP variable sign of choice is the same (the $ symbol) I've doubly escaped it. Each target is executing a bash command like this:

<?php
$i = 0; 
while( $i<5 ){ 
    echo 'PHP Script $@, run ' . $i++ . PHP_EOL; 
    sleep( 5 / random_int( 1,10 ) ); 
}

I execute the three recipes in parallel using:

make -j pll

To get the typical parallel processing output where the output from each command is mixed with all the others.

[](https://theaveragedev.com/wp-content/uploads/2019/08/make-parallel-1.png)

To order the output a bit, I can use the -O flag (GNU version of make only, I use that on my Mac machine):

make -j -O pll

The target order might, in this case, be random, but the output is grouped by the target, depending on their completion order.

[](https://theaveragedev.com/wp-content/uploads/2019/08/make-parallel-2.png)

Failing some parallels task

The next step is making sure the parallel execution does not break when one or more, of the tasks, fail.
With the objective of speeding up wp-browser builds by running each suite in parallel, really running any Codeception command in parallel, failing builds are a reality I have to cope with and react to.

I've added a second group of test targets running this randomly failing PHP script:

<?php
$i = 0; 
while( $i<5 ){ 
    echo 'PHP Script $@, run ' . $i++ . PHP_EOL;
    $random_int = random_int(1,10);
    if($random_int  < 3){
        // Fail if the value is less than 3.
        exit( 1 );
    }
    sleep( 5 / $random_int ); 
}
echo 'PHP Script $@ completed.' . PHP_EOL;

Running the scripts with, and without, the -O option yields the following results:

[](https://theaveragedev.com/wp-content/uploads/2019/08/make-parallel-3.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/make-parallel-4.png)

In the last screenshot not only I'm running the parallel "builds" using the -O flag, but I'm also printing the exit status code of the whole make run.
The exit code of 2 indicates an error in the Makefile; since all the build systems I know of take any non-zero value as an error, this is fine as an indication the build failed.

The problems of parallelism

As anything in development, parallelism comes with its own set of problems.
The first and most obvious one is that the order of termination is not guaranteed, while the second, and less obvious one, is that parallel processes accessing non-parallel resources generating race conditions and possible inconsistencies.
I can better explain this with an example.

I've added another target to the Makefile:

RUNS = 1 2 3
docker_runs = $(addprefix run_, $(RUNS))

$(docker_runs): run_%:
   @docker-compose run --rm wpbrowser run test_suite -f -q --no-rebuild

pll_docker_builds: $(docker_runs)

The target is running the same test suite, the test_suite one I've created to test this behavior, three times.
The test_suite suite contains only the following test case:

<?php

class FileTest extends \Codeception\Test\Unit
{
    protected static function getFiles()
    {
        $files = [ 'one', 'two', 'three', 'four', 'five', 'six', 'seven' ];

        return $files;
    }

    protected static function cleanFiles()
    {
        foreach (static::getFiles() as $file) {
            $filePath = codecept_output_dir($file);
            if (file_exists($filePath)) {
                unlink($filePath);
            }
        }
    }

    public function setUp()
    {
        static::cleanFiles();
    }

    public function tearDown()
    {
        static::cleanFiles();
    }

    public function filesDataSet()
    {
        foreach (static::getFiles() as $file) {
            yield $file => [ $file ];
        }
    }

    /**
     * @dataProvider filesDataSet
     */
    public function test_files_can_be_created_and_removed($file)
    {
        // Make sure no file initially exists.
        $filePath = codecept_output_dir($file);

        $this->assertFileNotExists($filePath);

        touch($filePath);

        $this->assertFileExists($filePath);
    }
}

The test case only has one test method: that test method iterates on seven file names, make sure each file does not exist already, create each file in the Codeception output directory, and make sure that file exists.

All tests pass when running one after the other:

[](https://theaveragedev.com/wp-content/uploads/2019/08/parallelism-issues-1.png)

But fail when running in parallel, seemingly at random:

[](https://theaveragedev.com/wp-content/uploads/2019/08/parallelism-issues-2.png)

In its simplicity, it showcases the issue of parallelism: shared resources, the host machine shared filesystem in this specific case.

In the docker-compose.yml file I'm using I'm binding the host machine local filesystem, specifically the wp-browser folder, in the /project folder of each container:

version: '3.2'

services:

  wpbrowser:
    # Instead of using a pre-built image, let's build it from the file.
    build:
      context: ./docker/wpbrowser
      dockerfile: Dockerfile
      args:
        # By default use PHP 7.3 but allow overriding the version.
        - BUILD_PHP_VERSION=${BUILD_PHP_VERSION:-7.3}
        - BUILD_PHP_TIMEZONE=${BUILD_PHP_TIMEZONE:-UTC}
        - BUILD_PHP_IDEKEY=${BUILD_PHP_IDEKEY:-PHPSTORM}
        - BUILD_XDEBUG_ENABLE=${BUILD_XDEBUG_ENABLE:-1}
    volumes:
      # Bind the project files into the /project directory.
      - ./:/project
    environment:
      # As XDebug remote host use the host hostname on Docker for Mac or Windows, but allow overriding it.
      # Build systems are usually Linux-based.
      # Use default port, 9000, by default.
      XDEBUG_CONFIG: "remote_host=${XDEBUG_REMOTE_HOST:-host.docker.internal} remote_port=${XDEBUG_REMOTE_PORT:-9000}"
      # I use PHPStorm as IDE and this makes the handshake easier.
      PHP_IDE_CONFIG: "serverName=wp.test"

That binding means each container has its own, independent, filesystem, but each container synchronizes, in real-time, its filesystem modifications to the host machine filesystem; this affects, implicitly, all other containers synchronizing their filesystem with the host machine filesystem. In even shorter terms: all containers are reading, and writing, from the same source.

Why would this be a problem for the wp-browser project? Or for any project using wp-browser to run WordPress tests?

Because wp-browser has extensions and modules using the filesystem.
The reason being that, at times, the only way to affect WordPress behavior is to scaffold themes or plugins (or must-use plugins) "on the fly" and remove them after the tests.
This need to modify the installation files could be involved in something simpler, like testing media attachments.

Long story short: the containers should not share their files; ideally each should work on its copy of them.

Can this filesystem isolation be achieved?

Resolving the container filesystem issues

The docker-compose file syntax allows limiting what containers can do with bound host directories by specifying that a bound directory is read-only.
While I do not want the containers affecting the host, I do need the containers to be able to write on their files, so the read-only approach is not the correct one.

The simple solution is modifying the test container Dockerfile to copy over the project files and rebuilding the container before each parallel run.

I've modified the docker-compose.yml file to remove the volume entry and update the wpbrowser.build.context to the project root folder:

version: '3.2'

services:

  wpbrowser:
    # Instead of using a pre-built image, let's build it from the file.
    build:
      # The context is the root folder any relative host machine path wil refer to in the Dockerfile.
      context: .
      dockerfile: docker/wpbrowser/Dockerfile
      args:
        # By default use PHP 7.3 but allow overriding the version.
        - BUILD_PHP_VERSION=${BUILD_PHP_VERSION:-7.3}
        - BUILD_PHP_TIMEZONE=${BUILD_PHP_TIMEZONE:-UTC}
        - BUILD_PHP_IDEKEY=${BUILD_PHP_IDEKEY:-PHPSTORM}
        - BUILD_XDEBUG_ENABLE=${BUILD_XDEBUG_ENABLE:-1}
    environment:
      # As XDebug remote host use the host hostname on Docker for Mac or Windows, but allow overriding it.
      # Build systems are usually Linux-based.
      # Use default port, 9000, by default.
      XDEBUG_CONFIG: "remote_host=${XDEBUG_REMOTE_HOST:-host.docker.internal} remote_port=${XDEBUG_REMOTE_PORT:-9000}"
      # I use PHPStorm as IDE and this makes the handshake easier.
      PHP_IDE_CONFIG: "serverName=wp.test"

I've also modified the docker/wpbrowser/Dockerfile to COPY over the project files during build:

## Take the BUILD_PHP_VERSION argument into account, default to 7.3.
ARG BUILD_PHP_VERSION=7.3

## Allow for the customization of other PHP parameters.
ARG BUILD_PHP_TIMEZONE=UTC
ARG BUILD_PHP_IDEKEY=PHPSTORM
ARG BUILD_XDEBUG_ENABLE=1

## Build from the php-cli image based on Debian Buster.
FROM php:${BUILD_PHP_VERSION}-cli-buster

LABEL maintainer="luca@theaveragedev.com"

## Install XDebug extension.
RUN pecl install xdebug

## Create the /project directory and change to it.
WORKDIR /project

## Create an executable wrapper to execute the project vendor/bin/codecept binary appending command arguments to it.
RUN echo '#! /usr/bin/env bash\n\
## Run the command arguments on the project Codeception binary.\n\
vendor/bin/codecept $@\n' \
> /usr/local/bin/codecept \
&&  chmod +x /usr/local/bin/codecept

## By default run the wrapper when the container runs.
ENTRYPOINT ["/usr/local/bin/codecept"]

## At the end to leverage Docker build cache.
COPY . /project

## Set some ini values for PHP.
## Among them configure the XDebug extension.
RUN echo "date.timezone = ${BUILD_PHP_TIMEZONE}\n\
\n\
[xdebug]\n\
zend_extension=xdebug.so\n\
xdebug.idekey=${BUILD_PHP_IDEKEY}\n\
xdebug.remote_enable=${BUILD_XDEBUG_ENABLE}\n\
" >> /usr/local/etc/php/conf.d/99-overrides.ini

I've finally modified the Makefile to force a build, that benefitting from Docker build cache, before running the tests:

RUNS = 1 2 3
docker_runs = $(addprefix run_, $(RUNS))

$(docker_runs): run_%:
   @docker-compose run --rm wpbrowser run test_suite -f -q --no-rebuild

build_test_container:
   @docker-compose build

pll_docker_builds: $(docker_runs)

Since parallel execution, in make applies to any specified target, I have to ensure the build_test_container target runs before the pll_docker_builds one.

It's as simple as concatenating two calls to make:

make build_test_container && make -j -O pll_docker_builds

[](https://theaveragedev.com/wp-content/uploads/2019/08/good-parallel-1.png)

You can find the code I've shown in this post here

Next

I need to put all this together to try and run all the suites in parallel.
New challenges await me as not all suites have no infrastructure requirements as the unit suite does, and many require a database, a web-server, and a Chromedriver container to run acceptance tests.

This post is the second in a series, and you can find the first one here.

In my previous post I've written code that would work on my machine, a Mac OS machine.
To make this of any value, to myself and others, I want to spend some time trying to make the simple setup I have work with Windows and Linux too.
Since PHPStorm and similar IDEs are widely available on all platforms, the part I need to tweak and update is the one dealing with the stack setup.

Making it work on Linux

Looking at the current version of the docker-compose.yml file, making the current set up work on Linux requires the XDEBUG_REMOTE_HOST environment variable to be correctly set:

version: '3.2'

services:

  wpbrowser:
    # Instead of using a pre-built image, let's build it from the file.
    build:
      context: ./docker/wpbrowser
      dockerfile: Dockerfile
      args:
        # By default use PHP 7.3 but allow overriding the version.
        - BUILD_PHP_VERSION=${TEST_PHP_VERSION:-7.3}
    volumes:
      # Bind the project files into the /project directory.
      - ./:/project
    environment:
      # As XDebug remote host use the host hostname on Docker for Mac or Windows, but allow overriding it.
      # Build systems are usually Linux-based.
      # Use default port, 9000, by default.
      XDEBUG_CONFIG: "remote_host=${XDEBUG_REMOTE_HOST:-host.docker.internal} remote_port=${XDEBUG_REMOTE_PORT:-9000}"
      # I use PHPStorm as IDE and this makes the handshake easier.
      PHP_IDE_CONFIG: "serverName=wp.test"

From a terminal emulator, I can get the docker host IP address with this command:

ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+'

It's not my finding or invention: I've found it around the web, precisely here.

Putting the pieces together, this one-line command will allow me to run unit tests, with XDebug support, on Linux:

XDEBUG_REMOTE_HOST="$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')" \
    docker-compose run --rm -e XDEBUG_REMOTE_HOST="${XDEBUG_REMOTE_HOST}" wpbrowser run unit

It works, as shown in the screenshots below; it's not the most "accessible" and most effortless command to remember, but it works.

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-linux-2.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-linux-3.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-linux-1.png)

Making it work on Windows

According to the Docker for Windows documentation Docker on Windows, I'm excluding WSL at this stage, should expose the container host at the host.docker.internal hostname.
After I've installed Docker on Windows, I launch cmder and make sure all works as expected with the following commands:

docker --version
docker run --rm -it alpine ping host.docker.internal

[](https://theaveragedev.com/wp-content/uploads/2019/08/docker-on-windows-output-1.png)

I've already installed PHP, the extensions required by wp-browser , and Composer on the Windows machine using chocolatey.
Time to install Composer dependencies using composer install and wait for the dependencies installation to finish.

I configure PHPStorm the same way I've configured it on Mac:

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-windows-1.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-windows-2.png)

Running the same docker-compose (with some differences) command I've used to run the tests on Mac should yield the same results.

docker-compose.exe build
docker-compose.exe run --rm wpbrowser run unit

So it does.

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-windows-3.png)

Windows is not my daily driver, but I have to admit the experience, so far, is a pleasing one.

Making it work on Windows Subsystem for Linux (WSL)

The round would not be complete without trying out WSL.
I've installed Bash on Windows, the required PHP packages and made sure I can connect to the Docker daemon from the bash session.
I've followed this guide and it worked... flawlessly.

[](https://theaveragedev.com/wp-content/uploads/2019/08/docker-on-wsl-1.png)

Due to the number of layers involved, I'm taking baby steps to ensure all the moving parts are there are working as expected.

The first step is making sure I can get the IP address of the host machine (Windows) from WSL. I use the same command I would use on a Linux host:

ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+'

This command, though, does not have the desired effect: the docker0 network does not exist in the context of a WSL session.

[](https://theaveragedev.com/wp-content/uploads/2019/08/docker-on-wsl-2.png)

This result is not surprising: the docker client is connecting to the Docker host, see the blog post I've linked above, via the tco://localhost:2375 address.
On a real Linux machine, the Docker server would be available on a socket; it's not the case in WSL.

If I'm still using the Windows implementation of Docker when using WSL, then this means I should be able to ping the Docker host from the containers using host.docker.internal.
I'm using the alpine container to make sure of that:

docker run --rm -it alpine ping host.docker.internal

[](https://theaveragedev.com/wp-content/uploads/2019/08/docker-on-wsl-3.png)

As expected, it works.

Since the Docker host is still the Windows machine, the PHPStorm configuration is not changed and the XDebug connection just works:

[](https://theaveragedev.com/wp-content/uploads/2019/08/xdebug-on-wsl-1.png)

As the last step I want, in the context of Windows and WSL, know what the PHP_OS_FAMILY constant value is.
I run the same command on a cmder and and WSL (Ubuntu 18.04) session:

[](https://theaveragedev.com/wp-content/uploads/2019/08/php-on-windows-1.png)

From PHP point of view, then, WSL is a Linux system.
For the sake of completeness running the same command on Mac and Linux yields the following results:

[](https://theaveragedev.com/wp-content/uploads/2019/08/php-on-mac-1.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/php-on-linux-1.png)

Next

The next logical step is hiding the complexity involved in "looking around" and running the correct commands in a script.
Codeception is using the Symfony Console component to manage its own CLI, and wp-browser is doing the same with its custom commands; adding another command, maybe more, dedicated to running the tests in containers seems a reasonable choice.

The problem

In a perfect world, tests should "fail fast".
Failing "fast" means the tests should fail as soon as possible, providing me enough feedback to go back, fix the code, and try again.
As the volume of tests and test suites mounts, this proposition is far removed from the seconds-long run time of unit tests to move into the minutes-long run time of multiple acceptance suites.
Even worse: at times the testing load is so heavy for a local machine, or a locally-deployed infrastructure, that tests time out or otherwise fail for the wrong reasons.

I run into this issue while working on wp-browser and dealing with its test base; the solution is to close any other application and cross my fingers as my laptop churns along.
Yet this is not a sustainable approach.

In this series of posts I try and document the steps, and the thought process behind those steps, to set up a Docker-based parallel build that I should be able to run locally and on the build system of choice (currently Travis CI).
Without having to quit the world to run them from start to finish, possibly.

The files I'm showing in these posts are current to wp-browser version 2.2.18.

Disclaimer about my incompetence

I'm not a DevOps person. I know how to set up and use the systems I need to deliver code, but not much more. I'm laying out my reasoning and documenting my approaches, but those might be wrong at any step: please keep that in mind while reading through.

The current build system

Before I start to write code like a mad man it's worth illustrating what I have now as a working (although with some hiccups, at times) build system.

When I open a Pull-Request or push code in any way Travis CI runs the build from this configuration file (you can see the configuration file here):

sudo: required

language: php

notifications:
  email: false

matrix:
  include:
    - php: '5.6'
      env: CODECEPTION_VERSION="^2.5"
    - php: '5.6'
      env: CODECEPTION_VERSION="^3.0"
    - php: '7.0'
      env: CODECEPTION_VERSION="^2.5"
    - php: '7.0'
      env: CODECEPTION_VERSION="^3.0"
    - php: '7.1'
      env: CODECEPTION_VERSION="^2.5"
    - php: '7.1'
      env: CODECEPTION_VERSION="^3.0"
    - php: '7.2'
      env: CODECEPTION_VERSION="^2.5"
    - php: '7.2'
      env: CODECEPTION_VERSION="^3.0"

services:
  - docker

cache:
  apt: true
  directories:
    - $HOME/.composer/cache/files

addons:
  hosts:
    - wp.test
    - test1.wp.test
    - test2.wp.test
    - blog0.wp.test
    - blog1.wp.test
    - blog2.wp.test
    - mu-subdir.test
    - mu-subdomain.test

env:
  global:
    - WP_FOLDER="vendor/johnpbloch/wordpress-core"
    - WP_URL="http://wp.test"
    - WP_DOMAIN="wp.test"
    - DB_NAME="test_site"
    - TEST_DB_NAME="test"
    - WP_TABLE_PREFIX="wp_"
    - WP_ADMIN_USERNAME="admin"
    - WP_ADMIN_PASSWORD="admin"
    - WP_SUBDOMAIN_1="test1"
    - WP_SUBDOMAIN_1_TITLE="Test Subdomain 1"
    - WP_SUBDOMAIN_2="test2"
    - WP_SUBDOMAIN_2_TITLE="Test Subdomain 2"
  matrix:
    - WP_VERSION=latest

before_install:
  - make ci_before_install
  - make ensure_pingable_hosts
  # Make Composer binaries available w/o the vendor/bin prefix.
  - export PATH=vendor/bin:$PATH

install:
  - make ci_install

before_script:
  - make ci_before_script

script:
  - make ci_script
  - php docs/bin/sniff

I've configured the build to run a matrix of tests for PHP and Codeception versions, but There are some parts here that are worth pointing out in more detail.

I use make

You can see the Makefile here.

I find myself comfortable using make and Makefiles to automate my stuff.
I would not use make in code I share with others, but I'm fine using it here.
I've been using make in my previous life and enjoy its low-level approach to the tasks; if you've never used make before it's (I apologize to the experts for the over-simplification) an automation tool like composer run, npm run, gulp, grunt and similar would be.
But meaner and badder.

I run the tests on the host

I currently run the tests from the host machine, as one can see looking at the part of the Makefile in charge of running the tests, the make ci_script target. I use a docker-compose stack to spin up the containers I need, set them up, and run the tests from the host machine relying on the containers to serve the web requests and the database.
This detail is crucial as it's one of the limiting factors of the current build system: since each container with access to the code might modify the content of the host files (e.g., by adding a must-use plugin) having multiple Docker containers all accessing the same host files at the same time could lead to inconsistencies.

You can see the docker-compose file here.

I run the test suites sequentially

You can see the relevant make target here.

Due to limitations imposed by WordPress use of globals and constants, I'm running each suite with a dedicated command.
I want, and must, keep this behavior, but I would like the test runs to be non-blocking and parallelized, not dissimilar to what Codeception documentation suggests.
My initial idea is to leverage make built-in parallelism support to run each suite in parallel to the others; it might not be attainable, but I would like to keep the moving parts simple, if I can.

Objective 1: running unit tests in the container

I've forked latest master, version 2.2.18 of wp-browser, and started working on that.
The first objective is the ability to run the unit suite tests with one command, and the tests should run in the container.
Furthermore, I should be able to debug the tests using XDebug; this means I will bind files into the container in place of relying on the container-managed files.

I want to keep the ci_script target in the Makefile to "hide" more complex commands, issued to the docker-compose binary, in it, but am not against having something I could run in a Composer script too.

I feel the following command is an acceptable "API" to run the unit suite from the project root folder:

docker-compose run --rm wpbrowser run unit

Breaking down the command a bit: * I'm running the wpbrowser container and removing it afterward, which means I'm using the container as an executable * in that container I'm running the run unit command

I'll be implementing the build infrastructure with a test-driven development approach; with some adaptations, of course.
Running the command, right now, yields this error:

ERROR:
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

Time to create a docker-compose.yml file in the root folder:

version: '3.2'

services:

  wpbrowser:
    # Instead of using a pre-built image, let's build it from the file, see later.
    build:
      context: ./docker/wpbrowser
      dockerfile: Dockerfile
      args:
      # In the CI environment I need to test against different PHP versions.
      # Default to 7.3 but allow for its override.
        - BUILD_PHP_VERSION=${TEST_PHP_VERSION:-7.3}
    volumes:
      # Bind the project files into the /project directory.
      - ./:/project

In the docker/wpbrowser folder I've created the following Dockerfile:

## Take the BUILD_PHP_VERSION argument into account, default to 7.3.
ARG BUILD_PHP_VERSION=7.3

## Build from the php-cli image based on Debian Buster.
FROM php:${BUILD_PHP_VERSION}-cli-buster

LABEL maintainer="luca@theaveragedev.com"

## Set PHP timezone to avoid warnings.
RUN echo "date.timezone = UTC" >> /usr/local/etc/php/php.ini

## Create the /project directory and change to it.
WORKDIR /project

## Create an executable wrapper to execute the project vendor/bin/codecept binary appending command arguments to it.
RUN echo '#! /usr/bin/env bash\n\
## Run the command arguments on the project Codeception binary.\n\
vendor/bin/codecept $@\n' \
> /usr/local/bin/codecept \
&&  chmod +x /usr/local/bin/codecept

## By default run the wrapper when the container runs.
ENTRYPOINT ["/usr/local/bin/codecept"]

With these files in place, I can run the following command successfully and see the tests run:

docker-compose build
docker-compose run --rm wpbrowser run unit

Objective 2: debugging unit tests with XDebug

Whatever tests I'm running, I need to debug them using XDebug.

Code coverage generation requires XDebug so it's better addressing the issue now, while the stack is simple.

A rapid introduction to XDebug

Being able to debug one's code is considered a junior developer required skill; I very much agree with that.
XDebug, when used to run remote debug sessions, has two components to it: a client and a server.
Similarly to many client-server architectures, the client sends requests to the server, and the server processes them.
I've seen many developers failing to understand, though, that the machine running the PHP code is the client and the machine debugging the code is the server.
In this container stack, then, the client is the container, and the server is my debug tool of choice: PHPStorm running on my host machine (my laptop).
The second part of understanding is the one where the client needs to know-how, and when, to make requests to the server; this is accomplished by installing the XDebug PHP extension.

Installing and configuring XDebug in the container

Installing it is as simple as modifying the docker/wpbrowser/Dockerfile file:

## Take the BUILD_PHP_VERSION argument into account, default to 7.3.
ARG BUILD_PHP_VERSION=7.3

## Allow for the customization of other PHP parameters.
ARG BUILD_PHP_TIMEZONE=UTC
ARG BUILD_PHP_IDEKEY=PHPSTORM

## Build from the php-cli image based on Debian Buster.
FROM php:${BUILD_PHP_VERSION}-cli-buster

LABEL maintainer="luca@theaveragedev.com"

## Install XDebug extension.
RUN pecl install xdebug

## Set some ini values for PHP.
## Among them configure the XDebug extension.
RUN echo "date.timezone = ${BUILD_PHP_TIMEZONE}\n\
\n\
[xdebug]\n\
zend_extension=xdebug.so\n\
xdebug.idekey=${BUILD_PHP_IDEKEY}\n\
xdebug.remote_enable=1\n\
" >> /usr/local/etc/php/conf.d/99-overrides.ini

## Create the /project directory and change to it.
WORKDIR /project

## Create an executable wrapper to execute the project vendor/bin/codecept binary appending command arguments to it.
RUN echo '#! /usr/bin/env bash\n\
## Run the command arguments on the project Codeception binary.\n\
vendor/bin/codecept $@\n' \
> /usr/local/bin/codecept \
&&  chmod +x /usr/local/bin/codecept

## By default run the wrapper when the container runs.
ENTRYPOINT ["/usr/local/bin/codecept"]

Note that, in the file, I'm also setting up Xdebug with the following configuration parameters:

• zend_extension=xdebug.so - installing the extension is not enough; I need to tell PHP to use it explicitly.
• xdebug.idekey=PHPSTORM - the key by which the client request identifies itself. Sticking to the client-server architecture from before it's easy to understand how I could have many clients (machines running PHP code) running connecting to the server (my IDE, PHPStorm, running on my laptop): the server needs to know "who's calling"
• xdebug.remote_enable=1 - by default XDebug client and server would run on the same machine; it's not the case here as the machine running the code is the container (although virtual it's still a separate machine), while the server runs on my host machine (my laptop). The two machines communicate using an IP Address (sorta, see later) and not on localhost, so this is a remote connection.

I'm configuring the client and there is still one information missing: the server address, remote_host in XDebug terms.
Before any code example, it's important to understand remote_host means "what IP address or hostname should the client call to connect to the server?".
That IP address is **the IP address of my host machine, from the point of view of the container".
That IP address (hostname, really) is, in Docker for Mac and Windows, host.docker.internal.
Since I use Mac OS as my daily driver, it's reasonable, for me, to use host.docker.internal as a default, yet I can leave further scripting the possibility to set that value dynamically.

Below the docker-compose.yml file to accompany the update to the docker/wpbrowser/Dockerfile:

version: '3.2'

services:

  wpbrowser:
    # Instead of using a pre-built image, let's build it from the file.
    build:
      context: ./docker/wpbrowser
      dockerfile: Dockerfile
      args:
        # By default use PHP 7.3 but allow overriding the version.
        - BUILD_PHP_VERSION=${TEST_PHP_VERSION:-7.3}
    volumes:
      # Bind the project files into the /project directory.
      - ./:/project
    environment:
      # As XDebug remote host use the host hostname on Docker for Mac or Windows, but allow overriding it.
      # Build systems are usually Linux-based.
      # Use default port, 9000, by default.
      XDEBUG_CONFIG: "remote_host=${XDEBUG_REMOTE_HOST:-host.docker.internal} remote_port=${XDEBUG_REMOTE_PORT:-9000}"
      # I use PHPStorm as IDE and this will make the handshake easier.
      PHP_IDE_CONFIG: "serverName=wp.test"

I have, so far, configured only the XDebug client, it's time to configure the server: my IDE, PHPStorm.

Configuring PHPStorm to listen for XDebug connections

The debugging tool, my IDE in this example, is the server listening for XDebug connections from the client (the container) and "processing" them.
That processing is me scouring the code line by line or looking at variable values; it's slow and inefficient "processing", yet it's still a process.

When I've finished debugging, and allow the code to "move on", the client receives an "OK" response from the server and moves on. The cycle restarts for the next breakpoint, and so on.

From the values set in the section above I've configured PHPStorm to listen on port 9000 for XDebug connections:

[](https://theaveragedev.com/wp-content/uploads/2019/08/phpstorm-xdebug-config-1.png)

Also, set up the Servers accordingly:

[](https://theaveragedev.com/wp-content/uploads/2019/08/phpstorm-xdebug-config-2.png)

Running the tests with XDebug

Now that it's all set up, it's time to set a breakpoint in the code, run the tests, and see the code execution stop:

[](https://theaveragedev.com/wp-content/uploads/2019/08/phpstorm-xdebug-set-breakpoint.png)

[](https://theaveragedev.com/wp-content/uploads/2019/08/phpstorm-xdebug-execution-stopped.png)

Next

In my next post I will try to refine this configuration a bit further to make it more portable (e.g., easily usable on Linux) and more flexible: I've hard-coded XDebug usage in the code and having XDebug active all the times might not be the best choice in terms of speed.

This post is the second of a series, read the first post here. The code for this post can be found on GitHub if you want to follow along while reading it.

Conventions

I'm running any terminal command I show in the post, unless otherwise noted, from within the VVV virtual machine, from the root folder or the plugin.
To log into the virtual machine I change directory to the ~/vagrant-local folder on my laptop and run vagrant ssh.
Once I'm inside the box and am presented with the prompt I change directory to the plugin root folder:

cd /srv/www/wordpress-one/public_html/wp-content/plugins/reading-time-blocks

Writing the first feature

I've set up my test environment as detailed in the first post, and it's now time to write the first real test to drive my development.
I start, as I usually do, with an acceptance test to provide me with an "umbrella" under which I can modify the code, at this stage still in draft version, to make it behave as intended first, and implement it as I want second.
I've removed the first acceptance test I've created, the one whose only purpose was to confirm the setup was correctly working, and created a new feature to drive my behavior-driven development (BDD) and write as little code as possible to make it work as intended.
From the plugin root folder I run:

codecept g:feature acceptance "Base estimated remaining reading time block"

With this command, I've created a feature file in the tests/acceptance/Base estimated remaining reading time block.feature file.
This file, with the .feature extension is a Gherkin syntax file as those used by other BDD frameworks like Cucumber or Behat; Gherkin is a structured file written in plain language that, and I'm doing do that below, can be translated to test code.
The "philosophy" advantage of Gherkin is that its format enforces thinking about behavior in place of implementation; I find this, on a high-level, an excellent way to drive my development.
After filling in the "blanks" of the feature file I've ended up with this:

Feature: Base estimated remaining reading time block
  In order to show readers the estimated remaining reading time
  As a post editor
  I need to be able to drop the estimated reading time block anywhere on the page

  Scenario: when inserted at the start of the post the block shows the est. remaining reading time for the whole post

"Steps" should appear in the "Scenario" section; since there are currently none running the feature yields a success:

Writing the feature steps

Adding steps to a feature usually involves, at least, two phases:

1. write the step in plain language
2. if not already implemented implement the step code I've yet to write anything, so I have to do both, at least initially, for all the steps I'm adding.
   I've updated the feature text to this:

Feature: Base estimated remaining reading time block
  In order to show readers the estimated remaining reading time
  As a post editor
  I need to be able to drop the estimated reading time block anywhere on the page

  Scenario: when inserted at the start of the post the block shows the est. remaining reading time for the whole post
  Given a post of "400" words
  And the "tad/reading-time" block is placed at the start of the post
  When I see the post on the frontend
  Then I should see the block shows an estimated reading time of "about 2 minutes"

When I rerun the test Codeception lets me know I've not implemented the steps yet:

To "translate" the steps into the code I can leverage Codeception built-in generator to kickstart my implementation:

codecept gherkin:snippets acceptance

I can copy and paste the code in the tests/_support/AcceptanceTester.php as they are but that would lead, as I add more and more steps, to a monster class with far too many responsibilities.

Organizing the steps

To keep the code clean and organized I've created a tests/_support/Traits/Gherkin/ folder and, in there, I've created two traits to start with:

• tests/_support/Traits/Gherkin/Editor - this trait implements all the steps dealing with the backend editor UI; e.g., adding and removing blocks, entering content, saving the post and more.
• tests/_support/Traits/Gherkin/Frontend - this trait implements all the steps dealing with the site front-end and UI like navigating to the single post page or making assertions on the front-end version of a post.

I've also modified the names, slugs and icon of the block to remove the example placeholders created by the create-guten-block scaffold operation and rebuilt the assets using npm run build; currently the plugin is registering one editor block with the tad/reading-time namespace and slug.
I mention this here as I'm using the slug to add the block during the tests.
I've modified the composer.json file to autoload classes in the Test namespace in the context of development (in the autoload-dev section):

{
  "name": "lucatume/reading-time-blocks",
  "description": "Gutenberg blocks to display a post reading time.",
  "type": "wordpress-plugin",
  "license": "GPL-2.0-or-later",
  "authors": [
    {
      "name": "Luca Tumedei",
      "email": "luca@theaveragedev.com"
    }
  ],
  "minimum-stability": "stable",
  "require": {},
  "require-dev": {
    "lucatume/wp-browser": "^2.2"
  },
  "autoload-dev": {
    "psr-4": {
      "Test\\": "tests/_support/"
    }
  }
}

Finally I add use the two new traits in the AcceptanceTester class:

use Test\Traits\Gherkin\Editor;
use Test\Traits\Gherkin\Frontend;

class AcceptanceTester extends \Codeception\Actor {

  use _generated\AcceptanceTesterActions;
  use Editor;
  use Frontend;

  /**
   * Traits will store in this associative arrays all relevant information generated or used in test steps.
   *
   * @var array
   */
  protected $data = [];
}

The Frontend trait

I start the translation of plain language into code from the Test\Gherkin\Frontend trait as that is the easiest one.
Since the acceptance suite is using the WPWebDriver module I leverage the methods provided by it to implement the steps:

/**
 * Implements all the steps dealing with the site front-end and UI; e.g., navigating to the single post page or making
 * assertions on the front-end version of a post.
 *
 * @package Test\Gherkin
 */

namespace Test\Traits\Gherkin;

/**
 * Trait Frontend
 *
 * @package Test\Gherkin
 */
trait Frontend {

  /**
   * @When I see the post on the frontend
   */
  public function iSeeThePostOnTheFrontend() {
    // Use the `post_id` previously saved in the shared data array.
    $this->amOnPage( '/index.php?p=' . $this->data['postId'] );
  }

  /**
   * @Then I should see the block shows an estimated reading time of :readingTimeEstimation
   */
  public function iShouldSeeTheBlockShowsAnEstimatedReadingTimeOfMinutes( $readingTimeEstimation ) {
    $this->see( $readingTimeEstimation );
  }
}

The WPWebDriver from wp-browser is an extension of the WebDriver one from Codeception and inherits its methods. Here I'm not doing anything fancy, but it's worth pointing out that, in the iSeeThePostOnTheFrontend method, I'm using the post_id stored in the $data array. That array is initialized in the AcceptanceTester class and is meant exactly to serve as a data sharing buffer between steps that are, in this case, implemented in different traits.

The Editor trait

This trait is not different, in concept, from the Frontend one but proved to be difficult due to the many changes, in UI and interaction, the new WordPress editor brought with it.
The main difference, and the code fully highlights it, is that almost any interaction with the new editor has to happen via its Javascript API.
The WPWebDriver module inherits, from the base Codeception WebDriver module, the almighty executeJs method.
This method executes a string of Javascript code in the page the driven browser is visiting (Chrome in this case, driven by Chromedriver, see the first article for the setup); this is equivalent to opening the developer tools, switching to the console, and executing Javascript code in it:

The trait methods are leveraging this to interact with the editor, add blocks to it and save the post.
Currently, wp-browser does not offer a module dedicated to the new WordPress editor, but I feel like it will soon.
After many trials and errors here's the code of the Test\Editor trait:

<?php
/**
 * Implements all the steps dealing with the backend editor UI; e.g., adding and removing blocks, entering content and
 * so on.
 *
 * @package Test\Gherkin
 */

namespace Test\Traits\Gherkin;

/**
 * Trait Editor
 *
 * @package Test\Gherkin
 */
trait Editor {

  /**
   * Whether the scenario already logged in or not.
   *
   * @var bool
   */
  protected $logged = false;

  /**
   * Adds an editor block of the specified type to the current editor.
   *
   * @param      string $type The block name in the format `namespace/name`.
   * @param array $props An array of properties to initialize the block with.
   * @param null  $position The position, in the editor, the block should be inserted at.
   *                        Default to append as last.
   *
   * @return string The block `clientId`.
   */
  protected function addBlock( $type, $props = [], $position = null ) {
    $positionString = null !== $position ? ' at position ' . (int) $position : '';
    $jsonProps      = json_encode( $props, JSON_PRETTY_PRINT );
    codecept_debug( "Editor: adding block of type '{$type}'{$positionString} with props:\n" . $jsonProps );
    $position = null === $position ? 'null' : (int) $position;

    $js       = "return (function(){
      var block = wp.blocks.createBlock('{$type}', $jsonProps);
      wp.data.dispatch('core/editor').insertBlock(block, {$position});
      return block.clientId;
    })();";
    $clientId = $this->executeJS( $js );

    codecept_debug( "Editor: added block of type '{$type}'{$positionString}, block clientId is '{$clientId}'." );

    return $clientId;
  }

  /**
   * Triggers the save post action on the editor.
   *
   * After triggering the save action the method will wait for a set number of seconds before
   * returning.
   *
   * @param int $wait How much to wait for the post to save after triggering the save actions.
   *                  The saving will happen asynchronously.
   *
   * @return int The saved post ID.
   */
  protected function savePost( $wait = 2 ) {
    codecept_debug( 'Editor: saving the post.' );
    $postId = $this->executeJS( 'return (function(){
      wp.data.dispatch("core/editor").savePost();
      return wp.data.select("core/editor").getCurrentPostId();
    })(); ' );
    codecept_debug( "Editor: saved the post, post ID is {$postId}" );

    $this->wait( $wait );

    return $postId;
  }

  /**
   * Clicks the tooltip close ("X") icon to disable tooltips for the current user.
   */
  public function disableEditorTips() {
    codecept_debug( 'Editor: disabling tips.' );
    $this->executeJS( 'document.querySelector("#editor button.nux-dot-tip__disable").click()' );
  }

  /**
   * Edits the post adding properties to it.
   *
   * The post is not saved after it, use the `savePost` method to save the changes.
   *
   * @param array $props An associative array of properties to set for the post.
   *
   * @return int The edited post ID.
   */
  protected function editPost( array $props = [] ) {
    $jsonProps = json_encode( $props, JSON_PRETTY_PRINT );
    codecept_debug( 'Editor: editing the post with properties: ' . $jsonProps );
    $postId = $this->executeJS( 'return (function(){
      wp.data.dispatch("core/editor").editPost(' . $jsonProps . ');
      return wp.data.select("core/editor").getCurrentPostId();
    })(); ' );
    codecept_debug( "Editor: edited the post, post ID is {$postId}" );

    return $postId;
  }

  /**
   * @Given a post of :wordCount words
   *
   * @param int $wordCount The number of words in the post content.
   */
  public function aPostOfWords( $wordCount ) {
    if ( ! $this->logged ) {
      $this->loginAsAdmin();
    }

    $this->amOnAdminPage( 'post-new.php' );
    $this->disableEditorTips();
    $this->fillField( '#post-title-0', 'Test post' );

    // Create the test content: the word "test" repeated n times.
    $content = implode( ' ', array_fill( 0, (int) $wordCount, 'test' ) );
    $this->addBlock( 'core/paragraph', [ 'content' => $content ] );
    // Save the post to commit the changes.
    $this->editPost( [ 'status' => 'publish' ] );
    // Save the post ID in the shared data array.
    $this->data['postId'] = $this->savePost();
  }


  /**
   * @Given the :type block is placed at the start of the post
   *
   * @param string $type The block type, in the format `namespace/type`.
   */
  public function theBlockIsPlacedAtTheStartOfThePost( $type ) {
    $this->addBlock( $type, [], 0 );
    // Save the post to commit the changes.
    $this->savePost();
  }
  }

This code presents an efficiency, and possible cause of errors, in the savePost method: the arbitrary wait could either be too long or too short leading, in the first case, to wasted running time and, in the second, to false negatives (a test that fails for the wrong reason).
I plan to revisit the code with more time to wait for exactly as much as needed after firing the post save operation; for the time being the method gets the job done and I can run the test to see it fail:

As expected the frontend test is failing as my block is not printing anything of value:

In my next post, I will work to make this test pass and document my progress.
The code for this post can be found on GitHub.

The final version of the plugin code shown in this article is on Github.

You can read the second post in the series here.

What is Gutenberg, the abridged version

"Gutenberg" is the codename of the project that meant to replace the default WordPress post editor, the TinyMCE based one, with a new, more modern and more flexible one.
From a development point of view there are number of changes: a shift toward Javascript, the change of paradigm from an editor and meta-boxes to the unifying concept of "blocks" and other significant changes; find out more about Gutenberg heading to the introductory page dedicated to it or run a quick search on the web.
Many words have been written in support or against the project, that I'm leaving out, to concentrate on the exciting part: can I develop a simple Gutenberg block with behavior-driven development techniques?

The blocks I'm developing

Since I've got still to write a single line of code in the context of Gutenberg, I start with something that seems simple enough: an estimated reading time block. Or rather: two blocks.
The first block should allow a post author to drop the block anywhere in the post content to display, the estimated remaining reading time from that point forward. E.g., if a post requires 4 minutes to read the block should display "Reading time about 4 minutes" when placed at the beginning of the post and "Reading time about 2 minutes" when positioned halfway through the post.
The second block should provide the reading time of a post section, attached to a header element it should provide the estimated reading time of that header tree. E.g., a post could have an estimated reading time of 6 minutes and 3 roughly equal sub-sections each starting with an h3 header: attaching the block to one of those h3 headers should display "Reading time about 2 minutes".
It's important to note this is what I would like to accomplish: I have no idea if I can do this or if it's possible at all; I will reconsider my objectives along the way.

What I would like to touch

I'm way more accustomed to PHP testing than I am with Javascript testing and development; I would enjoy writing a bunch of Javascript tests alongside the PHP ones.
My BDD/TDD tool of choice is Codeception and, with it, wp-browser; what I love about Codeception is how it ties and manages all the types of testing in one* API: I would like to see if Javascript testing can be part of that flow.
I also would like to learn to work with Gutenberg the only way I can learn: with tests.

The VVV based development environment

My day-to-day local environment is a mix of Valet, Local by Flywheel, Docker containers and Virtual box managed virtual machines: the perk of being a freelancer is that I get to work with many different technology stacks and approaches.
For repeatability and ease of use I will work, for this project, using VVV: it's the "official" way of developing on WordPress, the one that gets the Core-team support and pretty much host-machine independent; the installation guide can be found here.
I will detail, in this article and the next ones, any significant change I make to the default machine and try to provide as much information as I can about what I'm doing.

Installing VVV

I'm not going into the details of how to install VVV are detailed in the installation guide.
I've made no changes to the default installation, and I'm working, for the plugin, in the www/wordpress-one folder.
These folder contents are served, locally, at http://one.wordpress.test/; that's the suggested installation to use to develop plugins and themes, and I stick with that.

A quick check at the site address, http://one.wordpress.test/, and at the site admin area, http://one.wordpress.test/wp-admin/ confirm everything is working as intended.
To access the admin area I can use admin as username and password as the password.

Creating the plugin

As listed on the VVV installation welcome and management screen VVV provides a fully-contained and virtualized environment not only to serve WordPress but, especially, to develop on it. From a terminal window, I can change directory, on my host machine, to the ~/vagrant-local folder and log into the VVV virtual machine with the command vagrant ssh to look around and see what commands are available.
My first check is to see if Composer is available system-wide:

It most definitely is, and it is, furthermore, the latest version as I'm writing.
I use Composer daily and know how detrimental to my flow waiting minutes for it to finish can be; I install the hirak/prestissimo Composer plugin globally immediately to ease the pain:

composer global require hirak/prestissimo

Now that's done it's time to set up my plugin in the correct folder.
Since I'm focusing on the creation of a Gutenberg block, I'm using the create-guten-block CLI tool.
I change directory to the WordPress installation plugins folder and run the create-guten-block scaffold command:

cd /srv/www/wordpress-one/public_html/wp-content/plugins
npx create-guten-block readin-time-blocks

The command above downloads, creates and sets up all the files required to develop and register a Gutenberg block.
I then change directory to the freshly created plugin folder (reading-time-blocks) and initialize a Composer project in it answering some questions:

cd reading-time-blocks
composer init

I've not immediately installed the plugin development dependencies to do that in a second moment; I've slightly modified the main plugin file (reading-time-blocks/plugn.php) and make sure the plugin is showing up as expected in the plugins management screen at http://one.wordpress.test/wp-admin/plugins.php:

The plugin code is accessible, on my host machine, at ~/vagrant-local/www/wordpress-one/public_html/wp-content/plugins/reading-time-blocks; that's where I'll work anytime I need to modify the plugin code.
So far these are the main plugin file, reading-time-blocks/plugin.php, contents:

<?php
/*
Plugin Name: Reading Time Blocks
Description: Gutenberg blocks to display a post reading time.
Version: 0.1.0
Author: Luca Tumedei
Author URI: http://theaveragedev.local
Text Domain: reading-blocks
 */

require_once __DIR__ . '/src/init.php';

The final line makes sure to initialize and register the block, with WordPress and Gutenberg.
I build the block code using npm and wait for the Javascript assets to compile correctly:

npm run-script build

After having activated the plugin, I can add and see the block is displaying correctly, in its initial form, in the post editor:

Time to set up the testing environment.

Installing Codeception and wp-browser

Now that I've got a hold of where the plugin lives it's time to pull the plugin dependencies.
I install wp-browser as a development dependency:

composer require --dev lucatume/wp-browser:^2.2

After a bit, the installation is done, and wp-browser is ready in the vendor folder relatively to the plugin folder.
To ease my flow, I want to make sure I can run any binary installed in the vendor/bin folder without having to prepend vendor/bin every time, e.g. codecept to run Codeception in place of vendor/bin/codecept.
To do that I make a temporary change to the PATH, the list of folders the shell looks up to find commands, and a permanent one:

export PATH=vendor/bin:$PATH
sudo echo 'export PATH=vendor/bin:$PATH' >> ~/.bashrc

While the effect of the first command is reset each time I log out of the virtual machine the second command permanently modifies it, no matter how many times I log out and back in.
To make sure all is working I run codecept --version:

It's now time to bootstrap the wp-browser installation, from the VVV shell I run the command:

codecept init wpbrowser

And answer the questions about my development setup:

Working from inside VVV offers many advantages, among those:

• uniform WordPress path: /srv/www/wordpress-one/public_html
• localhost database at localhost, user root, password root
• the databases I need for both the acceptance and functional tests and the integration tests exist already; the first is the same database that's serving the site at http://one.wordpress.test, wordpressone; the second is the database that's pre-installed to run WordPress tests, wordpress_unit_tets.

The final step is creating the starting database fixture, a database dump, that will be used in the acceptance and functional tests.
Again using WP-CLI, from the plugin root folder in the VVV shell, I remove all content from the site, activate the reading-time-blocks plugin and export a SQL dump in the plugin tests/_data/ folder:

wp plugin deactivate --all
wp plugin activate reading-time-blocks
wp site empty --yes
wp db export tests/_data/dump.sql

Speeding up the acceptance and functional tests

While wp-browser would be ready to run now, I prefer leveraging VVV built-in mysql binary to import the SQL dump in tests.
The PHP-based solution is portable and flexible but slower; the SQL dump I use as starting fixture in acceptance and functional tests, the one I generated during the previous step before, is so small not to make a real difference but good habits are good.
Furthermore, there is no need for URL replacement, VVV provides the mysql command, and it makes sense to speed up the tests as much as I can.
I've updated the tests/acceptance.suite.yml and tests/functional.suite.yml files modules.config.WPDb configuration adding the populator parameter :

populator: 'mysql -u $user $dbname < tests/_data/dump.sql'

This parameter tells Codeception that, when in need of populating the database, it should not use PHP functions but the mysql CLI command as detailed.
The entry is taken straight from Codeception documentation, and VVV environment setup makes it a copy-and-paste well done.

Installing Chrome and Chromedriver

The quick test run confirms that wp-browser is correctly set up.
Since Gutenberg, and with it the blocks my plugin adds, are managed and rendered via Javascript the WPBrowser module that is currently acting a "browser" in acceptance and functional tests does not provide much value in acceptance tests as it does not support Javascript.
To run the tests, I replace it with the WpWebDriver module but that, in turn, requires a "real" browser to be "driven".
In straightforward terms, the test code I write in Codeception will be translated, by WpWebDriver in commands for a browser driver.
The browser driver will then drive a real browser according to those commands.
In my experience, the Chrome browser and its related driver, Chromedriver, provide a smooth, reliable and fast solution for tests and that's what I'll set up the acceptance tests to use.
VVV is built from Ubuntu 14.04 and I've followed this blog post instructions with some modifications.
Here is the full list of the commands I ran to have Chrome and Chromedriver correctly installed in VVV.

First, install the required dependencies:

sudo apt-get install libxss1 libappindicator1 libindicator7 xvfb sudo libgconf-2-4 unzip -y

Download and install Chrome:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome*.deb
sudo apt-get install -f

Download, install and make Chromedriver executable:

wget -N https://chromedriver.storage.googleapis.com/2.44/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
chmod +x chromedriver
sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver

Finally, check the chromedriver command is correctly installed by running:

chromedriver --version

Mind that, in the above commands, I'm downloading the version 2.44 of Chromedriver which is the latest one at the time of this writing; the correct version to run (depending from the installed version of Chrome) might be more recent. Now that I've installed Chromedriver I open a new terminal window, from my host machine, and run vagrant ssh to open a new terminal in VVV. I need to do this as the chromedriver binary must run in the foreground to work and will "capture" the terminal window that's running it.
I start Chromedriver in this new terminal and make sure it's listening for connections on port 9515 (the default port):

chromedriver --url-base=/wd/hub

Chromedriver waits for "driving instructions" to be sent from the WpWebDriver module and will then, in turn, drive a headless (no UI) instance of Chrome through the test steps.
I've updated the tests/acceptance.suite.yml configuration file to use the WpWebDriver module and drive an headless version of Chrome:

actor: AcceptanceTester
modules:
    enabled:
        - WPDb
        - WPWebDriver
        - \Helper\Acceptance
    config:
        WPDb:
            dsn: 'mysql:host=%TEST_SITE_DB_HOST%;dbname=%TEST_SITE_DB_NAME%'
            user: '%TEST_SITE_DB_USER%'
            password: '%TEST_SITE_DB_PASSWORD%'
            dump: 'tests/_data/dump.sql'
            #import the dump before the tests; this means the test site database will be repopulated before the tests.
            populate: true 
            # re-import the dump between tests; this means the test site database will be repopulated between the tests.
            cleanup: true
            populator: 'mysql -u $user $dbname < tests/_data/dump.sql'
            waitlock: 0
            url: '%TEST_SITE_WP_URL%'
            urlReplacement: true #replace the hardcoded dump URL with the one above
            tablePrefix: '%TEST_SITE_TABLE_PREFIX%'
        WPWebDriver:
            url: '%TEST_SITE_WP_URL%'
            adminUsername: '%TEST_SITE_ADMIN_USERNAME%'
            adminPassword: '%TEST_SITE_ADMIN_PASSWORD%'
            adminPath: '%TEST_SITE_WP_ADMIN_PATH%'
            browser: chrome
            port: 9515
            window_size: false
            capabilities:
                chromeOptions:
                    args: ["--headless", "--disable-gpu", "--proxy-server='direct://'", "--proxy-bypass-list=*"]

To make sure Chromedriver is working as intended I've generated a simple test scenario:

codecept g:cept ChromeDriver

Added some simple steps to it:

<?php
$I = new AcceptanceTester( $scenario );
$I->wantTo( 'use Chrome for acceptance tests' );

$I->havePostInDatabase( [ 'post_title' => 'Test post', 'post_status' => 'publish' ] );

$I->amOnPage( '/' );

$I->see( 'Test post' );

I run it from the terminal to make sure it works correctly (notice the tab dedicated to Chromedriver above):

The test passes and I'm ready to get down to real development.