When you write code you have the choice to do per process, per thread, or sequential.

The problem is that doing multiple tests in a shared space doesn't necessarily match the world in which this code is run.

Per process testing allows you to design a test that matches the usage of your codebase. Per thread already constrains you.

For example: we might elect to write a job as a process that runs on demand, and the library we use has a memory leak, but it can't be fixed in reasonable time. Since we write it as a process that gets restarted we manage to constrain the behavior.

Doing multiple tests in multiple threads might not work here as there is a shared space that is retained and isn't representative of real world usage.

Concurrency is a feature of your software that you need to code for. So if you have multiple things happening, then that should be part of your test harness.

The test harness forcing you to think of it isn't always a desirable trait.

That said, I have worked on a codebase where we discovered bugs because the tests were run in parallel, in a shared space.

* Use multiple processes, but multiple tests per process as well.

* Randomly split and order the tests on every run, to encourage catching flakiness. Print the seed for this as part of the test results for reproducibility.

* Tag your tests a lot (this is one place where, as many languages provide, "test classes" or other grouping is very useful). Smoke tests should run before all other tests, and all run in one process (though still in random order). Known long-running tests should be tagged to use a dedicated process and mostly start early (longest first), except that a few cores should be reserved to work through the fast tests so they can fail early.

* If you need to kill a timed-out test even though other tests are still running in the same process - just kill the process anyway, and automatically run the other tests again.

* Have the harness provide fixtures like "this is a temporary directory, you don't have to worry about clearing it on failure", so tests don't have to worry about cleaning up if killed. Actually, why not just randomly kill a few tests regardless?

I wrote some more about tests here [1], but I'm not sure I'll update it any more because of Github's shitty 2FA-but-only-the-inconvenience-not-the-security requirement.

[1]: https://gist.github.com/o11c/ef8f0886d5967dfebc3d

The one thing we've had to be aware of is that the execution model means there can sometimes be differences in behaviour between nextest and cargo test. Very occasionally there are tests that fail in cargo test but succeed in nextest due to better isolation. In practice this just means that we run cargo test in CI.

I'm not actually clear what he means by 'test' to be honest, but I assume he means 'a single test function that can either pass or fail'

Eg in python (nose)

class TestSomething: def test_A(): ... def test_B(): ...

I'm assuming he means test_A. But why not run all of TestSomething in a process?

Honestly, I think the idea of having tests have shared state is bad to begin with (for things that truly matter, eg if the outcome of your test depends on the state of sys.modules, something else is horribly wrong), so I would never make this tradeoff to benefit a scenario that I never think should be done.

Even if we were being absolute purists, this still hasn't solved the problem, the second your process communicates with any other process (or server). And that problem seems largely unsolveable, short of mocking.

Basically, I'm not convinced this is a good tradeoff, because the idea of creating thousands and thousands of processes to run a test suite, even on linux, sounds like a bad idea. (And at work, would definitely be a bad idea, for performance reasons)