remix logo

Hacker Remix

How rqlite is tested

247 points by otoolep 4 days ago | 52 comments

andrewaylett 3 days ago

A couple of extra points I've found useful:

The first test of a class of tests is the hardest, but it's almost always worth adding. Second and subsequent tests are much easier, especially when using:

Parametrised tests let you test more things without copying boilerplate, but don't throw more variants just to get the count up. Having said that:

Exhaustive validation of constraints, when it's plausible. We have ~100k tests of our translations for one project, validating that every string/locale pair can be resolved. Three lines of code, two seconds of wall-clock time, and we know that everything works. If there are too many variants to run them all, then:

Prop tests, if you can get into them. Again, validate consistency and invariants.

And make sure that you're actually testing what you think you're testing by using mutation testing. It's great for having some confidence that tests actually catch failures.

oweiler 3 days ago

That's why I prefer test frameworks that make parametrised tests so easy that they become the default.

Kotest and Spock are such frameworks.

andrewaylett 2 days ago

Great :). We use Junit5, and a mixture of @EnumSource, @MethodSource, and dynamic tests.

Or Pytest, or a short shell script, or Busted, or Jest, or Node's built in test runner, or a "--selftest" flag to a variant of the native executable.

otoolep 3 days ago

>The first test of a class of tests is the hardest, but it's almost always worth adding.

Agreed, this has been my experience too.

t43562 3 days ago

The testing Pyramid makes sense. The problem for (perhaps) a lot of us will be that we're working on things where some or even all of the levels are missing and we have to try to bring them to a sensible state as fast as possible....but we have limited ability to do so. It's managing imperfection.

We're also possibly working with multiple teams on products that interact and it ends up being "nobody's job" to fill in the e2e layer for example.

Then when someone bites the bullet to get on with it....the whole thing isn't designed to be tested. e.g. how does anyone do testing with Auth0 as their auth mechanism? How do you even get a token to run an E2E type test? I have to screen scrape it which is awful.

Without those E2E tests - even just the test that you can login - the system can break and even when it's a test environment that makes the environment useless for other developers and gets in everyone's way. It becomes the victim's job to debug what change broke login and push the perpetrator to fix it. With automated e2e tests the deployment that broke something is easy to see and rollback before it does any damage.

I suppose I'm challenging the focus in a sense - I care about e2e more because some of those issues block teams from working. If you can't work because of some stupid e2e failure, you can't get fixes out for issues you found in the unit/integration tests.

hitchstory 3 days ago

The pyramid makes sense for certain types of application - very logic heavy with light integration touch points with everything cleanly dependency injected. rqlite fits this pattern.

If you're building a logic lite application which has a lot of integration touch points (I find most commercial code actually fits this pattern) then it makes zero sense - an integration test heavy test suite is what you want - an upside down pyramid if you will.

If you have a ball of mud on your hands then it doesnt matter what kind of app it is, E2E tests are the only thing that makes sense and you need to build very sophisticated fakes (requiring a lot of engineering skill) to avoid things like flakiness bugs and the token scraping problem you referred to.

If you're writing a parser, you probably want 100% unit tests and property tests for all those millions of parser edge cases. No pyramid of any shape is required, just a slab.

The testing pyramid reminds me of microservices: the people who came up with Their Grand Idea had no clue what it was about their specific circumstances that made their approach work for them but still managed to market it as The Way.

Realistically, the ultimate shape of your test suite should be an emergent property based upon a series of smart, local decisions - ideally TDD'ed tests which match the kind of code you are writing right now. Making a particular shape a goal is asinine.

dpc_01234 3 days ago

It's very common mistake in SWE to generalize and extrapolate from own situation.

The way things to do things is very dependent on the project. They way one maintain, debug, test e.g. a math library vs embedded sw vs a video game vs a database vs a enterprise app vs a service in a distributed system are all just different.

Almost all advice should be qualified by "in this domain, one this type of project ...".

didip 4 days ago

I love how dedicated you are with this project, Philip. Been watching it for many many years.

fegu 4 days ago

There seems to be some copy pasta in the FAQ on if any node can be contacted for writes or reads, the paragraph on reads mentions writes.

otoolep 4 days ago

Thanks for flagging -- fixed.

atoav 3 days ago

My feedback regarding the presentation is that I think there should be slightly more focus on why one would choose rqlite over say sqlite. That probably means more info on the distributed part.

I happen to know raft and the kind of problem it solves, but the average reader might not. A practical demonstration of the problems it solves might be in order.

So that also means describing for whicj applications rqlite is more wellsuited (and for which it might be worse) than other databases.

otoolep 3 days ago

I actually touched on this topic somewhat in a recent talk at GopherCon: https://youtu.be/8XbxQ1Epi5w?t=2223

The section of the talk at the timestamp above is titled "Should I use Raft"?