How to Sanely Test Complex Systems

And examples in the wild of how to use them effectively

"We don't have tests for that, because it's too complicated."

If you've found yourself saying that, then these two concepts are for you. They are simply: test seeds and test clocks.

These are first-class features of your system, not something you can buy or install or tack on afterwards.

Photo by Michal Matlon on Unsplash

Seeds

Test seeds (aka static test objects) are data objects that are unchanging in your system. Any interactions with them give the same answer every time without fail. It’s not just that they’re idempotent. It’s that they’re designed on purpose to not be changeable so that you can safely have starting points to run other tests from them.

Let’s say you’re making some kind of Shopify competitor. Imagine not having a test environment for your credit card systems. If you want to have different logic for "Insufficient Funds" vs “Normal” spend, you'd need to go out and max out a credit card.

This is potentially expensive enough that you’d almost never do it. And even if you tested the pieces separately, you’d never be 100% certain that the system is working correctly, because of those pesky connection bits potentially being wrong.

Fortunately we have reasonable credit card gateway vendors like Stripe that provide test cards with specific dispositions, such as "Insufficient funds decline".

So what does a test seed look and feel like you’d make? It likely will be an object in the system that doesn’t exist in the database. If you turn the system on in test mode, it’ll get injected so that people can use it, and maybe you’ll persist it to the db for referential integrity.

When you make a test seed, they need some specific attributes to be useful:

• single purpose

• memorable

• stable

Test seeds with more than one purpose become trickier to use and useful, because you need to remember this special behavior. They need to be memorable with simple identifiers, even if they were long.

Finally, people need to be very stable so people can trust them: they need to be able to live in docs, and other places for long periods of time and have organizationally well understood behavior, so that people don't feel like dropping back to manual tests or smoke tests.

Some other common examples of test seeds:

• Twilio has a magic number

• Plaid has several different test_credentials to simulate various things

Clocks

This concept is relatively straight forward: the ability to freeze time or advance time on command in a system in a user session or an API request.

This solution is especially good for handling problems like "we can only test this on the 1st of the month” or “ we need to wait 24 hours to see what happens in the system”. If you can just do those on command, you can iterate faster on your solution.

The way I’ve done this in the past for UIs is to have special area to go to allow users to set the time for the systems. For APIs, Stripe has possibly the most comprehensive solution to this problem I've seen available to developers to play with time for their tests. Plaid is another vendor with several options.

Conclusion

They do take work, and you must treat them like first-classes features. They are features to make your own lives easier in engineering, QA, and Product.

You can’t get to the end of build and add these things usually. You can’t get a vendor to install them for you. You need to have thought of them from the beginning and accommodate them in your approach.

So plan for it, and you too can have sanity even in the most complex systems.