Discover more from Robert Roskam's Newsletter
The Ghost Ship Project
Saving sanity and time without permission
Once upon a time, I had a system that needed to be tested heavily for any change we made to it. The system related to money and could easily cause millions of dollars in losses in minutes to hours if a crash-free but semantically incorrect change got made.
So testing it was critical.
However, testing it was also very, very time consuming.
Each test case took 30 min by hand. However, the nature of the changes often meant the bare minimum quantity of tests was easily 20 cases.
To put this in perspective, I would literally spend 1 minute making the change, and 10 hours validating that I did not in fact mistype the numbers or otherwise misunderstand the change. However, the business wanted this level of testing to be confident of the changes.
The desired cadence of the business was same day turn around. So we often had two people working in parallel doing the testing. And they often wanted to do multiples of these changes per week.
On occasion, the change was much larger (1-2 hours to make the change), and then they’d wanted 80 test cases. We only have 4 people available to test the change.
If we all worked with absolute attention flat out, we could reasonably get done by the following morning. Practically, however, the cadence for those changes caused a lead time of a week.
Many months went by. And after a while, I said to myself, “There must be a better way.”
So came up with an idea to automate the test entry and evaluation, and I proposed it to my leadership. I thought it would probably take me a 2-3 weeks to make (~100 hours), and I predicted it would reduce the load down to 15-30 min to validate each change. So the pay off period would likely be about 4-8 weeks.
I presented my findings. I made the case that the present model was unsustainable, and was error prone. Also, it didn’t take into account the fact that the business wanted to make these changes everyday if they could have it.
I got told no.
So then I had a choice, I could listen to what my overlords said, or I go just do it anyhow as part of all this testing I was doing all the time.
I called it my ghost ship project. It was a cursed ship with no crew, but I had hopes that it would magically show up and save my day at some point.
Over a period of about 6 weeks, I made a system with Selenium to drive the UI, and I used it increase my efficiency of my workload that I had. Instead of automating the entire thing at once, I took each task request as an opportunity to complete another link in the change of the automation.
By the end, I would supply a YAML file with the required attributes of the test, and I would get a random sample of test seeds from a library that would fit the requirements.
And it actually performed better than I expected.
I could process about 100 test cases in about 1 minute instead of 40 hours.
Since I made it part of my normal work, nobody cared.
From this single experience I learned the value of asking forgiveness rather than permission. It has served me well since then many times.