)
)
In the world of tech, failure isn’t just inevitable...it’s invaluable!
From broken APIs to unscalable systems, each stumble offers a rare opportunity to learn, pivot, and build something even better.
"Fail fast" has become a mantra for a good reason: it’s only through trial, error, and the occasional code red epic failure that developers and organisations can evolve their approaches and create truly impactful solutions.
When tech goes wrong, it can go hilariously, head-scratchingly, and sometimes spectacularly wrong.
That’s why we’re diving into the development trenches with 3 Sided Cube’s Technical Team Leads, Kev Borrill and Tom Yeadon!
These two brave souls are taking us on a whistle-stop tour of their most chaotic moments: the unexpected bugs, the near-disasters, and the hindsight moments of “Ooooooh, that’s what we should’ve done.”
From scaling smarter to avoiding API-induced nightmares, Kev and Tom are here to share the hard-earned lessons that every developer—and anyone working in tech—can learn from.
Their experiences remind us that failure isn’t the end of the story; it’s where the best war stories begin.
So, buckle up!
This isn’t your average dev diary. It’s messy, real, and packed with lessons you’ll want to carry into your own projects.
“Cube’s approach to problem solving is very good and, Kev’s right, we adapt and learn and we never make the same mistake twice.” Tom Yeadon, Back End Development Team Lead, 3 Sided Cube
That time a project instantly became unfeasible due to fees…
Sometimes even the greatest ideas just aren’t feasible!
Tom discusses a project he worked on, where payments were collected from users to offset their carbon footprint based on their journeys.
However, although the project was viable from a technical perspective, there were a few associated costs that neither we or the client had foreseen:
Minimum payment fees
Google Maps look up fees
This meant that smaller carbon offsetting transactions that would have been 15p was then turned in to almost £1.
And users did not want to or could not cover those additional fees for every journey they made.
But Kev and Tom aren’t just talking about these mistakes!
What were the learnings from this?
It comes down to: Calculate your costs!
This includes developer, third-party, and any other expenses that might be relevant to the project.
Looking into these beforehand would have meant that the team could have figured out a different user experience pathway early on, rather than finding an alternative solution last minute.
“Just because a product is technically viable, doesn’t mean its financially viable.” Tom Yeadon, Back End Development Team Lead, 3 Sided Cube
)
That time users got sent hundreds of emails…
We’re casting our minds back to an email marketing solution Tom worked on with a previous employer.
It worked by picking up how many recipients there were to be sent to, sending as many as it could, then marking those emails as ‘sent’.
This process was set to run every minute.
This worked when tested with hundreds of recipients, but this was a different story when it came to sending emails to nearly thirty thousand people.
It got stuck in a loop!
Although it was set to run every minute, when dealing with thousands of emails, the email marketing solution took longer than one minute to process all of the recipients and mark them as ‘sent’.
As a result, it was running on top of itself, so it kept picking up the same recipients and sending them the same email!
“The person at the front of the list had got 40 emails, and it sort of started triggering down, so the people at the front had got lots of emails and the people at the end of the list had got no emails.” Tom Yeadon, Back End Development Team Lead, 3 Sided Cube
What learnings did we take from this?
1. Always consider scale
It can be an aspect of a project that is often overlooked!
Your software development can be great, you can have really secure tests around it, you can run through it with test data, but testing something at scale is tricky to do.
You can’t assume that just because it works on a small scale, that it will also work on a large scale.
There’s manual tests and other tools out there that we can utilise to simulate larger scales!
But also, you can’t overengineer your development project, because you’ll never ship it.
There’s a fine balance of considering scale and thinking about it, but not necessarily dealing with the ‘Google scale’ problems at the very start.
“There’s also an element of understanding the real world, in a way.” Kev Borrill, Web Team Lead, 3 Sided Cube
2. Understand how the project is going to be used
If you think about how the project is going to be used in the ‘real world’ and who the people using it are, then you can consider the necessary estimates and simulations.
For example, in this case, an understanding of the sheer amount of emails that would be sent could have encouraged the testing of sending emails to thousands of recipients.
The two actual solutions for this project were:
The emails were picked up in batches, rather than picking up as many as it could
The emails were marked as ‘sent’ as they were sent.
)
That time there was an hour long form with no drafting…
In this project, there was a long form where users would answer different questions that would have taken about an hour to fill out if you knew all the answers.
So, the form was split into different steps that the user could progress through.
But what if a user got bored? Had to do something else? Or they give up and want to come back to it later?
In hindsight, drafting (where a user can save their progress) makes so much sense!
Although the solution was to store updates in the back-end, the need to organise a way to do this towards the end of the project led to technical debt, delays, and increased costs.
What were the learnings from this?
The main learning from this project was: work with end users!
That way, users can provide feedback on what they need from a product and how they will interact with it.
Also, if you can’t work with end users, you can instead be conscious of what they need!
Now, our team is more conscious of anticipating how the users will interact with a product so we can include the best features.
)
That time our project stopped working for no reason…
This one kept Tom guessing.
It happened outside of working hours (during one of our summer parties!) and we were notified that one of our biggest projects had stopped working for no reason!
Thankfully Tom had his laptop, and started investigating while sitting in someone's car on the beachfront.
This project relied heavily on third party data, processing tons of data, and sending out hundreds of thousands of notifications to users.
As Tom dug deeper, he noticed we weren’t getting any data.
It turned out the third-party supplier we relied so heavily on, had turned off their API!
However, it actually turned out that this supplier had communicated six months earlier that this was going to happen.
The problem was that this was an inherited project from another developer and the API notifications were going to them, not us!
Thankfully the issue was resolved as we re-wrote the application to use their new API.
What learnings did we take from this?
1. Monitor dead or old email addresses within your organisation
That way, important emails don’t get missed!
2. Risk planning and disaster recovery
Risk analysis for your applications is so important and can help you mitigate and avoid significant impacts. It’s also important to keep the analysis updated too!
However, you don’t need to act on that risk straight away if the likelihood of it occurring is lower. For example, many organisations would struggle if Gmail went down, but that is less likely to happen as Gmail is quite reliable.
“If you have an application that is heavily heavily reliant or completely dependent on a third party API, you are tied to their success. If they go down, you go down too.” Tom Yeadon, Back End Development Team Lead, 3 Sided Cube
)
That time we had to manually input thousands of rows of data…
This one is a real throwback!
The project had been running for about 8 months, and involved us building a headless content management system (CMS).
A crucial part of this project was that the client needed to import content on a large scale rather than needing to manually add thousands of rows of data.
But there was a simple solution!
The team built a way for the user to export a CSV file that they filled out, sent it to be translated, then sent to us for us to import (or the client could import it themselves).
Perfect! (Or so we thought).
Some time passed as the client worked on filling out the file and then they cam back to us with the spreadsheet in… Google Docs.
This wouldn’t have been a problem as the team could transfer it back to a CSV file, but it turned out that the spreadsheet was completely new!
The format was different and it now used colour coding that wasn’t replicated across all the data.
Although the colour coding made it easier for the translating process, these changes meant that the file couldn’t be uploaded as it was no longer in the specific file format.
The only option was to manually input the data row by row.
This took a lot of time!
But Kev reflects that this was not solely the client's ‘fault’.
What were the learnings from this?
1. Communication is key!
Even though the client made the changes, we could have communicated more clearly how important it was for the data to stay in the necessary format.
Explaining the reasoning would have saved a lot of time on this project.
2. Explore different options to find the ‘best of both worlds’
A different file format, such as Google Sheets, could have been used instead.
This would have allowed for more formatting changes to be made and therefore satisfy the clients’ needs, without affecting the ability to import the data back into the CMS.
However, there are much more solutions now compared to when the team was working on this project.
“We’ve learned, we’ve evolved, we’re pivoting to a better solution.” Tom Yeadon, Back End Development Team Lead, 3 Sided Cube
Join the conversation!
If you want to hear more about Kev and Tom’s learnings then you’re in luck!
They recorded their full conversation for an episode of our Igniting Change podcast.
Whether you’re a developer or just love hearing how we turn challenges into triumphs, this one’s worth a listen (even just to hear Kev and Tom's boss call in to tell them off).
For more things tech for good, stay tuned for our latest blog posts or shout us a holla to get in touch with us, we would love to hear from you 💚
Published on 17 December 2024, last updated on 17 December 2024