A key point is understanding the business (or personal) goal of the system.
Is this a mission-critical system that is core to how the company makes money? Then make sure its robust.
Are you a startup figuring out product-market fit? Then thorough and robust systems matter less. The goal is figuring out how to get users and customers, not stabilize.
I actually wrote more about this a while back: https://www.buildthestage.com/when-should-you-over-engineer-...
A few years later, the park began offering cottages along the route and started paving critical sections, particularly steep areas where trucks (presumably transporting building materials for the cottages) struggled. Had those bridges not already been built to support these trucks, the park would have needed to close the trail for upgrades. Or make a second trail with the new requirements in mind.
The trail expanded faster than it was paved, but each year you could go further as the trail grew longer, and faster as key sections were improved.
If you believe you're inventing a solution, doing just enough to solve the immediate problem and stopping is the consequence.
If you believe you're discovering a solution, diving deeper into the problem to try and uncover some truth about it, stopping at the first solution is not enough if you can't explain why it works or why it will keep working in the foreseeable future – because that involves solving a category of problems, not just this single instance.
People like to think in terms of under vs. over engineering, but I don't think this is the right angle to discuss. You can certainly over engineer the first solution because you focused on a single instance of a problem and "missed the forest for the trees" – identifying a general pattern is useful to find what category of problem you're dealing with, research prior-art on it and uncover elegant/economical solutions.
Something I like to do is funnel the desire to build for something I don't need right now into TODOs and even FIXMEs. Sometimes they just get deleted, but other times I'll come across an old TODO during refactoring and be able to pay down a bit of accumulated technical debt.
This is a good metaphor and is more effective than the neologisms and acronyms of the rest of the article. The author claims a "middle path" but doesn't even connect it to this real world metaphor.
It seems the author is really advocating for the "left path", but only if you are a experienced programmer and with a sprinkle of QC. In the real world metaphor, this would be like hacking through the jungle, but making a little effort to ensure your path is visible to someone else, not unnecessarily dangerous, and makes pragmatic compromises (we will go around this cliff instead of bringing climbing gear or other heavy dependencies).
If you polled 100 SWEs on the example of 'skipping the JSON library and implementing de/serialization for a few objects ' and asked whether they thought it was a "left path" or "right path" solution I'm certain you would have a strong leaning to the left, and not a 50-50 that suggests a secret middle path.
"We don't really need to use REST, we can just create some endpoints that have undocumented side-effects. We don't need to abstract vendor calls into a separate class, we can just implement that functionality directly in our endpoint code."
These sorts of decisions aren't actually materially faster, they're just lazier. And maybe that's "a sprinkle of QC"? But it's a lot of unforced errors that don't really save time to implement, and also create a lot of problems later on.
On the other end, with the "right-pathers", you can have people that really try to over-engineer at any opportunity. This is sort of typical of people who have worked in much larger teams. This can mean building out a k8s cluster when you're still a team of 2-3 people, splitting into 10+ microservices, deciding to use Kafka when a simple queue system would work, building out in-house load balancing for dubious reasons, etc.
The middle path is really something that resembles the "Best Simple System for Now" — when I've done this, I think about how I can solve a problem and not have to rebuild it entirely within 12-18 months.
If the problem is one you faced a billion times, then either use the existing trusted solution and modify it, or if there is no such thing do the proper thing from the start.
If you have a problem where later adjustments are an issue for you (e.g. time wise), solutions that take that into account are superior to ones that are not.
I work with art students and program probably three new projects a week. It is okay to anticipate the needs of the people you work with and not have them spell out everything, it is okay to make a good solution even if no one asked for it, especially if you work with hardware.
E.g. knowing my "customers" I knoe they will return 2 hours before their exam is something is wrong. Guess when that debug mode comes in handy? Exactly then.
There are however ways that needlessly overcomplicate solutions or add more moving parts than needed or simply waste valuable time in the wrong moment. These need to be avoided.
What is the best way to write a program? Depends.
But I actually think that things like JSON Schema, UML, and READMEs are not unnecessary complexity, but rather function as a kind of social language. Just implementing things and not adding complexity to the library means, on the flip side, that there's a high risk of creating a system that only those who already know it can understand.
People always say 'You should YAGNI!' but that often just leads to tribal knowledge. In a startup, that knowledge tends to stay only with the founding members. It would be great if this tribal knowledge were always passed down, but there inevitably comes a point when it breaks, and then you're tied to the founders' bus factor. The code I'm brought in to maintain is exactly like this. Layers of tacit knowledge, like how certain hardware issues were missed, so if you code it the 'correct' way, it breaks. In other words, there have been quite a few cases where you couldn't put everything into code
Of course, documenting everything and drawing UML is also a failure. Personally, I don't think documentation is always necessary either, because keeping documentation up to date also costs time and effort.
And in reality, codebases are never clean. They change shape according to the organization's power structure. If the DevOps team is powerful, the infrastructure code gets thicker. The way API boundaries are drawn shifts depending on how responsibilities are split between backend and frontend teams.
For example, when I participated in API design as a backend developer, the frontend company asked me to put all the metadata for a single entity into one API. Their reason was that it was hard for them to handle multiple requests and they'd rather do the filtering on their own side. In reality, the right design would have been zero-trust, where I only send what's necessary. But since they were a tier above me, I just went along with it.
In that sense, I wonder if Silicon Valley culture, which carries a narrative of starting small and growing into one unified whole, is why these practices are seen as universal. I personally think using JSON Schema or writing libraries is a kind of social convention, but I don't necessarily agree with it. That said, I think the OP's opinion can be summarized as: 'Scale up when you need to scale, and don't create unnecessary boundaries that don't fit your organizational structure.'
A small team moves fast, sees user feedback, and redesigns boundaries through refactoring when needed, growing the system along the way. It's cliché, but it's also the hardest part, and it varies depending on the programmer's experience.
I envy developers in Silicon Valley. The idea of owning code and being able to make these kinds of arguments feels so foreign to me.
When I deliver software, based on my experience, I just paste in the most complex template I can think of, regardless of scale. Honestly, that's a bad programming habit too. For small code, opening, writing, and closing within a single method is often enough. The key question is whether the program keeps running, so there's no need to overcomplicate with layers.
Smart programmers usually know at what scale to stop when designing. But for a copy-paste-style coder like me, who just assembles code blocks that worked well before, it's a different story. That often ends up taking more time.
Whenever I start a new project, I immediately think about error policies, validation tables, evidence tables, and so on. I struggle through them, which sometimes delays things. But reading posts like this always feels fresh.
Sometimes I wonder: am I really a programmer, or just a factory worker?