Similar experience here. About 6 months ago I inherited a team of 30 that had spent 4 years building some monstrous microservice "thing" that didn't work and never seemed to build reliably and barely worked even if it could be deployed. Lots and lots of late nights trying to "deploy to testing" and wasted weekends restarting things.
I stopped the work, broke the team up (and let a few of them go), re-assigned the goals of the project to 3 people who, working half-time, from scratch, and in Python, rebuilt the entire thing in 3 months, delivered it and it worked (and continues to work). "Builds" are now an scp to a VM, and a restart of a flask application.
My wife recalls a similar experience where a complex application she worked on (and ran reliably on very little infrastructure and with an entire team of 4), and serviced thousands of simultaneous users, was replaced (when a new CTO came in) by a giant, consultant driven Java project that required approximately 6x the infrastructure, 3x the staff to keep it going, and at launch could only serve 9 to 12 users at once. It was a debacle. But sunk cost fallacy forced her company to throw millions of dollars of consultants at it until it achieved some minimal level of barely available service. She and her boss left around that time.
She heard that a couple years later the CTO was let go, the consultants fired and it was all replaced by a team of 5 rewriting it all again from scratch (in Java) but like you also without all the architecture barf dropped in from orbit, and immediately went back to servicing the thousands of users again.
I'm convinced that this is the fault of some kind of consultant industry that seems to infest enterprise software circles that is really only good at inventing ways of designing systems that require more of their services, but never seem to actually deliver working systems.
As someone who fell into a similar situation just last year, I can say that it helps a lot when you have previous work that had been done before you were involved. Perhaps the existing project doesn't show you how things should have been done, but it certainly shows you some of the challenges that were faced and what not to do about them so you don't fall into the same traps as they did.
So it's not always "those architecture astronaut guys used the company's resources to study their webscale fantasies and I came to save the day", but rather, "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight".
I agree with what you say here to a large degree. We were able to reuse things like external service connections, accounts, contracts, some of the requirements documentation and some of the already provisioned hardware, VMs and so on. Those things are not trivial to put together and easily saved us several months of work.
Where I think I part ways with you is with this "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight". One of the big lessons we've learned as an industry involved in software development is to start small, iterate often, get feedback from users. What I see time and again with these types of overblown space elevator projects is a kind of fundamental...immaturity -- the people who run these don't seem to understand how to achieve results with the minimum required to do it.
They've never started small and iterated to large. The projects they've worked on have all been so enormous, and have taken so long, that they have very few data points to draw lessons from. Each iteration and growth cycle in a mature project creates lots of information that can be reapplied elsewhere. But for people who've only ever grown up in large enterprise software projects, that's the only approach they know, and there's only a handful of lessons they've learned.
It's not only architecture astronauts, but the entire ecosystem that supports "enterprise" software engineering: vendors, consultants, scaled agile experts, design tools -- even university programs that churn out enterprise ready engineers. The appearance of these things, and how widespread they are in certain circles, demonstrates how those circles have regressed and tossed into the rubbish bin the lessons that we've learned. The penalty that's being paid is that these same lessons have to be continuously relearned again and again, but instead of pushing against outside models of how to do things (e.g. trying to adopt physical engineering approaches to software), we have to push against our own industry.
When you go against billion dollar enterprise software industries, you end up sounding like a heretic. It's really only when you get a long portfolio of success stories can you succeed. But that's very hard to get in today's climate where people spend their entire careers trying to kill ants with nuclear weapons dropped from the orbit of Mars.
This was very well put and definitely agreed with many of the sentiments you've put forth about bottom-up (get up and running with as little rpc as possible) vs sideways development (copy the entire netflix architecture and try to make it work)
I just didn't put the blame on the previous team, maybe I should have for sticking with it for two entire iterations, still not entirely convinced. I think they couldn't help it and the decisions were already made for them by large faceless corporations. I guess what's the point of calling themselves engineers if they don't feel like they are in a cockpit pressing lots of buttons right from the start?
By profession, I call myself a computer programmer, not an engineer or architect in order to remind myself that I should weigh problems by starting from fundamentals such as a state machine, an integer constraint system or a breadth first search, before setting up service discovery or training neural networks.
We had a similar debacle back during the dot-com boom. We were a Microsoft shop and we got a new CTO who despised our monoculture and demanded a shootout against Java. A large team of consultants and employees labored for almost 10 months to produce a system that scaled to .. 2 simultaneous users while running on the latest Compaq servers with tons of memory. Meanwhile our legacy VB6 app was serving 1600+ users on mid-tier hardware.
I stopped the work, broke the team up (and let a few of them go), re-assigned the goals of the project to 3 people who, working half-time, from scratch, and in Python, rebuilt the entire thing in 3 months, delivered it and it worked (and continues to work). "Builds" are now an scp to a VM, and a restart of a flask application.
My wife recalls a similar experience where a complex application she worked on (and ran reliably on very little infrastructure and with an entire team of 4), and serviced thousands of simultaneous users, was replaced (when a new CTO came in) by a giant, consultant driven Java project that required approximately 6x the infrastructure, 3x the staff to keep it going, and at launch could only serve 9 to 12 users at once. It was a debacle. But sunk cost fallacy forced her company to throw millions of dollars of consultants at it until it achieved some minimal level of barely available service. She and her boss left around that time.
She heard that a couple years later the CTO was let go, the consultants fired and it was all replaced by a team of 5 rewriting it all again from scratch (in Java) but like you also without all the architecture barf dropped in from orbit, and immediately went back to servicing the thousands of users again.
I'm convinced that this is the fault of some kind of consultant industry that seems to infest enterprise software circles that is really only good at inventing ways of designing systems that require more of their services, but never seem to actually deliver working systems.