Your company builds a CRUD app that serves 5,000 users. Your interview asks candidates to optimize distributed systems at scale. What are we doing here?

The post details an all-too-common problem, an application that honestly doesn't need to deal with scale, but their interview process (and likely their engineering team) is caught up worried about problems they'll never have.

But it sets the tone, this is how you should be approaching problems. We build for scale, we build to be a billion dollar unicorn (which is a complete engineering misstep and frankly in my opinion: negligent), it doesn't matter that we only make $2m/yr and our growth is 9%/mo. We don't build for what we're going to do, we build for the never-will-be.

That tone carries over to the whole industry.

What ARE we doing here?

Over my 2+ decades of experience I've seen:

• Time-wasting automation: where neither time nor consistency improved (but a job to maintain those scripts was created)
• Microservices: I'm going to come out and say it, for the vast majority of companies, microservices harmed, significantly. Likely if your company isn't shoving 9+ figures of revenue through its doors, this initiative was for job-creation, if you think you'll be a 9-figure company, like every other 9-figure company, fix it when that's on your horizon, not your day-dreams.
• Mediator/Actor models/CQRS/etc: Needed in very specific circumstances, I've seen it complicate and make systems fragile and slow more often than robust and efficient.
• Kubernetes: If I'm lucky a Kubes deployment is kept simple, most of the time they're engineering it to need to orchestrate a thousand servers, no. (Expect me to publish an article about frustration that Kubes is kind of an annoying-but-best go-to, this doesn't apply to those deployments)
• Rewriting Libraries: I've seen many engineers that didn't like how something was done, didn't just sit down and tolerate it, and made a more fragile, more opinionated version that now was the company's problem to maintain.

Why Do Engineers Keep Doing This?

James Anderson from our team put a name to it: "Resume-driven development." It's pretty on the nose, but there are a handful of reasons it happens:

• Padding their resume with "accomplishments" - those accomplishments aren't actually improving your bottom line, but they'll sure improve their resume when they jump ship to a new job.
• Read too much Reddit / lobste.rs / Hacker News - A flavor of "keeping up with the Joneses", other Engineers get to work with cool new toys, I want to also!
• "This is what we did before" - Engineers basically rubber-stamping the same solution their entire career.
• Not actually engineering - A flavor of above, but to a point.

There are so many engineers that just kind of… don't think, they rubber-stamp solutions regardless of how ill-fitting they are.

So How Do We Stop This?

Most of this just comes down to better vetting and better management.

Those are honestly two entire other articles in and of themselves, stay tuned for those.

The bottom line is this: engineering is expensive… mentally - it takes a very concentrated and pragmatic approach to do well, and the best engineers I've worked with weren't the ones chasing the newest pattern or padding their resume – they were the ones solving the actual problem in front of them. Knowing what your business actually needs is what separates an engineer from someone just screwing around on company time.

So we've seen it all over social media, every engineer can be a 10x engineer now, get rid of 90% of your team, no wait software engineers are cooked, everyone can be a 10x engineer now! Actually just get rid of your engineering team!

But it's been over 3 years of this, Devin AI, the AI to "replace all devs" has been out for over 2 years, and where are we really today?

Learning Programming Languages Was Never The Barrier

I've picked up languages in a few days, I have over 15 under my belt, many engineers are thrown at languages that were developed for specific platforms, shoot I've seen projects where all-too-silly engineers developed their own language then used it at work.

Zech Sloan was nearly hired a few job placements ago to lead a Ruby team - zero Ruby experience, the founder wanted his technical acumen that badly, the language didn't matter, the skill to make well informed and solid decisions did. Zech turned down the offer.

At the end of the day, learning how to write an if statement wasn't the big barrier that AI influencers want you to believe. What is basically being said is "oh look, now people who can't be bothered to put in a little bit of effort can do this too!", that's not a good thing.

10x Development!

Are engineers hitting 10x productivity? No not really, we've seen a lot of projects still crawl, releases aren't that frequent, road map is still way behind schedule, us as users, owners, or investors are left still just waiting.

We're seeing no renaissance of software - have we seen 20 years of software development in the past 2? No, not even close. If anything we're seeing an unprecedented slump in software quality. Windows had one of the worst rollouts since Vista, and Vista had an excuse! AWS has gone down at least twice due to AI-assisted deployment changes. Cloudflare's AI-driven Bot Management system took down a chunk of the internet. These kinds of incidents during the "Boring Cloud" last decade would get people fired.

We do see some software building fast, stuff like OpenHamClock is a decent example, but I ended up not using it because it kept changing frequently, branched from the original app it replaced (functionality wise), and kept adding so many random things it made what use I got out of it less useful.

OpenClaw is a new flavor of abandonware: endless issues generated by bots, bots endlessly PRing against broken builds, activity everywhere and care nowhere, the bots just make the project worse continuously, this is the 10x you get.

Have Non-Engineers Write Code!

I love how I'm watching the flavor-of-the-month tell teams to do this, guys we just came out of low/no-code. We just experienced what happens. Engineers aren't just English -> Code translators, a good team pushes back, a good team understands the industry, a good team guides product development to be better.

We have seen zero people handle this for anything non-trivial. No/low code has always had a very important role, are you a mom and pop shop that needs something that's better than an Excel spreadsheet? Low/No code works great for this, when your budget is $100 and your own time. I think Vibe coding still sits in this comfortably, can you reach further? Yes. Will it bite you? Even worse than old low/no code did.

I personally have had multiple projects come to me from individuals taking the bait on this, usually the story is the same: "this is a nightmare to work with, I need [quote a 5-figure number]", what people don't understand is a few very clear problems:

1. As a non-engineer these platforms will let you do terrible things
2. They have a tendency to make a mess of the code base by default
3. They're immensely fragile, usually that's why you've reached out, because you've gotten to the point where you can't fix anything without breaking something else
4. Me re-architecting this is a lot of work, not only time wise, but it's stressful and some of the worst work to do, that'll come at a premium

We Were Never Short of Work To Do

And the biggest point, I've never, in over two decades of professional software engineering, ever been close to not having a backlog, not only that, AI allows us to consider SO MUCH MORE on our backlog.

• Need a tool to do something? Build it, the cost to do things manually instead of having this tool has compressed greatly.
• Would a UI flow catered to this specific flow be better than using the already-existing flow? Do it, it's a better UX, cost is no longer the major barrier for these little UX improvements.
• Want to reconsider a UI reflow because we've been adding features? The cost of prototyping it just fell through the floor, see if anything sticks.
• Quality of life improvement? A must-have now. I have more flexibility to wager if we can fit it into the pipeline.
• We need to refactor this code? I can prototype those refactors quickly, for trivial-but-heavy moves (lots of code but easy approach) LLMs will make quick work.

No, Tests Don't Save You

I don't care what anybody says, I've watched multiple teams do this, and I've experienced it myself multiple times, on every project without manual review and correction the same thing holds true: Claude will lie. Claude will build tests that are useless, it'll inflate that test count quickly, but your software will still be amazingly fragile.

• I've had Claude generate hundreds of tests, we went from 30 -> 230 tests in a week, confirming payload parsing was good, real-world testing showed 25% error rate, I had to go in and very surgically build tests and fixes, reduced it to <3% (packets outside of spec, so honestly effectively a 0% error rate). Those 200 tests dedicated to packet parsing were checking garbage.
  • We had a documented specification
  • We had over 25,000 example packets
  • Claude was given clear instructions multiple times, including leveraging the packet dumps to find parsing errors
  • Claude still helped - but required significant hand-holding to get it to actually do what we wanted it to and many iterations of useless tests
  • I still don't entirely trust the new test suite now, how many of those 200 tests are redundant? How many could I trim and lose nothing?
• One of our colleagues gave Claude the job of generating regression tests for a major refactor they were doing, they had to fight with it a lot to keep the tests on-track
• I've had to argue with Claude to not create overly-fragile tests.
• We had worked on a data issue for one of our platforms, where we had Claude write a test to confirm a bug was fixed - we wanted Claude to check two reports against each other, explicitly checking the report results, but it decided to write a 40 line internal-gut-checking test instead of the 5 lines of "did this function output match this expected value" (it also ended up not doing it right and passing when it should have failed, resulting in a bug in production persisting)

You can always roll around and say "well you have to…", no, over all the projects I've seen, everyone is doing this wrong, and those telling me I need to hold it sideways? I bet they're doing it wrong too.

The People Saying It Has a Low Error Rate Are Insane (and bad at their jobs)

This comes in from the test portion above too, so I get a number of people, usually people that are coding with mostly vibes, who insist Claude isn't making mistakes – now this doesn't pass a simple logic sniff test (most engineers don't pass one in general), how do you validate something that you don't spend the time to validate because you "trust" it?

Well I have it test itself

That… is stupid, you have to assume the tests it's generating are good to begin with.

The problem is we do check, and we do correct, A LOT.

Well you need to make agent personas

OK, see that's the thing, I did that, I spent a bunch of time going down that path, building a team to handle some basic tasks. I gave it a task to do, it eats 70% of my tokens for the day (OK it was a large task), I review the output, out of like a 20-point list, assumption 2, a basic assumption, was massively wrong. Like "it didn't even try" wrong.

I'm not spending those tokens, that downtime, on that. Sure if it isn't your money, your time, your chair spins while your employer pays you, have at. It's my time, my money, my investments.

This behavior is a known psychological concept: Automation bias. It's people's tendency to believe the computer, we've seen it a lot recently - police and hotel staff trusting AI algorithms that are plain-as-day misidentifying a person. We've seen it in data analytics, a bad number is spit out and the majority of people will trust it instead of saying "this doesn't look right…". Machine learning is super nasty in a way people aren't used to, it has a very atypical state when it "isn't working" – when a program fails to run, we usually expect it to crash or error, ML algorithms will just display bad data basically every time. Its failure state isn't a nice pop-up error message, it just lies.

But the Job Market

Job markets are complex and multi-faceted, we're coming off the heels of COVID era ridiculousness, people coming out of high school saw "hey, get a 6 figure job as your first job", those are the fresh people hitting the market today as juniors.

That was not sustainable, nothing about that was, people predicted "this is the new norm", none of it was (keep that in mind when the same people are predicting AI too).

• We have a significant number of tourists; people who are effectively only here for a 9-5 job and couldn't care less, they always underperform people with a passion for the work by a significant margin
• Companies (especially startups) practiced IT-heavy hiring, effectively over-hiring because big IT teams always seemed impressive (though more likely a dysfunctional team)
• The end of low interest rates reduced the available cash to be thrown around (almost for free) for investments
• There is a bit of "follow the leader" on seeing that you can cut your IT team and "totally not have a problem" (we saw this start with Elon Musk and X, the main fallout I was worried about: people thinking X wasn't in a unique position of gross over-hiring)
• There is some flow back to outsourcing, mostly to low bidders, we know what that gets us in the long-run (it's why we keep ebb and flowing back and forth)

The above isn't AI, AI has been an excuse to try to keep investors happy, vs. "we're preparing for not-so-great performance soon".

AI Doesn't Leave Us Less Busy

Maybe for some, who see it as a 15 minute break 10 times a day, for me even if I'm chugging on something big, I'm spinning around to do a bunch of smaller "light-duty" work that wasn't possible to fit into budgets before. I'm constantly busy, with a backlog we still cannot get all done. I am still 100% budget constrained, and a lot of this work on our backlog is revenue generating, or cost cutting. It's stuff with tangible business value.

And I've never had it any other way in 2 decades, I don't feel AI is changing that.

Mantle (heymantle.com) is a SaaS product that sits on top of Shopify's App Billing for managing your app's payments and general sales/marketing flow. It's something I've eyeballed off and on between when I was working on Shoplift and our own platform Raptify. It helps manage all of your Shopify payments, your contacts, set up tiers and discounts and usage tracking and all that good stuff, sounds great right? This all takes significant time to set up…

So… Why Didn't We Use It?

Initially we shot it down because of requirements (and we ended up building out before Mantle was even a thing), but with Raptify we were much more aligned: we needed stronger usage-based billing, we wanted to pull all feature sets from billing information, and I had zero sales-pipeline requirements beyond "put money in my bank account". Wonderful, the gaps are not a problem, and the benefits are great… or so I thought.

So we heavily developed with Mantle in mind, wiring up to their tooling, manually loading our plan data, testing, validating. Things were going sort of well minus one major roadblock.

I had no good way to keep our 6 environments synced plan wise (staging + hotfix + production + 3 engineers having their own), this basically meant we were chasing a lot of bugs that boiled down to poor plan configuration in Mantle. Surely we have a way to keep this synced in Mantle right?

Nope, well – what about their API? Nope, couldn't do what we needed.

This missing functionality was going to be a deal-breaker for us, we've already spent a significant amount of QA time chasing bugs that were configuration mis-matches, doing it manually was a huge time sink and would be error prone…

When AI Focus Kills Your Product

So I asked – "How can we keep these in sync? Or have any sort of copy ability? Is this on the roadmap?"

No we don't support that, but maybe you can use our MCP to do that?

What? That had been their response to a lot of people lately, I had noticed there'd been a significant dip in focus on maturing the platform and going all-in on messing with their MCP a lot (at least from my perspective), most of their updates seemed about it, a lot of answers were "just use the MCP" without clarity on if the MCP provided the additional exposure to actually do the things we need, consistently and reliably enough and without additional overhead coaxing the LLM to do what we need…

Should we have just used the MCP? It's a deterministic flow we need – so I'm iffy on that one at best, and the first thing we're seeing get knee-capped in the world of AI is automated flows. Human-in-the-loop at least is probably going to be the last that gets priced to the moon.

Want Me To Use AI That Much? We'll Just Use It To Replace Your Platform

Ultimately, even with Claude, their platform would have been worth the money… if it didn't incur additional overhead to do what we needed.

I'm extremely familiar with many payment platforms, we even built our own back at my OrthoBanc days with Zech and interfaced directly with banks. Stripe being one of my favorite due to a really polished developer experience and a pretty good business ops experience, is an easy choice. I'm familiar with the pitfalls of Shopify's billing system (at least a lot of them), so I draft up a plan and get to work with Claude.

It's maybe a weeks worth of work if even that, and we have exactly what I want: Raptify supports Stripe for standalone accounts and Shopify for in-store customers. One platform, two payment providers, Plan data is internal and seeded for devs and test environments, editable in production, supports all the extra bells and whistles, and I don't need anything else fancy because remember: baseline payment requirements is "I just need to get paid", I'm not going to try to do weird unreliable projections based on Shopify's weird payment behaviors.

So… I Let Out A Sigh Of Relief

Today, we get this announcement: https://docs.heymantle.com/migrating-off-mantle/wind-down – Mantle is shutting down, end of services is in September.

I still had questioned if I made the right call, even with Claude, code costs money to maintain (even without the looming AI cost increases popping up), but with Mantle shutting down, I have no choice but to admit: we made the right choice. Porting off of this would have been a nightmare (especially in 90 days) and I feel sorry for those having to deal with this.

And… There is a Void

On the Shopify Dev Discord TeamDijon pointed out:

This causes a void
Alternatives will surface sooner rather than later

And I'm torn on the viability of filling that void. Why did Mantle decide to shut down? Did they know of some lingering change on Shopify's horizon? Did the business model not pan out to be worth the effort? Were there just greener pastures their team was drawn to? Hard to tell right now…

We'll see, but I don't feel those platforms will really be on our radar.

They're Not The Only Ones

I feel I would have stuck around if they leveraged AI to improve their platform instead of trying to add the latest AI-tooling bells-and-whistles to their product. They're not the only product that I feel ignores the thing that is actively making them money, and the vertical that could continue to make more money, to play with the hottest new fun tech stack (don't worry, we have an article about how Engineers get distracted with fun toys at the cost of your company coming up soon).

But without further ado.

Yes Just Use Postgres

The idea to write this came from a Reddit post: Why does PostgreSQL + .NET feel so much better than SQL Server these days?

Here at RoushTech we average shipping a new mainline market product about every 2 years, with smaller products dotted around, it's just how the work comes in, but regardless when it comes down to database choice we always "evaluate what our options are", then we choose Postgres, we really should cut out the middle man on this.

What About Other Engines?

OK, again keeping this fair and sticking to RDBMS platforms, let's talk about our options:

• Microsoft SQL Server - Okay, I'm going to be honest, I love MSSQL… until I have to license it. It's an extremely powerful engine, a good set of tools, some really nice failover options. Two major downsides: we've seen massive performance degradation when running on Linux due to how Microsoft got it to actually run on Linux, and secondly, the biggest one… Licensing. In the past we'd easily drop thousands or tens of thousands on licensing. It's become a major barrier. 2 months of development for an MVP or your SQL server license?
• MySQL/MariaDB - A very outdated engine, InnoDB hasn't seen meaningful improvements in forever, Oracle is effectively doing nothing with it (and no, MariaDB isn't different enough to warrant its own line), and my favorite deal-breaker in why we almost never run it. ALTER TABLE STATEMENTS ARE IMPLICIT COMMITS, that's right! When your software is running a migration, if it fails, it'll leave the DB in an invalid state because every table alter breaks out of your transaction! That's great!
• Oracle - Did you look at Microsoft's license and determine you were a glutton for punishment beyond that?
• [Insert Scale-Out DB Here] - Platforms like CockroachDB are cool, but scale-out DBs are a completely different class, complexities change a lot, design options are limited, latency floors are usually pretty high.
• SQLite - OK I love SQLite but only really great for local/embedded DBs.

OK But How Does Postgres Stack Up?

This is the wonderful part, and some of the really fun parts people miss out on:

• It's extremely performant in a lot of cases, we've shoved a ridiculous amount of traffic through it.
• You can use it as a doc store! Yeah, okay, I usually don't opt for using engines "incorrectly" but Postgres has come a long way, the ability to dig and index into JSON columns is great.
• You can beat the everliving snot out of an RDBMS if you're a half-decent engineer. We've had single-instance Postgres systems doing tens of thousands of non-trivial transactions per second, and yes, that's even after someone trots out the "well this key-value or document store can do 400k/sec" comparison everyone loves, sure, but those 400k are trivial ops on a single key or document. One non-trivial Postgres transaction is doing joins, constraint checks, and atomic writes across multiple tables. They're not the same unit and the comparison only flatters them if you pretend they are.
• Has a good amount of tuning options allowing you some decent influence on performance.
• Has a healthy extension market with a lot of additional bells-and-whistles you can add (PostGIS for spatial, pgvector for embeddings/RAG, TimescaleDB for time-series).
• The MVCC system is complex but powerful.

Where Am I Going To Get Bit?

• I've had the query planner lie to me about costs, just straight up lie, "this is cheap" then thread-spins.
• Like any RDBMS, beware of locking, a sloppy ALTER TABLE statement may be the difference between something that takes 100ms and 100 minutes (or longer). Learn how the engine allocates data.
• There are many ways to set up clusters, each with their own pros and cons.
• Postgres uses a process-per-connection model (every client connection forks its own backend OS process), which gets expensive at scale. PGBouncer is your friend, and your enemy. There is no free lunch with connection limits and port limitations, PGBouncer can make these logistics easier, but port exhaustion is still a thing.
• If you need something truly scale-out, it likely isn't here, but you should also be making $100+m/yr, why are you even here?
• Postgres is case-sensitive by default, unlike basically every other DB engine on the face of the planet. Configure case-insensitive collations on day one or you'll be patching application bugs forever.
• Run VACUUM FULL, you'll have a good time (insert massive amounts of sarcasm here). Imagine a hard drive defrag from back in the day but it locks your hard drive while doing so. MSSQL got online index rebuilds right at least, at least if you're paying for Enterprise.
• Major version upgrades can be annoying, not super bad though.

I Hate To Say It…

But yeah, just use Postgres, our options have narrowed over the years, and I agree, it's annoying, I almost feel like I'm not doing my job, but Microsoft's licensing and Oracle's lack of caring for MySQL did that for me.

So with the new site comes some major new changes, we have some new pages better describing what we do, we have some regional pages up and we'll probably have more in the future, we have a list of our projects publicly (we used to have a hidden portfolio a couple of sites ago)

One of the major changes: public pricing

Okay, but public pricing?

Along with us using Claude with a lot of our development, Claude has helped a bit with this site – Claude constantly complained "Don't put up pricing", "it doesn't allow you to drive higher margin if people know what you charge"

Yes Claude that's the point, we're not like every dev house where your pricing fits your business size (oh, you got lots of money, guess our hourly rate went up…). We work backwards from what we pay our engineers, since that's the basis that we have to clear as a business, we put a few levels of margin on top of that which represent our different discount tiers, and that's it. Nothing else special.

I mean Exhibit A of our MSA that shows our rates is basically the same for everybody (there are some edge cases where people get some slightly tweaked rate sheets but mostly driven by stability of the contract).

Weird, so why?

There is a significant number of customers that appreciate that we just shoot straight, someone asks what we think something will cost, we'll throw some numbers based on our experience, not based on me having to come back and figure out what we can squeeze them for, just honest off-the-cuff assessments of what they're trying to do.

Of course we're generally in agreement: I'm giving examples, details can drive differences, but I get a lot of "I can't get a vendor to just shoot straight with me, Will"… well we should really do that up-front too. If the pricing makes you a bit skittish, if you're unsure of how or why we do things, talk to us, I'm more than happy to discuss how this makes sense.

Or hang around for more articles. Along with industry insights, I'm planning to explain a lot of the ethos behind why we do what we do and where these systems and theories come from, some maybe more hot takes than others…

See ya around!