I'd argue that the brilliance of SpaceX is the opposite. They stick to technology and markets that are proven and use technically conservative approaches. Falcon 9 is about relentless improvement in small ways, not bold new ideas -- unless you count not getting caught up in the politics and psychology of bold new ideas as a bold new idea.

Sure, they talk about Mars, and in-space refueling seems radical, but they've yet to succeed at doing anything radical... yet.

Rumor has it they were struggling with the payload fraction w/ the first generation of Starship and they switched to a second generation that struggles with blowing up. A big advantage of the two-stage architecture is that you can develop the two stages independently. Presumably they will eventually get Starship to orbit and bring it home, they will have plenty of time to improve it get the payload fraction up just as they did with F9.

SpaceX is very close to demonstrating an architecture that ameliorates almost all of the drawbacks of two stage to orbit architectures. The tyranny of the rocket equation ensures that while a SSTO carrying all of it's oxygen is possible, it's never going to be able to carry enough mass to be useful.

Therefore nobody is ever going to invest the tens of billions required to develop a rocket based SSTO.

If somebody develops an engine that makes air breathing most of the way to orbit feasible, this has a chance of competing a Starship style architecture.

For the reasons you espoused, this is highly unlikely. However "highly unlikely" is more likely than "never".

Jet engines have on the order of 10x the specific impulse of a chemical rocket.

Atmospheric density reduces exponentially with altitude, which implies that you would need to go exponentially faster to maintain mass flow into your engines and lift over your wings. The truth is that breathing air only gets you a third of the way to space, at best, so you have to have a rocket, and now you're battling that complexity. If your space plane doesn't breathe air, it probably is just better to punch your way out the way conventional rockets do.

Of course, the rocket equation is logarithmic, so reducing the amount of mass you loft gives you an exponential gain. This is true for all propulsion systems to an extent (different constants) but getting into space is the hardest propulsion problem we face. A space plane may or may not be better in this regard (it's been a while since I've looked into that kind of thing, so no opinion either way) but imo the inherent complexity is enough on its own to kill the idea.

There are two performance parameters for a rocket/jet engine. The first is thrust and the second is specific impulse. You are thinking about thrust. The others in the tread are talking about specific impulse. Thrust is important for some stages, especially the early stage booster engines (as opposed to later stage sustainer engines). As a simple example, any space rocket will need a first stage with an enormous thrust so that it can lift itself, the subsequent stages and the payload off the launch pad. Additionally, the rocket has to finish its initial vertical climb as fast as possible. Otherwise the propellants will be wasted in just lifting off (this is called gravity loss). That will also require a high initial thrust.

However, the requirement of the high thrust disappears once you finish the vertical climb. There's no danger of falling back to ground once you reach orbit. What you need at this stage instead, is to add velocity (deltav) to the craft to change its orbit/trajectory. This can be done even at very low thrust, because you have all the time you need. The limiting factor now is that you have only a finite amount of propellant onboard. You want to add as much deltav as possible before it runs out. A high thrust doesn't help because the engine will simply consume the propellant faster and exhaust it before you get the required deltav. This is where specific impulse comes into play. The maximum deltav you can get is proportional to the specific impulse of the engine (see rocket equation for details). As you can imagine, high specific impulse is critical for space missions requiring high deltav, like the New Horizons spacecraft that imaged Pluto or the Parker solar probe (interestingly, getting to the sun is harder than escaping the solar system). Rockets/jets with low thrust and high specific impulse are called sustainers.

The general trend seen is that specific impulse drops off as thrust increases. For example, the space shuttle solid booster has Tmax = 15 MN, Isp = 268s, and space shuttle orbiter cryogenic engine RS25 has Tmax = 2.28 MN, Isp = 452s. Meanwhile, the NEXT xenon ion thruster used in the DART mission has Tmax = 236 mN and Isp = 4200s. Note that the thrust has changed from Mega newtons to milli newtons. You would hardly recognize it if the ion engine thrusted against your body.

I don't think any of the considerations you mention there change my conclusion.

We have to ask: what exactly is a scramjet vehicle delivering? It's enabling the use of air instead of liquid oxygen. But how valuable is this? LOX is the second cheapest industrial liquid after water. The fuel part of a rocket propellant combination typically dominates the propellant cost. If a scramjet launcher uses more fuel (especially hydrogen) than a rocket vehicle would, it will end up increasing propellant cost per unit payload to orbit. It will also likely increase propellant volume per unit payload to orbit, especially if LH2 is used (LH2 being just 5% of the density of LOX).

All scramjet launchers need a rocket to reach stable orbit (since a scramjet cannot produce thrust at apogee to circularize above the atmosphere. So one can ask, what the tradeoff between the delta-V this rocket provides and that of the scramjets? From what I've heard, all such trade studies end up optimizing to 100% rocket and 0% scramjet.

Everybody may have their own criteria of preferences, so the same solution with fail for one and succeed for another.