Flip the Table

The game now is building code generation pipelines. If I’m focused on that, it means that my front end developers are becoming my backend developers and the product team is building the UI. The whole software engineering hierarchy just shifted.

And if we’re building code generation pipelines then the inference spend just got even more real. Local is going to matter more and more and more for so many reason. Cost is one, compliance is another. Hospital systems are extremely security conscious. Also, financial services, insurance, legal, many of these sectors have huge regulatory overhead and keeping their inference on prem is going to be crucial for them.

At current rate, Apple is set to hit about 2 TB/s gpu bandwidth in 2-3 years, unless they have a major release this year. If they hit 2 TB/s it changes the game. That competes directly with an H100, which currently sell for around 30k, and have modest vram. You know what else they’re working on? Thunderbolt for clustering, which the mac studios can already do.

I’m telling you, Apple is going to walk in the door and flip the table everyone else has been playing at.