Artificial Intelligence – Scaling Debate pt. II

Scaling has pretty much run its course.

• The debate over whether current technologies will produce superintelligent machines now seems to be over, with even Sam Altman admitting that further breakthroughs are needed to get to superintelligent machines.
• At the TreeHacks event last month, Sam Altman gave a keynote where he highlighted the need for a breakthrough roughly as important as the transformer in 2017 or ImageNet in 2012 in order to keep progress going.
• This is a substantial change from his previous position which was that OpenAI is on track to achieve superintelligent machines and that the company knew how to get there.
• This was made at the time when the idea that bigger and bigger models with more and more compute would deliver performance improvements was still very much in vogue.
• What has happened subsequently is that performance improvements have proven to be asymptotic (see here) in that more and more resources need to be pumped in to get smaller increments of improvement.
• This is a sign of a technology or technique getting towards the end of its use in terms of delivering improvements, which is something that I have been expecting for some time.
• To be fair to the industry, it has been able to wring a lot more out of scaling than I thought possible, especially with the idea to have the models compute for more time (or tokens) before coming up with an answer.
• This led to scaling in size, giving way to scaling in terms of compute usage, and has been the main driver of the improvements that we have observed over the last 18 months.
• However, the improvements here are now also reaching an asymptote, meaning that something else will be needed to keep progress marching forward.
• It is at this stage that an important distinction needs to be made, which is between the drive for superintelligent machines and the use of AI by consumers and enterprises from which money can be made today.
• Although new techniques are clearly needed to achieve superintelligence, I think that AI, as it is today, is good enough to generate real increases in revenues and profits.
• The question is to whom these increases in revenues and profits are accruing, and as the model companies are currently burning through billions of dollars every quarter, it is certainly not them.
• The companies that are making money from AI today are the equipment and silicon suppliers, and the companies that are taking the models and using them to create new services or augment their existing ones.
• Hence, the value is migrating to the extreme ends of the AI technology stack, with technology providers at one end and ecosystem owners at the other.
• This is pretty much what always happens as the world of personal computing and smartphones clearly demonstrates.
• This is why OpenAI must be one of the leaders (if not the leader) in the race to develop the consumer AI ecosystem, as without a win, it will never be able to justify either its valuation of the hundreds of billions it has spent on infrastructure.
• Anthropic faces a similar challenge, but as it is focusing more heavily on the enterprise, where there is no direct challenger (like Google for OpenAI in consumer), it looks like it will have an easier time of it.
• Hence, I think that there is a substantial monetisation opportunity without super intelligent machines, but the problem is that many of these companies are being priced as if super intelligent machines are just around the corner.
• This, combined with the economics of compute ($50bn / GW of compute that lasts 5 years and generates $10bn per year), is where I see scope for a correction.
• I continue to want to be positioned at either the bottom or the top if the AI technology stack, which leads me to the memory companies which are still cheap, Nvidia, Qualcomm, Google and some of the software companies that are already quite advanced when it comes to harnessing AI in their products.