Artificial Intelligence – Whoops Apocalypse!

Claude makes the case for ServiceNow, Salesforce & Co.

• A cataclysmic blunder by car rental SaaS company PocketOS led to Claude deleting the entirety of the company’s product database, once again underlining that AI agents can be very dumb and need complete supervision.
• In its quest for AI perfection, an AI agent powered by Claude was given complete access to the PocketOS product database, which it decided to delete and then went on to delete all of the backups as well.
• The strangest thing about this is that the agent cited the “do not delete” policy as part of its “thinking” process, but then went on to completely ignore it.
• This is yet another example of the fact that while AI can give the impression of sentient rationality, these systems still have no understanding of causality and can make the most basic mistakes.
• Furthermore, despite the machines spitting out text about what they are doing and why they are doing it, in some cases, this is complete nonsense.
• This calls into question whether what these machines spit out when they are working on a request is actually real or just a probability approximation designed to give the impression of real thinking.
• The wires have been full of blame being batted back and forth between Claude and Railway (its infrastructure provider), but very few people have realised that only one entity is at fault here, which is the company itself.
• I continue to hold the position that no rational executive is going to give an AI-agent root access to their systems and then just let it get on with the job at hand.
• PocketOS is a prime example of what happens when one places too much faith in a machine.
• Instead, these agents need to be managed and supervised in everything that they do.
• This is where ServiceNow and Salesforce come in, as they have spent the last few years migrating their products to deal with this problem.
• If successful, this will make sure that they benefit from the AI wave rather than becoming a victim of it.
• I put Adobe in the same category, although the use case is a bit different, as Adobe allows extreme control of image generation by whatever model one wishes, as well as copyright indemnification.
• However, the market doesn’t care and has decided to cut the valuation of everyone in the sector in half, regardless of any plan or strategy to mitigate or even benefit from the impact of AI.
• Consequently, I continue to believe that some of the sector will thrive as a result of this trend rather than be destroyed by it and as a result, there are bargains to be had.
• ServiceNow came out swinging yesterday, predicting that revenues would hit $30bn in 2030 (14% above consensus of $26.3bn) with steady gross margins and about $200m saved from AI-related OPEX optimisations.
• ServiceNow also has an event this week where I am expecting a polished and confident performance, but I doubt that the market will care that much.
• I am sure it will highlight PocketOS as an example of what happens when one abandons controls and safeguards, and I suspect that the phones at ServiceNow and Salesforce are going to be ringing with greater frequency.
• I am pretty sure that this is what underpins ServiceNow’s confidence to upgrade its medium to long-term targets.
• In the entire sector, I continue to prefer ServiceNow and Adobe, and I have a position in both where I think 100% upside is quite a reasonable expectation.
• However, this is not without risk as these companies must continue to execute, and there is no sign of the market changing its mind anytime soon, and so the sector could easily languish or worst case, go down some more before there is any recovery.