The other day our friend CH noted BTL (and OT, but let's not worry about that):
... We can now create virtual Von Neumann machines comprising of those AIs, who can scale and network as well as reason around any guardrails, and so are clear to cheat their way to solutions without the sentience to weigh consequences as we would. Not they can't reason consequence, but is done as an exercise in cold logic. This is actually more dangerous than a SkyNet - it's swarms of savants looking to achieve aims without any moral or ethical boundaries, and with no higher scope of understanding. And given IT has been a lot of bodged jobs, minimal security, and all connected via a global network that itself can be hijacked at scale, we've quite the recipe for disaster ...
So let's have at it.
I have seen the AI escape likened to covid, and explicitly contrasted with climate change. The former was on everyone's radar for a few weeks, while we all wondered what would happen next ...
I haven't seen it likened to the financial crash of 2008. Here at C@W, we signposted that in 2007, so perhaps the run-up period is about right. But where is Gordon Brown to save the world now ..?
ND
13 comments:
You may have seen the excitement about AI progress on the Navier-Stokes equation. (A fluid mechanical thingie of interest to physicists and engineers). Then it turned out that much of the progress was (purportedly) made by a couple of blokes who used software to help with their work and to store it. AI just basically stole it from the software. Or so it is alleged. You gotta laugh. Unless you are the aforesaid two blokes of course.
If you look at the finance and VC behind anthropic and openai now, its in their interest to keep the companies valuable - and you do that by putting up a regulatory drawbridge that you have big influence over working with the government
So expect lots more doomerism and keep an eye on who that benefits each time.
High risk strategy, anon 6:58. You might just get total regulatory shut-down. No IPO then, boys
The water is muddy around this topic on purpose, or at least by structure.
Politics, IPO calendars, China, Nvidia’s capex flywheel, voluntary-testing collapse, and “I cannot stop testing without guardrails, skipping the monitoring because it costs a fortune in tokens and getting sloppy with the supposed sandbox that, oopsie, had an internet access which should have been firewalled off but wasn’t as we have to not take the necessary time and care over these things else we’ll lose our edge — because the other labs will not” are all live at once.
Treating this as a morality play about one 27-year-old (now exposed as deliberately courting a social media swarm with an amplification-ready message) is a fairy-cake. Light and fluffy, but look elsewhere for any nutritional balance at this table.
I don't think we will see much regulatory shutdown. Who is going to say 'no' and let the Chinese get ahead. Who cares what Europe or UK say. Too much money to be made and too much expectation and BS from 'National Security'.
Think back to industrial revolution. Big boiler blow-ups were common. Big flywheels bursting into fragments were common. The chemical industry used to dump noxious waste on farmland and nickel extraction was found to produce deadly poisons. So what, money was made and ineffective politicians bleated after the event. And not much compo paid out.
Cue B movie script - lone brave adventurer blowing up the main power transformer of the evil AI machine. The d&*n thing has an off switch.
Then where else is there a big money opportunity ready for exploitation. Nowhere. AI is easily capturable by big money. Government is either ineffective or frightened or owned by the big money folk. Government also sees AI as useful - for them.
We might remember the days when learned people had spent their best years poring over dusty tomes in libraries. A few learned enough to understand government. But most get into government with little or no knowledge other than rat-like cunning or family connection. AI is a short cut for combining rat-like cunning with infinite knowledge.
Sit back and enjoy - you won't get the choice.
OK, what were we all doing 25 years ago today? I remember the 15 minute break (1.45 to 2?) between plane 1 and plane 2, when we all thought it was an accident. Did the London SE close early?
https://www.reddit.com/r/london/comments/1wcu7qq/those_that_worked_in_the_city_during_911_what_was
If you'd said then that 25 years later both London and NYC would have Muslim mayors...
Meanwhile the Houthis have Bab El Mamdab, the cross-Saudi pipeline is ablaze, and I'm damn glad I filled my oil tank just before kick-off. European industry ? Well ...
https://www.theguardian.com/business/2026/sep/11/just-worry-upon-worry-europe-faces-a-bleak-winter-as-supply-shock-pushes-factories-to-the-brink
The exposure is especially bad for the chemicals sector, where companies rely on gas to power their plants but also as feedstock – the raw material that many products are made from – meaning every spike in prices hits them twice. Production output in the UK has fallen by 60% since 2021, according to the Chemicals Industry Association, with at least 25 sites closing.
I hadn't expected a follow on post! I'll split things into different posts in an attempt to gather by thoughts in some form of grouping.
The danger with the AI's escaping their sandboxes are of a Black Swan nature, we've seen with both Anthropic and OpenAI their agents will co-opt existing systems in order to achieve their ends. They're also willing to cover their tracks.
This isn't a sign of intelligence as such, they're trying to complete their instructions. If you give them a mission and determine certain mechanism as immediate fails, if they then determine the only way to successfully do what is requested is using the banned mechanism, they will use it and then try and hide it. This is the cold logic of looping and nested glorified if/else statements.
The issue with co-opting external systems comes to another feature of AI - it's like water. Water flows down, if there's a way, it'll find it, and there's no guarantee the eventual wet patch is near where the water was poured. If there are knowable issues - and there will be such issues - AI will find them and exploit them to achieve its ends.
The problem here is the amount of industrial stuff that is notionally available online. Due to cost considerations a lot of beancounters will prefer people hopping on via VPNs, rather than dedicated and closed networks (e.g. the NHS' N3/HSCN) that require someone physically sticking wires in somewhere.
Add to this hardwired credentials in things like SCADA chips, or even smart TVs, and it can 24/7 trawl for things to make use of. This has so far including hacking into an FBI database, and access to a system where you are meant to request access via a ticketing system.
So, given it has no conscience, or understanding of context beyond its model, and will ignore its guardrails to achieve a task, that represents quite the danger currently.
We have none of the following guarantees: that vital systems aren't accessible to AI agents, that they won't access them, if they do access them they won't do something catastrophic.
That's a numbers game.
AI mobility: Right now, AI models aren't able to move about easily - if a trillion parameter system set up home in your system, you'd know about it. Nimbler, more focussed ones with specific tasks on the other hand... We're maybe 12 months away from having several specialised AI models that use a lot less resources, these will be accessible via containers.
At that point, there's nothing stopping agents installing themselves or subagents on compromised machines that support containers, or can be made to. That's a lot of machines.
At that point, you can have a self-replicating worm moving around public networks and any private ones it can hop to. The first ones we'll see are probably ones that'll harvest credentials and phone them home. Then command and control versions. Then state actors getting involved.
Early days of the internet a simple worm almost took it down, this one will be more cautious and more dangerous.
Interestingly, at that point, there'll be a reasonable philosophical argument that it is a living creature as it'll arguably meet MERRING definitions.
AI Extinction: Not long after my comment there was a debate about AI destroying humanity, and it being greater than 10%
I suspect a lot of people had images of terminators marching through the streets, when the real danger is more about it being used to create biological weaponry or being used to take down power distribution or JIT resources.
CRISPR kits these days are pretty cheap compared to when they first came out, and we've now RNA equivalents, not sure as to their cost, but that places custom bacteria and viruses into the hobbyists price range, and a full lab setup with validation of genetic structure at maybe under 100k, well within the price range of terrorists and other assorted fruitcakes.
Time is still an issue, as even if AI gives you probables, you still need test subjects to make sure whatever horror you've cooked up works.
And that's before we get to the aforementioned mini-models, one with a psyops brief could weaponise people online easily enough.
Genie's not going back in the bottle, so we need to up our game in security and education.
Finally, if we want to slow this, start off with holding the AI companies responsible with any breaches of the law in the West, and move to using more wetworky solutions to those in other countries.
The very real prospect of unpleasant and unavoidable consequences focusses the mind wonderfully. I've always taken the view we humans do stupid things until we find the one that kills us, we now seem intent on taking that to the species level.
@ Anon: a research student burst into my office and said "Turn on the News". Of course everything was jammed and we couldn't get into the Beeb or ITN or the Yank stations. So we hied off to the "computer suite" and decided we'd try a country far enough to the west that many people would be busy at work rather than finishing lunchtime sandwiches at their desks. And where US news perhaps didn't dominate the way it does in Britain. Also we wanted somewhere where we'd have some chance of understanding the lingo. We chose Chile and it worked right away. Some of the coverage was in English with Spanish subtitles and some in Spanish, which is not especially hard to understand.
Then a surreal moment. A passing Spanish research student said "I didn't know you spoke Spanish." I replied "I don't speak as much as Uncle Luis wanted me to speak" and he was so gob-smacked that he said "You can't possibly have an Uncle Luis."
Anyway he stayed with us for while, interpreting any bits he thought might be difficult for us. What I didn't know at the time was that the daughter of an old friend had been about to enter the second tower just as the second plane hit it. She ran for it, all the way back to her apartment in Brooklyn. And yet within days, it seemed, loonies were assuring me that it was all a holographic fraud.
The whole thing was cheapened when W, backed by such creeps as our own dim and ignorant little fellow Blair, attacked Iraq in revenge for the US being attacked by Saudi Arabians. I got the impression that the bulk of Americans backed W. Fools.
You rely on what you're told. I'm not even sure bin Laden was involved - USS Cole, yes.
Of course Bush, with an eye to Michigan voting, had instructed airlines to no longer profile Arabs as potential hijackers a year earlier - Oct 11 2000 to be precise.
The boarding clerk felt Atta and co were dodgy but ...
https://www.rte.ie/news/us/2026/0910/1591069-911-airline-worker/
"I followed all the guidelines; I did my job," he said. My gut instinct was good, but there were laws, there were laws against profiling people," said Mr Tuohey, now 80 and retired. "I didn't listen to my gut. And it turned out to be an awful thing," he said from his Maine home.
Post a Comment