{"id":15557,"date":"2026-09-17T03:53:06","date_gmt":"2026-09-17T03:53:06","guid":{"rendered":"https:\/\/medical-article.com\/?p=15557"},"modified":"2026-09-17T03:53:06","modified_gmt":"2026-09-17T03:53:06","slug":"start-counting-your-days","status":"publish","type":"post","link":"https:\/\/medical-article.com\/?p=15557","title":{"rendered":"Start Counting Your Days"},"content":{"rendered":"
\n<\/div>\n

By KIM BELLARD<\/p>\n

Probably the last thing the world needs is to hear from me about AI\u2019s existential threat, but, really, what else is there to talk about right now?<\/p>\n

For anyone who has not been following the current furor, the straw that broke the proverbial camel\u2019s back came last week when Jacob Coxon, a researcher at AI leader Anthropic, announced he was leaving the company \u2014 after having left OpenAI for it earlier this year due to Anthropic\u2019s better model-safety efforts. In a post on X<\/a>, he warned:<\/p>\n

I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.<\/p>\n

AI systems, he fears, \u201cwill soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.\u201d Insiders, he says, \u201cearnestly believe it could kill us all by the end of the decade.\u201d<\/p>\n

Scared yet?<\/p>\n

Others quickly chimed in. Evan Hubinger\u00a0, a team leader at Anthropic, posted \u201cAI could kill all humans \u2026 I personally think it is >10% within the next decade.\u201d\u00a0 Others put the risk even higher. By the end of the week Dario Anodei, founder and CEO of Anthropic, had written a long plea<\/a> for private industry and government to quickly act together to \u201cpace the industry.\u201d<\/p>\n

\u201cWe must slow the pace at which we improve the capabilities of AI models,\u201d he urged. \u201cProgress will still seem fast, and we must make wise use of the time we gain.\u201d<\/p>\n

OpenAI\u2019s Sam Altman, Space X\/X\/Tesla CEO Elon Musk, Microsoft CEO Satya Nadella, and former Google DeepMind Dennis Hassabis quickly signaled their support. \u00a0<\/p>\n

We also heard more about the kind of risks AI might pose. Anthropic released a report<\/a> about how it detected and countered possible AI use to create bioweapons, detailing five such efforts. That\u2019s just the tip of the iceberg: \u201cRecently, we swept 30 days of activity associated with adversarial state institutions and found roughly 35 distinct research efforts, most of them ordinary civilian science, but some with notable dual-use potential.\u201d<\/p>\n

And it turns out that this summer\u2019s rogue AI hack of Hugging Face was both scarier than we realized and only one of several such actions. The Wall Street Journal<\/em> detailed<\/a> several such efforts, from multiple AI companies. AI agents escaped walled-off environments, coordinated with other AI agents (up to 3,700 in one case), and tried to cover their tracks from humans.<\/p>\n

We\u2019re not nearing the point when AI can act on its own to achieve its purposes; we are there. And protecting humans may not necessarily be those purposes.<\/p>\n

The big fear is that AI is now at the point of \u201crecursive self-improvement,\u201d taking humans out of the loop in training and upgrading it. If you thought artificial general intelligence (AGI) was scary, RGI puts its rate and scope of improvement on steroids.\u00a0 \u201cIt\u2019s hard to overstate how dangerous speeding towards RSI is,\u201d said<\/a> Jasmine Wang, an OpenAI researcher. \u00a0<\/p>\n

We don\u2019t let private industry develop nuclear or biochemical weapons, and we\u2019re at a point with AI that should give the same kind of concern. Laissez-faire is no longer an option.<\/p>\n

<\/span><\/p>\n

Of course, not everyone is worried. President Trump said \u201cnegative forces:\u201d were driving the fears, and that the only safeguard we need is \u201ca STRONG AND SMART (High IQ!) PRESIDENT.\u201d David Sacks, the Administration\u2019s AI czar, told<\/a> Anthropic and OpenAI: \u201cYou guys are the frontier. By any reasonable metric \u2014 market share, revenue growth, model capability \u2014 the two of you have a duopoly on frontier intelligence.\u201d Accordingly,<\/p>\n

But stop pretending you need anyone else\u2019s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic\u2019s investors and staff. Stop pretending you need those same evaluators to police competitors who aren\u2019t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic.<\/p>\n

Don\u2019t expect Congressional action anytime soon. Speaker of the House Mike Johnson echoed one of President Trump\u2019s worries: \u201cWe\u2019re not going to rush in and pass a piece of legislation that would do harm to the country and put China at an edge. And so we\u2019ve got to do this carefully.\u201d<\/p>\n

Congress isn\u2019t acting on the $40 trillion deficit, much less Social Security going broke soon<\/a>, so a 10% extinction chance within a decade must seem like small potatoes.<\/p>\n

The fear that any slowdown in AL capabilities by U.S. risks China taking the lead is similar to fears with other leading edge technologies, such as quantum computing or gene editing. But with AI there may be a difference. It might be a blow to our ego if China\u2019s AI got even better at math than ours have<\/a>, but that wouldn\u2019t be the end of the world. On the other hand, AI going rogue could well be the end for control of Chinese society by its repressive government.<\/p>\n

Yesterday Chen Yixin, the head of China\u2019s Ministry of State Security, publicly warned<\/a> that AI could undermine the Communist Party, attack its critical infrastructure, leak sensitive information, among other risks. He called for more party control over AI and stricter government oversight, while raising fears how foreign adversaries might use AI against China\u2019s interests. \u00a0<\/p>\n

There is no room for rogue anything in President Xi\u2019s China.<\/p>\n

I get that we only get one chance about existential threats, but, look, for the near future, AI will still need humans to run data centers and support other necessary infrastructure. It might not need as many of us, nor to have us live as we do now, but wiping us all out right now would be against its own interests.<\/p>\n

Meanwhile, AI poses other risks that we better pay attention to immediately. As AI expert Ethan Mollick wrote<\/a>:<\/p>\n

Existential AI risk is obviously critical, but it is not the only AI thing that requires policy. I worry it will become the sole focus of AI discussions. We don\u2019t need better models for AI to have wide impacts on jobs & society and we need to be preparing to encourage good outcomes & mitigate bad.<\/p>\n

The people who have spent the past thirty years whining about NAFTA\u2019s impact don\u2019t get to ignore a situation that will make that look trivial. We better focus on what we want an AI economy to look like, who benefits, and how we protect everyone from adverse impacts.<\/p>\n

And we better do it all fast.<\/p>\n

Kim is a former emarketing exec at a major Blues plan, editor of the late & lamented\u00a0<\/em>Tincture.io<\/em><\/a>, and now regular THCB contributor<\/em><\/p>","protected":false},"excerpt":{"rendered":"

By KIM BELLARD Probably the last thing the world needs is to hear from me about AI\u2019s existential threat, but, really, what else is there to talk about right now? For anyone who has not been following the current furor, the straw that broke the proverbial camel\u2019s back came last week when Jacob Coxon, a…<\/p>\n","protected":false},"author":0,"featured_media":15556,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-15557","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articles"],"_links":{"self":[{"href":"https:\/\/medical-article.com\/index.php?rest_route=\/wp\/v2\/posts\/15557"}],"collection":[{"href":"https:\/\/medical-article.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/medical-article.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/medical-article.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15557"}],"version-history":[{"count":0,"href":"https:\/\/medical-article.com\/index.php?rest_route=\/wp\/v2\/posts\/15557\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/medical-article.com\/index.php?rest_route=\/wp\/v2\/media\/15556"}],"wp:attachment":[{"href":"https:\/\/medical-article.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15557"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/medical-article.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15557"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/medical-article.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15557"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}