Tech Singularity

The technological singularity — or simply the singularity — is a hypothetical point in time at which technological growth becomes uncontrollable and irreversible, resulting in unforeseeable changes to human civilization. According to the most popular version of the singularity hypothesis, called intelligence explosion, an up-gradable intelligent agent will eventually enter a “runaway reaction” of self-improvement cycles, each new and more intelligent generation appearing more and more rapidly, causing an “explosion” in intelligence and resulting in a powerful super intelligence that qualitatively far surpasses all human intelligence.

https://en.wikipedia.org/wiki/Technological_singularity

Grist

A “runaway reaction”…

What if, instead of a hyper-intelligent AI, the complexity of the tools and systems being used outstrip the capacity of the people using them to understand them? It could be that the documentation was never written and brain drain has taken it’s toll, or the cough incentive structure cough does not invest in and/or reward/respect the deep thinking required to master the subject at hand.

We’ve already seen multiple mission critical functions marked as DO NOT CHANGE because the person who wrote it didn’t document it and ain’t around anymore; so we have to wrap it in a facade to keep it working with the rest of the code base. Basically a black box that no one knows what it does but(apparently) it’s important. Of course there are no tests so no one knows if it ever worked correctly let alone if it’s still working correctly… (see how the SAMBA team knows more about the SMB/CIFS protocol that Microsoft…)

NASA is having to reverse engineer the F1 engines because the program was shut down so quickly that no one wrote down how they did what they did. Of course NASA understands how a liquid fueled gas generating rocket engine works, but the don’t know how the original engineers got the power/efficiency out of the design that they did. So they can make a copy, but they can’t improve it based on the docs that they have (ala 40K…)

The Toyota Way was developed to overcome the specific challenges of building cars in post-war Japan where they could not benefit from the massive economies of scale that the “Big Three” enjoyed here; so they had to come up with another strategy. So naturally as it passed through hands over time things like “no excess inventory” devolved into “no inventory”. The consequences of which we’ve learned the hard way during the COVID unpleasantness.

We’ve also seen what “High Frequency Trading Algorithms” can do the the stock markets. Huge volumes trade signals are generated by algorithms trying to trip up other algorithms looking to create a window to get over on the market. So now we have “flash crashes” and “circuit breakers” to limit/contain the damage that they can do.

We have endless examples of bad training data being used to create “AI” algorithms that lead to (at best) undesirable outcomes. 1) The wolf example, 2) classifying humans as gorillas, 3) the example of running stuff between two algorithms that ended up with them hiding info in the data set via steganography.

Once your app has grown to a certain scale then poking an algorithm (aka adjusting a weight on a value) could drown someone in an undesirable amount of “engagement” or destabilize a country; roll the dice champ…

We already know that exposed self-learning algorithms can be exploited by bad actors; see the MS chat bot that got hijacked…

We already know that chat bots who talk to chat bots can “go feral”; there’s at least one example where the bots in question created an unintelligible to the human language.

The Qanon nonsense. Originally started on 4-chan as a prank that worked a lot like an Alternative Reality Game (ARG) with the cryptic “q drops” as prompts for the community to add their own stuff two. Then it started to expand into the the real world: 1) that guy who blocked the highway with an armored vehicle, 2) family members abducting their own kids, 3) assaulting people in public (that supermarket incident), 4) murdering their own kids (the guy in Mexico), 5) spreading into the “soccer mom” demographic via the “save the kids” campaign.

A deceptively crude machine showing up at the right time can be the event that turns the whole world on its head. It’s reasonable to assume that it won’t be recognized or appreciated until well after its happened:

The 1751 Machine that Made Everything

I always assumed that it was the invention of the steam engine that fueled the rapid growth in productivity and such.

Funnily enough [redacted] always called his lathe the “mother tool” because he could use it to build the tools (eg a mill) needed to build the tools that he needed to build the thing he was actually trying to build; always pointed out that it could be an iterative process: build a simple lathe to build a simple mill to build a better lathe to build a better mill (repeat as required)