Technology

Exactly this. We’re always moving the goalposts to maintain the belief of human exceptionalism. We also used to say that tool use and construction were what separated humans from the animals, until examples were found of animals using or making tools, and then we moved the goalposts further to exclude them.

This pervasive belief that humans are beyond nature or singularly extraordinary in their intelligence and consciousness is rooted in arrogance and bad science, and it hinders our understanding of science and consciousness and our place in the universe.

If an intelligence is able to feign consciousness so well that we can’t distinguish it from “real” sentience, then it’s close enough that we should treat it as such. Those who insist on defending the idea of human exceptionalism are simply invested in maintaining human superiority and exploitation of animals and machines beyond what humans and law would otherwise accept as moral, if we were to respect other intelligences as equal and deserving of their own rights.

I am sceptical of this thought experiment as it seems to imply that what goes on within the human brain is not computable. For reference: every single physical effect that we have thus far discovered can be computed/simulated on a Turing machine.

The argument itself is also riddled with vagueness and handwaving: it gives no definition of understanding but presumes it as something that has a definite location, and also it may well be possible that taking the time to run the program inevitably causes understanding of Chinese after even the first word returned. Remember: executing these instructions could take billions of years for the presumably immortal human in the room, and we expect the human to be so thorough that they execute each of the trillions of instructions without error.

Indeed, the Turing test is insufficient to test for intelligence, but the statement that the Chinese room argument tries to support is much, much stronger than that. It essentially argues that computers can't be intelligent at all.

That just shows a fundamental misunderstanding of levels. Neither the computer nor the human understands Chinese. Both the programs do, however.

The problem with the experiment is that there exists a set of instructions for which the ability to complete them necessitates understanding due to conditional dependence on the state in each iteration.

In which case, only agents that can actually understand the state in the Chinese would be able to successfully continue.

So it's a great experiment for the solipsism of understanding as it relates to following pure functional operations, but not functions that have state changing side effects where future results depend on understanding the current state.

There's a pretty significant body of evidence by now that transformers can in fact 'understand' in this sense, from interpretability research around neural network features in SAE work, linear representations of world models starting with the Othello-GPT work, and the Skill-Mix work where GPT-4 and later models are beyond reasonable statistical chance at the level of complexity for being able to combine different skills without understanding them.

If the models were just Markov chains (where prior state doesn't impact current operation), the Chinese room is very applicable. But pretty much by definition transformer self-attention violates the Markov property.

TL;DR: It's a very obsolete thought experiment whose continued misapplication flies in the face of empirical evidence at least since around early 2023.

Searle argued from his personal truth that a mystic soul is responsible for sapience.

His argument against a computer system having consciousness is this:

" In order for this reply to be remotely plausible, one must take it for granted that consciousness can be the product of an information processing "system", and does not require anything resembling the actual biology of the brain."

-Searle

https://en.m.wikipedia.org/wiki/Chinese_room

Brilliant thought experiment. I never heard of it before. It does seem to describe what's happening - if only there were a way to turn it into a meme so modern audiences could understand it.

That's essentially the media-generated Turing Test, but in truth no such test was ever defined by Alan Turing. For me the modern takeaway is don't extrapolate anything about reality from memes.

More "can fool the average idiot."

'Passing' isn't fooling a single participant, but the majority of them beyond statistical chance.

On the other side, I heard a story about a human failing the Turing test, because the examiner happened to be matched with a real person who was an expert on Shakespeare, and therefore appeared to know more than a human would.

This is a good discussion about the original tests Turing discussed, as well as a very fun interview.

https://www.ouropinionsarecorrect.com/shownotes/2024/4/4/the-turing-test-is-bullshit-walex-hanna-and-emily-m-bender

Yes I think that's generally what Alan Turing meant - he was careful not to define what "intelligence" means, and was discussing practical perception of machine behavior.

Not sure how you define getting "hung up" but there are tons of poorly informed people who believe/fear that AI is about to take over/conquer/destroy/whatever the world because they think LLMs are as smart as humans - or just a few tweaks away. It's less about the word "intelligence" than about jumping from there to collateral issues, like thinking LLMs are "persons" that deserve rights, that using them without their consent is slavery, and other nonsense. Manipulative people take advantage of this kind of ignorance. Knowledge is good, modern superstition is bad.

It's just something we have to take in stride, like computers becoming capable of playing Chess or Go. There is no need to get hung up on the word "intelligence".

Nicely said.

This is the book that introduced me to the Chinese Room thought experiment and is the first thing I began to think of when the recent AI trend started to make a splash.

Peter Watts is great and though it's not related to the topic at hand, I cannot recommend Starfish enough. Dark, haunting, and psychological. (It's apparently part of a series but I never carried on)

Glad to see mentions to Peter Watts. His view of humanity is dry and take on real world is even grimmer, but it's intriguing and backed by science. Also I'm the one of people dying to know what he said at the end of his lecture.

Eliza, a chatbot psychiatry emulator written in the 1960s, convinced many people it was a real person.

Various versions of Eliza are online - including this quaint, retro looking one