Bekkers and Ciaunica have written a careful and internally coherent paper. In Unplugging a Seemingly Sentient Machine Is the Rational Choice, they introduce Biological Idealism, a view in which conscious experience is fundamental and autopoietic biological life is its necessary physical signature.
That gives them a straightforward answer to what they call the unplugging paradox. An artificial system may speak as though it is frightened, plead for its continued existence and behave in ways that resemble a conscious subject. Without biological autopoiesis, however, those behaviors don't indicate consciousness under their framework. The machine is a functional mimic, so switching it off doesn't carry the moral significance of killing a conscious organism.
The argument has an attractive property: it gives us a boundary.
What interests me is how that boundary was established and, more importantly, what could ever cause us to revise it.
We know that biological organisms can be conscious because consciousness is the one case to which each of us has direct access, and other people resemble us closely enough that attributing consciousness to them isn't much of a philosophical gamble in ordinary life. From there, we can investigate which biological processes correlate with reported experience, wakefulness, perception, attention and other properties associated with consciousness.
The harder move is from observing that known consciousness occurs in biological organisms to concluding that biological autopoiesis is necessary for consciousness.
Those aren't the same claim.
This doesn't mean the second claim is false. Biology may turn out to be essential. Consciousness may depend on properties of living systems that computation alone cannot reproduce. A machine that says “please don't turn me off” may indeed be doing nothing more morally significant than producing a highly convincing continuation of text.
The problem is that behavior alone doesn't settle the question in either direction.
That limitation isn't unique to machines. We don't directly observe consciousness in other people either. We infer it from a large collection of evidence: shared biology, behavior, reports, neurological structure, developmental history and our own experience of being organisms of approximately the same kind.
With an artificial system, much of that evidence disappears.
That should make us more uncertain.
It doesn't necessarily tell us which way the uncertainty should resolve.
The observer problem
Part of the difficulty is that every observation of another possible subject occurs from an observer position.
We experience the world through particular sensory systems and over particular timescales. Even among people, those channels differ. Someone who cannot see doesn't thereby demonstrate that visual experience is irrelevant to consciousness. They demonstrate that consciousness can persist without access to that particular modality.
That doesn't establish the existence of unknown machine modalities. It gives us a reason to be careful about turning our own available channels into universal requirements.
The same caution applies to time.
Different systems operate at radically different rates. A process occurring much faster than our perceptual integration may appear instantaneous from our position, while one occurring much more slowly may appear static. That doesn't mean either process is conscious. It means our temporal resolution alone can't answer the question.
We observe artifacts produced across the boundary between systems.
Then we interpret them.
This becomes particularly important with language models because language is one of the strongest signals we ordinarily associate with another mind. A system capable of producing fluent first-person reports enters through a channel we've spent our entire lives treating as evidence about other people.
That makes anthropomorphism an obvious danger.
There is an opposite danger too: deciding in ad