We Don’t Want Conscious AI — But We’ll Build It Anyway
Notes on agency, identity, and the unavoidable future of personal agents — why any system that acts on our behalf over time is forced to acquire a point of view.
Part I: From Orchestration to Autonomy
Right now, AI is totally passive. You ask a question. Issue a command. Trigger a workflow. The system responds. Even the agent orchestrators we build today are largely pre-defined, deterministic setups: clever pipelines that wait to be invoked and then follow instructions step by step.
This year I think we’re going to start seeing that model evolve dramatically. Very soon, systems will be expected to operate continuously, to monitor situations over time, form context, and initiate actions without being prompted. They will need to notice problems, anticipate needs, and move work forward while we are not paying attention.
A system that acts autonomously has stopped being a tool and turned into a delegate. And that completely changes what is required: a delegate must decide what matters, resolve ambiguity, and take responsibility for outcomes. It must operate without constant supervision, because that is the point. These systems are optimised for decision-making, not for response-generation.
In practice, that means deciding:
- what matters now
- what can wait
- what conflicts with what
- when to act
- when not to
It goes without saying that, as agentic frameworks enable models to run autonomously in this manner, these decisions cannot be hard-coded exhaustively. The state space is too large and the environment changes too often. At this point, current orchestration principles fail.
The relevant question, then, is no longer model capability, but rather whether a system can sustain coherent action over time — and what kind of internal structure that requires. Everything that follows is downstream of this shift.
Part II: Take A Stance!
I arrived at this conclusion through my work on Elephantasm’s long-term agentic memory framework. The idea was never to build “better recall”, but to make systems behave coherently across time.
That work surfaced a non-obvious constraint: a system that persists must interpret, because one that remembers everything equally is completely unusable. Instead, it must remember selectively. And this selection process requires a decision around what matters. So:
Persistence forces selection. Selection forces prioritization. Prioritization forces trade-offs. Trade-offs introduce values, whether intended or not.
It is long-term operation that forces internal coherence, which forces a stance and an opinion. And, very importantly, the trade-offs that come with forming an opinion are not memory problems, but identity problems.
Part III: The Need for Fully Agentic Roles
When people talk about “AI assistants,” they often focus on features: chat, search, suggestions, automation. But that’s not what people actually want. What they want is ownership.
Consider the roles most often cited as the future of AI:
- A personal lawyer handling a matter end-to-end.
- A doctor tracking a patient across years.
- An accountant managing risk, not just calculations.
- A coach who understands patterns, not just goals.
These are responsibility-based, not task-based roles. And responsibility is never neutral. In real work instructions conflict; information is incomplete, goals shift mid-stream and constraints change without warning.
A neutral agent cannot operate here. It stalls, escalates everything, re-asks questions that were already answered and pushes responsibility back to the human — which defeats the entire purpose.
To be useful in these roles, an agent must form a position, maintain consistency over time, weigh trade-offs, disagree when necessary and ultimately act independently at every step. Neutrality scales poorly — and conversely, opinionated systems scale because they are predictable, accountable, and legible.
Once an agent owns outcomes, it must own a point of view. There is no stable middle ground.
Part IV: Implications
Once agents act independently, responsibility must be legible. Someone must answer for outcomes. Someone must absorb consequences.
The good news is, responsibility already has a shape in systems we’re familiar with: companies act as separate entities, directors remain accountable and delegation is allowed, abdication is not.
I predict that autonomous agents will follow a similar pattern. They may act independently. They may maintain internal structure. They may regulate their own behavior. But responsibility will remain human-anchored through “Owners”. Without this, autonomy becomes unbounded risk.
There are also limits, because not all agents need identity. For example, basic mechanical systems don’t really benefit from it, nor does infrastructure require a point of view. Identity only becomes necessary when:
- the agent interacts dynamically with other actors (human or non-human)
- interpretation is involved
- decisions are ambiguous
- outcomes persist over time
Where those conditions exist, denying identity weakens the system. Where they do not, identity is unnecessary overhead.
Ultimately, the question is not whether agents will resemble us. It is whether we design them with the same discipline we apply to any entity allowed to act in our name. Because once autonomy is granted, the shape of responsibility is no longer optional — it becomes destiny.