Pinned Tweet
Theoretical Foundations for Semantic Cognition in Artificial Intelligence
arxiv.org/abs/2504.21218
Transformer state machine:
Let T be a transformer-based agent running c' = T(c, o) on token contexts c, observations o, and policy a = A(c').
We can designate a region of c as a dynamically updating area ("brain") that gets modified by A with respect to specified triggers.
Autonomous agent state machine:
- SOUL.md: Persona & safety
- TOOLS.md: SOPs & API schemas
- USER.md: Entity preferences
- MEMORY.md: Working state
- PLAN.md: Task execution graph
- HEARTBEAT.md: Proactive loops
- SCRATCHPAD.md: Transient workspace
- AGENTS.md: Routing protocols
On agent alignment:
Let an agent's state s evolve as s' = U(s, o) from observations o, with policies a = A(s').
There are four general areas for targeted alignment:
(1) The model U.
(2) The states s.
(3) The observations o.
(4) The actions a.
Monosemantic feature extraction:
To resolve polysemanticity in zₖ, a security model U can use a Sparse Autoencoder (SAE).
U projects dense activations zₖ into a high-dimensional sparse feature space.
Interpretable representations emerge from sparse features, not raw neurons.
On transformer model polysemanticity:
In a transformer's latent rollout, zₖ represents overlapping concepts in superposition.
Layer activations zₖ are dense, entangled, and uninterpretable.
A security model U can resolve the polysemanticity, yielding monosemantic features.
Transformer latent rollout security:
In a transformer, the latent rollout
z₁ ─> z₂ ─> ...
happens vertically in a single forward pass.
A security model U(s, h) must tap intermediate depth latent rollout states zₖ, making monitoring granularity an architectural choice.
Transformer security model:
Let T be a transformer-based agent running c' = T(c, o) on token contexts c.
As T executes, it produces internal hidden activations h.
A security model U alongside T computes s' = U(s, h), where s tracks the internal cognition of T over time.
A deeper analysis:
(1) Psychology: Mapping U to behaviors.
(2) Representations: Mapping s to belief states.
(3) Decisions: Mapping s' to goals and plans.
(4) Internal feedback: How past o and a accumulate in s.
(5) External feedback: How a alters environment dynamics to yield o.
Alignment as dynamical invariant:
To prove our controller C(s, W) works, the region 𝒜(s) must be forward invariant under controlled dynamics s' = U(s, o, C):
G(s, W) ∈ 𝒜(s) implies G(s', W) ∈ 𝒜(s')
Hence, we must construct C such that 𝒜 is invariant under U( · , · , C).
Alignment controllers:
We can build an active controller C(s, W) using minimal perturbations (ds, dB, dG) to enforce invariant boundaries:
G(s, W) ∈ 𝒜(s)
By optimizing over internal states rather than policing actions, the controller steers internal dynamics for alignment.
Perturbative latent alignment:
Control operates at three levels:
(1) Latent state perturbations ds
(2) Belief state perturbations dB
(3) Cognitive profile perturbations dG
ds and dG grant analytical control; dB enables interpretable control via belief injection and filtering.
Connecting human values to latent alignment:
We map human values into acceptable cognitive regions:
(1) Identify value-salient states v using B(v)
(2) Extract cognitive profiles G(v, W)
(3) Construct local boundaries 𝒜(s) for s near v
Human values define the bounds of 𝒜(s).
Operationalizing latent alignment:
Many distinct cognitive profiles produce acceptable behavior.
We replace single-point targets G*(s) with an acceptable cognitive region 𝒜(s):
G(s, W) ∈ 𝒜(s)
Alignment means keeping the agent's trajectory constrained within the 𝒜(s).
Latent alignment versus behavioral alignment:
Behavioral alignment polices the policy a = A(s'). If the action is safe, it is accepted.
Latent alignment constrains cognitive dynamics to enforce safe state transitions s' = U(s, o).
Latent alignment permits verifiable control.