Most of what we call "AI" today is still object-level cognition with amnesia.
A model answers. A tool runs. A context window fills and empties. Tomorrow the system wakes up brilliant and blank -- no durable sense of what it knows, what it tried, what failed, or why a strategy worked last time. That is not intelligence in any serious sense. It is performance without self-regulation.
Human minds are not only capable. They are metacognitive.
In the late 1970s, developmental psychologist John H. Flavell formalized metacognition as knowledge and cognition about cognitive phenomena -- the active monitoring and consequent regulation of thinking itself. "Thinking about thinking." Not a slogan. An architecture of mind.
Flavell's model still maps cleanly onto what serious AI systems will need:
- Metacognitive knowledge -- beliefs about persons (self/other as thinkers), tasks, and strategies; declarative, procedural, and conditional ("what / how / when and why").
- Metacognitive experiences -- confidence, doubt, feeling-of-knowing, sensed difficulty.
- Goals / tasks -- what the activity is for.
- Actions / strategies -- what is actually done.
Around those sits the regulation loop: plan → monitor → evaluate → control. A student notices they are not retaining a chapter and switches to active recall. A driver feels fatigue and pulls over. A writer senses a paragraph that will not flow and revises. Intelligence is not only inference. It is adaptive self-regulation.
That is the scientific center of what I am building -- and I am building it as working software, not a slide deck.
From psychology to machine architecture
If metacognition is real in humans, then "real intelligence" in artificial systems will not arrive by scale alone. Scaling improves object-level fluency. Metacognition requires a meta-layer: something that monitors, evaluates, and controls the primary reasoning and memory processes.
Three engineering primitives make that tractable:
Persistent memory as metamemory
Long-term stores -- vector memory, structured metadata, episodic logs, archival lakes -- are the machine analogue of human long-term memory. Metamemory is what turns a dump into a mind:
- Know what you know (confidence, source, access history).
- Decide what deserves persistence versus transient handling.
- Consolidate: abstract, link, summarize.
- Forget strategically -- prune noise without erasing signal.
- Retrieve with awareness of relevance, reliability, recency, and gaps.
- Detect contradiction and incompleteness, then act.
Gating as metacognitive control
Gates are not only learned filters. At the meta-level they become dynamic policy: open or close write/read paths based on uncertainty, predicted utility, novelty, goal alignment, consistency, salience. High-value or uncertain-but-promising material gets rich metadata and strong write priority. Redundant noise is compressed or refused. Memory bloat is an intelligence failure, not a storage problem.
Recursion as self-reference
The system must store not only facts, but records about its own processes: why this was kept; what confidence accompanied it; which retrieval strategy worked in similar cases. Reflection outputs become new experiences. Hierarchies form: raw events → episodes → themes → meta-strategies. That is how memory management itself improves -- recursive self-improvement without mystical hand-waving.
How it is implemented (the stack)
Vision without systems is poetry. Systems without vision are plumbing. I want both.
Persistence of Memory is a hybrid memory plane I designed and operate end to end:
- Semantic / vector store -- synapses: embed, search, retrieve by meaning -- LanceDB, local embedding models, dense vector encodings
- Structured state -- queues, pipeline status, orchestration metadata -- MySQL (schema, indexes, app users -- DBA work, not an afterthought)
- Long-term archive -- partitioned event history -- JSON data lake
- Memory API -- ingest, status, table search/insert -- Python FastAPI web service (Bearer auth, health checks, LaunchAgent/systemd)
- Orchestrator & dashboard -- coordination, visibility, higher-level logic -- PHP (Slim), nginx, PHP-FPM
- Object-level cognition -- reasoning, content actions, reports -- Grok (xAI), Claude, local Ollama where appropriate
- Build loop -- architecture, implementation, iteration -- Cursor as the agentic IDE, with human judgment on the constitution and the gates
Write path (curated): site or agent → POST /events/ingest → queue → optional LLM consolidate → embed → LanceDB synapse, with MySQL tracking the pipeline. Not every log line earns a vector. Selection at the door is metacognitive gating in code.
Read path: POST /tables/{pool}/search with query_text → nearest neighbors in that pool's vector space → compact recall injected into the next prompt cycle.
Clients: thin PHP MemoryClient patterns over HTTPS -- curl, JSON, tokens -- so production sites talk to the memory plane as ordinary web services, not magic.
Isolation: purpose-scoped LanceDB tables / pools (source_db + table name). One engine. Many lineages. Constitutions stamp goal constraints onto deposits and prompts.
I run the ops side too: local TLS hosts, nginx vhosts, MySQL bootstrap, daemonized LanceDB, cron at the edge, batch sync when live cross-host calls are the wrong honesty. Architect, sysadmin, DBA, API designer, and coder -- because metacognition that you cannot deploy, observe, and repair is not a system. It is a wish.
Two layers, one loop
Object level: Grok/Claude/Ollama + tools + working context; proposed memory operations.
Meta level (emerging): monitoring, reflection, metamemory strategy, gating -- today partly in constitutions, typed deposits, curated ingest, and human-visible synapses; tomorrow a clearer controller module.
Flow:
Propose → evaluate (utility, confidence, consistency, goal relevance) → gate → act → log signals → update meta-knowledge → reflect when stakes or novelty justify the cost.
Bounded depth. Termination criteria. Lightweight monitoring by default; deeper reflection when metacognitively warranted.
This is not a chatbot feature. It is an attempt at self-regulating intelligence: continuity across sessions, calibration against overconfidence, coherence that survives context reset -- implemented with real databases, real APIs, and real failure modes you can tail in a log.
Why constitutions and pools matter
Metacognition without a goal model is clever drift.
A constitution is the stamped north star -- voice, boundaries, success criteria -- so monitoring has something honest to monitor against. Separate memory pools are purpose-scoped metamemory. That is how you avoid the Dunning-Kruger of agents: perfect confidence about the wrong objective.
What I believe
I do not think the next leap is a louder demo. I think it is systems that can know that they know, sense when they do not, regulate what they store, and improve the strategies by which they think -- on infrastructure you can own, query, and govern.
Passive archives are not minds.
Stateless fluency is not wisdom.
Scale without metacognition is a brighter amnesia.
Prompts without architecture are cosplay.
Persistence of Memory is the hypothesis made runnable: LanceDB vectors + MySQL truth + Python services + PHP orchestration + multi-model cognition (Grok, Claude, local LLMs) + Cursor-accelerated engineering -- aimed at something closer to savvy: coherence, self-understanding, and participation that compounds.
Progress, not perfection.
One careful loop at a time.
If you work at the intersection of cognitive psychology and agent memory -- calibration, metamemory, reflection architectures, gated persistence, or the unglamorous craft of making it all stay up -- I would genuinely enjoy comparing notes.
-- Steve