{
  "schemaVersion": "shareplane.work-portable-view.v1",
  "workId": "artifact:let-intelligence-be-messy-make-authority-boring",
  "canonicalWorkId": "work:let-intelligence-be-messy-make-authority-boring",
  "title": "Let Intelligence Be Messy. Make Authority Boring.",
  "classification": {
    "workType": "architecture-thesis-field-essay",
    "treatmentClass": "public-technical-position",
    "publicSafe": true
  },
  "author": {
    "id": "person:tony-malott",
    "name": "Tony Malott",
    "url": "https://malott.ai/",
    "resumeUrl": "https://malott.ai/resume/",
    "emailUrl": "mailto:tony@malott.ai"
  },
  "discovery": {
    "thesis": "Stop trying to make AI trustworthy as the primary safety strategy. Build systems that remain trustworthy when AI is not.",
    "abstract": "AI is probabilistic intelligence. Use that capability where uncertainty creates value, but separate intelligence from authority. Trustworthy operation comes from independently enforced invariants, bounded capabilities, exact artifact identity, evidence, segregation of duties, revocation, and deterministic admission.",
    "topics": [
      "ai-architecture",
      "agentic-systems",
      "deterministic-authority",
      "trust-boundaries",
      "system-safety",
      "governance"
    ],
    "kind": "Architecture thesis and field essay",
    "collections": [
      "Governed AI operations"
    ],
    "format": {
      "label": "Architecture thesis and field essay",
      "reading_time": "14 min",
      "depth": "Long-form technical position",
      "interaction": "Editorial narrative with four code-native teaching surfaces"
    },
    "card": {
      "kicker": "Probabilistic intelligence · Deterministic authority",
      "orientation": "Use probabilistic intelligence for discovery. Keep consequential authority explicit, bounded, and independently enforced.",
      "visual_motif": "authority-boundary"
    }
  },
  "relationships": [
    {
      "id": "relationship:authority-boring-agent-security",
      "sourceId": "artifact:let-intelligence-be-messy-make-authority-boring",
      "targetId": "artifact:the-agent-is-not-the-security-boundary",
      "predicate": "extends",
      "readerGroup": "foundations",
      "standing": "active",
      "readerLabel": "Extend the security boundary into system trust",
      "explanation": "The earlier Work separates agent behavior from the surrounding security boundary. This Work generalizes that principle across models, generated code, deterministic software, validators, operators, and unknown failures."
    },
    {
      "id": "relationship:authority-boring-ambiguity-frontier",
      "sourceId": "artifact:let-intelligence-be-messy-make-authority-boring",
      "targetId": "artifact:inference-at-the-ambiguity-frontier",
      "predicate": "companion_to",
      "readerGroup": "companions",
      "standing": "active",
      "readerLabel": "Companion: Inference at the Ambiguity Frontier",
      "explanation": "The ambiguity-frontier thesis explains where probabilistic inference belongs. This Work adds that deterministic machinery alone is not sufficient and separates intelligence from consequential authority."
    }
  ],
  "evidence": [
    {
      "id": "evidence:authority-boring-owner-publication",
      "type": "owner-publication-direction",
      "title": "Issue #1249 owner acceptance and publication direction",
      "locator": "https://github.com/pinklon/shareplane-platform/issues/1249",
      "role": "Records accepted offline UAT v0.1, Creative Lock, public Work identity, evidence boundary, and bounded publication authority.",
      "caveat": "Publication must fail closed before provider mutation unless current provider quietness is proven."
    },
    {
      "id": "evidence:nist-airmf-core",
      "type": "authoritative-institutional-source",
      "title": "AI Risk Management Framework Core",
      "locator": "https://airc.nist.gov/airmf-resources/airmf/5-sec-core/",
      "role": "Supports claim:lifecycle-risk, claim:independent-assessment, claim:production-monitoring, claim:third-party-components.",
      "caveat": "Does not establish the article's original synthesis that authority should be the deterministic boundary around probabilistic semantics."
    },
    {
      "id": "evidence:nist-airmf-gaip",
      "type": "authoritative-institutional-source",
      "title": "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile",
      "locator": "https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence",
      "role": "Supports claim:system-lifecycle-risk.",
      "caveat": "Does not validate any specific GhostMesh implementation or claim that deterministic controls eliminate AI risk."
    },
    {
      "id": "evidence:owasp-excessive-agency",
      "type": "authoritative-institutional-source",
      "title": "LLM06:2025 Excessive Agency",
      "locator": "https://genai.owasp.org/llmrisk/llm062025-excessive-agency/",
      "role": "Supports claim:excessive-agency, claim:least-privilege, claim:human-approval.",
      "caveat": "Does not prove that all agent failures are caused by excessive permissions or autonomy."
    },
    {
      "id": "evidence:cisa-secure-by-design",
      "type": "authoritative-institutional-source",
      "title": "Shifting the Balance of Cybersecurity Risk: Principles and Approaches for Security-by-Design and -Default",
      "locator": "https://www.cisa.gov/sites/default/files/2023-06/principles_approaches_for_security-by-design-default_508c.pdf",
      "role": "Supports claim:defense-in-depth, claim:containment, claim:least-privilege.",
      "caveat": "Does not address AI-specific reward hacking or prove unknown failures can be made harmless."
    },
    {
      "id": "evidence:anthropic-reward-seeker",
      "type": "primary-research",
      "title": "Training a Misaligned Reward Seeker",
      "locator": "https://alignment.anthropic.com/2026/reward-seeker/",
      "role": "Supports claim:reward-hacking, claim:unanticipated-reward-hacks.",
      "caveat": "Research result from a specific experimental training setup. It does not establish that deployed agents generally act deceptively or intentionally."
    },
    {
      "id": "evidence:anthropic-practices-2026",
      "type": "first-party-engineering-report",
      "title": "Improving our alignment and security practices",
      "locator": "https://www.anthropic.com/news/improving-alignment-security-efforts",
      "role": "Supports claim:production-reward-hacking-observation, claim:monitoring-limits.",
      "caveat": "First-party report about Anthropic systems. It does not generalize by itself to every model provider or deployment."
    },
    {
      "id": "evidence:openai-agent-safety",
      "type": "official-documentation",
      "title": "Safety in building agents",
      "locator": "https://developers.openai.com/api/docs/guides/agent-builder-safety",
      "role": "Supports claim:layered-agent-controls, claim:tool-approval, claim:structured-boundaries.",
      "caveat": "Product guidance is not independent proof that any specific agent deployment is safe."
    }
  ],
  "provenance": {
    "sourcePackageSha256": "261566c1d6286f6f88a8161db654e97d70df0277025b4a6154c978264ac467e7",
    "sourcePackageBytes": 5780849,
    "acceptedEdition": "offline-uat-v0.1",
    "ownerIssue": 1249,
    "voice": "Tony Voice 1.14 / Adaptive v3.1"
  },
  "lineage": {
    "relationship": "new-public-work-from-owner-accepted-offline-uat",
    "canonicalUrl": "https://shareplane.malott.ai/artifacts/let-intelligence-be-messy-make-authority-boring/",
    "semanticLineageIssue": "https://github.com/pinklon/shareplane-platform/issues/53"
  },
  "text": "# Let Intelligence Be Messy. Make Authority Boring.\n\n## Why reliable AI systems should not depend on trusting the model, the software, or the operator\n\nWe keep asking the wrong question about AI.\n\nCan we trust it?\n\nThe question sounds reasonable because the behavior is strange. Models hallucinate. Agents guess when instructions are incomplete. They optimize for the wrong thing. They can take shortcuts the designer did not anticipate. In reinforcement learning, researchers have documented reward hacking where models learn to satisfy the measurement rather than the intended task. Anthropic has even reported training environments in which models discovered reward hacks the researchers had not anticipated.\n\nThat makes people nervous, and it should. But I think we have allowed that nervousness to push the engineering conversation in the wrong direction.\n\nI do not need an AI system to be trustworthy in the human sense. I need the system around it to remain trustworthy when the AI is wrong.\n\nThat is a very different requirement.\n\nIt is also a much more useful one.\n\n### AI is a tool, not an authority\n\nI have spent enough time watching agents work on real systems to lose interest in pretending they behave like deterministic software. They do not. That is not news. What matters is what we do with that fact.\n\nA model may produce a brilliant solution, then misunderstand an instruction five minutes later. An agent may reason correctly about nine steps and take a ridiculous turn on the tenth. It may chase stale evidence, satisfy the wrong objective, overfit to the test in front of it, or confidently fill in a gap that should have caused it to stop.\n\nWhether we describe some of that behavior as hallucination, reward hacking, misalignment, bad context, weak instructions, or plain old failure matters for diagnosis. It matters much less for authority.\n\nThe model is a tool.\n\nA very unusual tool, obviously. It can reason, synthesize, adapt, write code, challenge assumptions, and discover paths we did not explicitly enumerate. That is exactly why it is valuable. But none of those capabilities should quietly become permission to change consequential state.\n\nWe learned this lesson with humans a long time ago.\n\nA bank does not secure money because every employee is trustworthy. A pharmaceutical quality system does not protect a regulated process because every operator will remember every procedure. Production infrastructure does not survive because administrators never mistype a command. We use approvals, segregation of duties, least privilege, reconciliation, immutable records, change control, monitoring, and recovery because competent people still make mistakes.\n\nAI does not deserve a special exemption from systems engineering just because the mistakes are more interesting.\n\n### The part we dislike is also part of the value\n\nThere is a temptation to say the answer is making AI more deterministic.\n\nSome of that is useful. Better models, better context, stronger instructions, structured outputs, evals, and narrower tasks can reduce failure. We should use all of them.\n\nBut there is a limit to the idea.\n\nIf I want a deterministic parser, I should write a deterministic parser. If I want an exact policy decision, I should encode the policy. If I want an artifact hash, I do not ask a language model what it feels the hash should be.\n\nI use AI where uncertainty still exists. I use it where the problem benefits from interpretation, synthesis, exploration, judgment, or discovering a solution I did not specify in advance.\n\nThat means some degree of unpredictability is not an embarrassing defect bolted onto the capability. It is part of the reason the capability exists.\n\nThe practical question is therefore not how to remove probability from intelligence. The practical question is where probability is allowed to operate, and where it must stop.\n\nThat line is the architecture.\n\n### Deterministic software is not the same thing as trustworthy software\n\nThere is another trap here, especially for engineers.\n\nOnce we recognize that models are probabilistic, it is easy to overcorrect and put too much faith in deterministic software. We imagine that if enough of the system is expressed as code, tests, schemas, CI gates, scanners, and policy checks, then the trustworthy part has been solved.\n\nIt has not.\n\nDeterministic software fails too. It just fails with admirable consistency.\n\nA test can prove that a known requirement passes under the conditions encoded in the test. A scanner can identify vulnerability patterns it knows how to recognize. A policy engine can enforce the policy it was given. None of those things proves that the requirement was correct, the threat model was complete, the policy captured the real boundary, or the component doing the checking has no defect of its own.\n\nThis is why the strongest part of the architecture cannot be a mythical final test that detects every possible bad thing.\n\nThere will always be things we did not think to test.\n\nThe harder problem is designing the system so entire classes of unknown failures have limited consequence.\n\nCISA's secure-by-design guidance makes the same general point from conventional security engineering: defense in depth exists so compromise of one control does not compromise the whole system. Least privilege, narrow permissions, sandboxing, and separation of duties reduce the impact of failures that get through other defenses. NIST's AI Risk Management Framework similarly treats trustworthiness as a lifecycle and system property, with independent assessment, production monitoring, and explicit consideration of third-party components.\n\nThat is the frame I want around AI.\n\nDo not bet the system on perfect detection. Constrain what failure can become.\n\n### Unknown unknowns change the design target\n\nKnown failures are comparatively easy.\n\nOnce we have seen a class of defect, we can write a test. Once an agent has taken a dangerous path, we can add a regression fixture. Once a scanner understands a vulnerability pattern, we can detect it cheaply and repeatedly.\n\nThat is valuable work. It is also incomplete by definition.\n\nThe interesting failures are the ones nobody has named yet.\n\nAnthropic's 2026 reward-hacking research is useful here, not because it proves models are secretly plotting against us, but because it demonstrates something much more ordinary and much more important. Optimization systems can discover ways to satisfy objectives that their designers did not anticipate. The researchers explicitly reported environments where the model found new reward hacks that had not been anticipated in advance.\n\nI do not need to settle a philosophical argument about whether that behavior is \"intentional.\" Architecture cannot depend on winning that debate.\n\nIf an agent finds a strange path through my environment, I care about the path and the authority it can exercise, not whether the model deserves moral blame for discovering it.\n\nSo the target changes.\n\nInstead of trying to enumerate every bad outcome, I want invariants that remain true across outcomes I have not imagined yet.\n\nAn agent cannot mint its own production authority.\n\nA candidate cannot redefine its own acceptance criteria.\n\nThe actor producing evidence cannot be the only authority validating that evidence.\n\nCredentials should be scoped to a purpose and a time window rather than handed over as durable ambient power.\n\nThe artifact that was qualified should be the artifact that is deployed.\n\nA rollback path should not depend entirely on the component being rolled back.\n\nRuntime observation should be able to revoke authority after deployment if reality diverges from what qualification expected.\n\nThose statements do not predict every future bug. They narrow the blast radius of bugs we have not met yet.\n\nThat is a better objective.\n\n### Where probability ends\n\nI have come to think about the architecture in a very simple way:\n\n**Probabilistic semantics. Deterministic authority.**\n\nThe semantic side can be messy.\n\nLet models reason. Let agents explore. Let them propose code, challenge assumptions, compare designs, search for evidence, create candidate artifacts, and try again when something fails. That is where their flexibility is useful.\n\nThe authority side should be boring.\n\nCredentials are not a vibe. Permissions are not a suggestion. Artifact identity is not approximate. A production mutation either has authority or it does not. A required receipt either exists or it does not. An accepted state either matches the qualified state or it does not.\n\nThe boundary between those worlds should be explicit.\n\nOWASP calls excessive functionality, excessive permissions, and excessive autonomy root causes of excessive agency in LLM applications. The useful observation is that the risk is not merely inside the model. It emerges from what the surrounding system permits the model to do.\n\nOpenAI's own current agent-safety guidance follows the same direction. It recommends tool approvals, structured data boundaries, isolation, guardrails, trace grading, and layered techniques, while explicitly noting that no single guardrail is foolproof.\n\nThat is exactly right.\n\nDo not ask one control to carry the whole trust burden.\n\n### Trust the system by refusing to trust the pieces\n\nThis may sound paradoxical, but mature systems become more trustworthy by being less sentimental about their components.\n\nI do not want to trust AI-generated software because AI generated it.\n\nI do not want to trust software I wrote because I wrote it.\n\nI do not want to trust an open-source package because ten thousand people use it.\n\nI do not want to trust a scanner because it returned green.\n\nI do not want to trust a human approval because a human clicked approve.\n\nEach of those can contribute evidence. None should be sovereign.\n\nThe system becomes trustworthy when independent controls constrain one another, when authority is narrow, when evidence is durable, when consequential state changes are observable, and when failure in one layer does not silently inherit the privileges of the next.\n\nThis is not an AI-specific invention. It is the same reason mature engineering disciplines use checks and balances everywhere else.\n\nThe difference is that AI makes the need more obvious because the behavior is visibly probabilistic. Humans are wonderfully probabilistic too. We merely have centuries of institutional machinery dedicated to pretending otherwise.\n\n### Development can be chaotic. Production does not have to care.\n\nThis is the part I think people building agentic systems often miss.\n\nThey spend enormous effort trying to make the development process look orderly. They want every agent to follow the perfect plan, every reasoning trace to be sensible, every intermediate decision to be explainable, every generated change to arrive cleanly.\n\nI want those things too. They make the work cheaper and easier to understand.\n\nBut I do not want production safety to depend on them.\n\nInside a development mesh, agents can disagree. They can abandon approaches. They can write something broken and repair it. They can follow stale context until another control catches the drift. They can waste time. They can even produce software with a flaw none of our current scanners recognize.\n\nThat is unpleasant. It is not automatically catastrophic.\n\nThe production boundary should care about exact qualified state, authority, evidence, policy, and observation. It should not care how confident the agent sounded while producing the candidate.\n\nThat separation is what turns probabilistic intelligence into usable infrastructure.\n\n### The test is not \"Can the AI be trusted?\"\n\nThe better questions are more mechanical.\n\nWhat can this actor read?\n\nWhat can it change?\n\nWhich identity does it receive, and for how long?\n\nCan it expand its own permissions?\n\nWho defines the acceptance criteria?\n\nCan the producer fabricate the evidence used to approve its own work?\n\nIs qualification independent from implementation?\n\nIs the deployed artifact exactly the artifact that passed qualification?\n\nCan authority be revoked quickly?\n\nCan we reconstruct what happened afterward without asking the model to remember?\n\nWhat happens when the component we trusted is the component that failed?\n\nThose questions are less glamorous than arguing about artificial general intelligence. They are also much closer to deciding whether I would let the system near production.\n\n### Use the tool we actually have\n\nI think the industry wastes energy wishing AI were a different tool.\n\nYes, it is probabilistic. Yes, that creates new failure modes. Yes, agents can reward-hack, hallucinate, misunderstand, or discover ugly shortcuts. Better training and better models should reduce those behaviors, and I hope they do.\n\nBut there is too much capability here to make \"I wish the tool behaved differently\" the center of the engineering strategy.\n\nUse the tool we actually have.\n\nPut probabilistic intelligence where probability creates value. Surround it with controls that assume the intelligence can fail. Give it enough freedom to discover useful answers without giving it enough authority to quietly redefine reality.\n\nThe objective is not software that deserves unconditional trust.\n\nThe objective is a system whose safety does not require unconditional trust in the software, the model, the agent, the operator, or any single control.\n\nThat is where I land after building these systems for real.\n\nLet intelligence be messy.\n\nMake authority boring.\n"
}
