IG

INFERENCE GATEWAY

Governed AI Agent Execution with Human-in-the-Loop Tool Verification

Author: rng

Apart Research Technical AI Governance Challenge

February 2026

The Governance Gap in Agentic AI

AI systems are evolving from text generators to autonomous agents that execute real-world actions.

Training RLHF, Constitutional AI Inference Content filters, moderation Execution Tool invocation, actions ✓ Solved ✓ Solved ??? GAP Our Focus

Current governance stops at model OUTPUT. We need governance at ACTION EXECUTION.

Three Key Innovations

🔒

Two-Stage Privacy Pipeline

Sensitive data processed locally before cloud inference. PII never leaves your infrastructure.

📱

Human-in-the-Loop Approval

Multi-factor authentication for tool execution. Duo Mobile, webhooks, and manual approval.

🔗

Provider-Agnostic Governance

8 providers, one interface. Same governance rules apply everywhere.

OpenAI Anthropic Azure Gemini Ollama vLLM TGI HF UNIFIED GOVERNANCE LAYER

Microservices Architecture

Client Applications (Chat UIs, IDEs, Automation) Gateway :3002 Unified LLM API 8 providers, 5 strategies API :3001 Management Auth, Keys, Projects GraphQL :3005 Queries Traces, Analytics Runner :3003 Workflow Execution LangGraph-style DAG Memory :3004 Vector Store RAG, Embeddings PostgreSQL pgvector Redis

Two-Stage Privacy Pipeline

User Input (may contain PII, credentials) STAGE 1: LOCAL PROCESSING (Ollama / vLLM / TGI) PII Detection + Redaction Credential Extraction Sensitive Tool Execution Sanitized Context STAGE 2: CLOUD INFERENCE (GPT-4 / Claude / Gemini) Complex Reasoning Response Generation Tool Selection Back to Local for Secure Execution

✓ Data Minimization

Only sanitized context reaches cloud providers

✓ Regulatory Compliance

HIPAA, GDPR, SOX compatible

Human-in-the-Loop Tool Approval

AI requests tool execution "Update database..." Risk Level? Low Auto Execute immediately Read-only operations Medium Manual In-app confirmation Development mode High 📱 Duo Push Mobile notification Production operations Critical 🔗 Webhook External system Compliance required

We treat AI agent actions with the same rigor as financial transactions. MFA for AI.

Duo Mobile Integration

ENROLLMENT FLOW 1. User initiates enrollment 2. Generate QR code (local) 3. Scan with Duo Mobile 4. Device registered ✓ RUNTIME APPROVAL AI Agent "Execute trade..." tool request Gateway Hold until approved push 📱 Duo Mobile Approve Deny response Execution continues

Risk-Based Approval Escalation

Risk Level Default Mode Override Audit Example
Low Auto Yes Optional Read public API
Medium Manual Yes Recommended Send notification
High Duo / Webhook Admin only Required Database write
Critical Duo + Webhook No Required + Alert Financial transaction
🔐

Defense in Depth

Critical operations require BOTH Duo approval AND webhook confirmation from compliance systems.

📊

Complete Audit Trail

Every approval decision is logged with timestamp, approver, and context for compliance.

Five Routing Strategies

1. Default Direct routing 2. Weighted 70% A | 20% B | 10% C 3. Round-Robin Sequential rotation A → B → C → A... 4. Failover Primary/backup chain A ✗ → B ✗ → C ✓ 5. Sticky Session affinity User X → always A Governance Implications: Failover Routing Creates audit trail of provider attempts before success Sticky Routing Ensures data residency - context stays with one provider Round-Robin Deterministic distribution for reproducible audits

Comprehensive Audit Trail

Trace: abc123-def456-789 ├── Gateway.receiveRequest ─────────────────────────────────── 2ms ├── Gateway.routeToProvider ────────────────────────────────── 1ms ├── Provider.openai.inference ─────────────────── 1847ms ├── Runner.executeWorkflow ────────────── 523ms ├── ToolApproval.duo.request ───────────────────── 8234ms ◀ HUMAN ├── Tool.database.execute ─── 127ms └── Gateway.sendResponse ───────────────────────────────────── 3ms ✓ Queryable via GraphQL Export to Jaeger/Zipkin

Defense in Depth Security

NETWORK LAYER Cloudflare Tunnel • mTLS between services • Rate limiting AUTHENTICATION LAYER API keys • Virtual keys • SSO (OIDC/OAuth2) • Duo MFA AUTHORIZATION LAYER Project-based ACL • Role-based permissions • Agent tool permissions DATA LAYER AES-256-GCM encryption • PII redaction • Audit logging • Retention policies Request flow

Real-World Applications

💰

Financial Services

Trade Execution Agent

  • Cloud: GPT-4 for analysis
  • Local: Llama for PII
  • Trade execution: Duo + Webhook

→ No trade without dual approval

🏥

Healthcare

Clinical Decision Support

  • LOCAL ONLY (PHI protection)
  • EHR read: Auto
  • Prescription: DENY always

→ AI assists, never prescribes

⚖️

Legal

Contract Analysis

  • Cloud: Claude for analysis
  • Local embeddings only
  • Client comms: Duo required

→ Attorney-client privilege preserved

🏭

Manufacturing

Process Automation

  • Air-gapped (Ollama only)
  • Monitoring: Auto
  • Emergency stop: DENY

→ Human oversight on all changes

Why This Matters for AI Governance

Training "Don't learn bad things" Inference "Don't say bad things" Execution "Don't DO bad things" ← OUR FOCUS
Property How We Address It
VerificationEvery tool call verified before execution
AuditabilityComplete OpenTelemetry traces
ComplianceSOC 2, SOX, HIPAA-compatible logging
ReversibilityDeny actions BEFORE they happen
AccountabilityClear approval chain (who approved what)
ProportionalityRisk-based escalation, not all-or-nothing

Key Takeaways

As AI systems gain agency, governance must shift from
controlling OUTPUTS to controlling ACTIONS.

🔒

Privacy

Two-stage pipeline keeps sensitive data local

✓

Verification

Human approval before high-stakes actions

📊

Auditability

Complete trace of every decision

Treating AI agent actions with the same rigor as
financial transactions is not just possible—it's practical.

IG

Thank You

Questions?

Author: rng

Apart Research Technical AI Governance Challenge

February 2026

1 / 15