← Back to plan All pages
08

Production Multi-Agent Workflows with Google ADK

Build a regulated-industry multi-agent system in Python with Google's Agent Development Kit, from first tool to a deployed, guarded, low-latency service. Includes a lifecycle ownership matrix (design, build, test, deploy, operate, feedback) across domain, data science, platform, risk and SRE teams.

GuidePythonGoogle ADKBankingHealthcare~35 min read

1The short version

What you are building, what to decide first, and what I would not skip.

Recommendation

Build the smallest agent graph that works (one coordinator, two or three specialists, deterministic code wherever a decision doesn't need language), put every control outside the model where it can't be talked out of its job, and treat latency as a design constraint from day one rather than a tuning exercise in month four.

Google's Agent Development Kit (ADK) gives you the pieces: LlmAgent for reasoning, workflow agents (SequentialAgent, ParallelAgent, LoopAgent) for control flow you want to be deterministic, a Runner and session service for state, and callbacks that run before and after every model call, tool call and agent. It is code-first Python, which matters in regulated work: your control logic lives in version control, gets code review and has unit tests.

We build one worked example end to end: a retail-bank assistant that answers spending questions and files card disputes. A healthcare variant (post-discharge medication questions, FHIR-backed) swaps the tools and tightens the controls; the architecture is the same. The code is deliberately compact so the shape is visible, and you will need to harden each piece for your own environment.

What you'll build

A coordinator with three specialists, a parallel evidence-gathering step, guardrails in callbacks, a streaming API and a Cloud Run deployment.

Where it gets hard

Not the agent code. Identity propagation, write-action approval, audit, and keeping p95 latency tolerable with guardrails switched on.

Assumptions

Vertex AI as the model backend, Python 3.11+, Cloud Run as the runtime, a UK or EU region. Model names move fast, so pin whatever is GA when you read this.

Check versions

ADK ships often and some APIs here (plugins, tool confirmation, streaming behaviour inside callbacks) have changed between minor versions. Pin google-adk in your lockfile, read the release notes before upgrading, and let your eval suite (guide 09) be the thing that tells you an upgrade is safe.

2Lifecycle and ownership at a glance

6 phases, 45 activities, who owns each. Jump to a phase:

ActivityDomainData sciencePlatformRisk & compl.SRE / supportOutput
1 · Design Decide what to build, how risky it is and how it is shaped
Define use case, success metrics, in/out of scopeACICIScope doc, KPIs
Risk tier and regulatory mapping (SS1/23, HIPAA, DCB0129, AI Act)RCCAITier, control set
Agent graph and routing (one agent or many)CACIAgent graph, routing table
Tool catalogue: read/write split, approval points, limitsRRACITool contracts
Data access, privacy and residency (DPIA, retention, PHI/PCI)CCRADPIA, data-flow diagram
Escalation, handoff and fallback journeysACICCHandoff rules, fixed safety text
Latency, cost and capacity budgetCRACPer-stage budget
Clinical safety case / model-risk entry and hazard logRRCAISafety case, hazard log
2 · Build Write the agent, tools, controls and infrastructure
Prompts and instructions, versioned in the repoCAIIPrompt versions
ADK agents, workflows, state and sessionsIACAgent code
Tool wrappers and backend integrationsARCITyped tools, fakes for tests
Knowledge and retrieval content (approved leaflets, policy)ACICCurated, versioned corpus
Guardrail plugins and callbacks (input, output, tool policy)ICACShared guard library
Identity, API exposure, gateway, quotasIACCAuthenticated streaming API
Infrastructure as code: Cloud Run, IAM, VPC-SC, CMEK, loggingACCTerraform module call
Model, library and config pinning; secretsRAIILockfiles, pinned model IDs
3 · Test Prove it behaves before a customer or patient sees it
Unit tests for tools and guards (no model)ACFast CI tests
Golden dataset and domain eval cases from real transcriptsARICVersioned datasets
Eval harness, baseline suites, judge calibrationCRACEval runner, calibrated judges
Trajectory and routing evalsCACRouting confusion matrix
Red-team: injection via data fields, impersonation, scope creepCRRAAttack set, findings
Latency and load test against the budgetCRAPerf report
SME review of sampled outputs (clinicians, claims, compliance)ACCLabelled sample
Release gate sign-off by tierRRRAISigned gate record
4 · Deploy Ship safely and be able to undo it
CI/CD pipeline: evals, plan, policy checks, attestationCACIReusable pipeline
Release approval and change recordARRCIApproved change
Shadow, canary and rollback planCRRARollout plan, rollback tested
Runbook, on-call rota, escalation contactsRCCARunbook
Staff and user communication, training, disclosuresACCITraining, notices
5 · Operate and observe Keep it healthy, visible and auditable
Tracing, structured logs, dashboards (OpenTelemetry, Cloud Trace, BigQuery)CAIRPer-turn traces
SLOs and alerts: latency, errors, guard triggers, escalations, costCCRAAlert policy
Incident response: page, triage, kill switch, post-incident reviewRRRCAIncident record
Cost, quota and capacity managementICARBudget reports
Model and provider changes, deprecations, upgradesIRACIUpgrade plan, re-run evals
Security monitoring, access reviews, key rotationRACReview records
Audit trail retention and regulator or customer requestsCRAIQueryable audit store
6 · Feedback and improve Turn live behaviour into fixes, tests and better models
Capture feedback: thumbs, edits, escalations, nurse or agent tags, outcomesCCAIFeedback events
Sampled online evals with LLM judge on live trafficIARCOnline scores by version
Human review queue for flagged and sampled conversationsACCLabels, corrected replies
Judge calibration against human labels (weekly or monthly)RACTPR/TNR report
Triage failures into fixes: content, prompt, tool or guardRACIFix backlog
Turn every confirmed failure into an eval caseRACNew regression cases
Preference data, tuning or RL candidates behind the release gateCARCCandidate model, eval evidence
Drift review and quarterly threshold and metric reviewARCCIUpdated thresholds
Re-tier, expand scope or retire the agentACCRIDecision record

A accountable: owns the outcome, one per row, usually does the work. R does part of the work. C consulted before it is done. I told afterwards. Blank means no role.

  • Domain is the business or product owner plus subject-matter experts (clinicians, claims, relationship managers, operations reviewers).
  • Data science covers ML and AI engineers who build and evaluate the agent. Platform is the shared engineering team (cloud, CI/CD, guard library). SRE / support runs it in production and carries the pager.
  • Small teams collapse columns (one team may be Data science and Platform). Keep the single accountable owner per row even then.
  • For tier-1 agents, Risk & compliance becomes accountable for release sign-off and for any change to thresholds. Everyone else's role stays as shown.

3What regulated industries change

The same agent, a different set of questions from your risk function.

In an unregulated product the question is "does it work?". In banking and health it becomes "can you prove what it did, why, for whom, and that it stayed inside its remit?" That moves a handful of design decisions early.

Banking
  • Model risk management. UK PRA SS1/23, US SR 11-7. An LLM agent is a model: inventory it, tier it, validate it, monitor it.
  • Consumer Duty / conduct. Outcomes for retail customers must be good and evidenced. No steering, no advice you aren't authorised to give.
  • Operational resilience. DORA in the EU, PRA/FCA rules in the UK: tolerances, exit plans, third-party concentration on the model provider.
  • PCI DSS, GDPR/UK GDPR. Card numbers never reach the model or the logs. Lawful basis and retention for chat transcripts.
  • EU AI Act. Creditworthiness assessment is classed high-risk; a servicing assistant usually isn't, but confirm with legal.
Healthcare
  • PHI handling. HIPAA (US) or UK GDPR special-category data plus NHS DSPT. Sign the BAA, use only covered services, minimum necessary data.
  • Clinical safety. In England, DCB0129/DCB0160 with a named Clinical Safety Officer and a hazard log. A hazard log is where "agent gives wrong dose advice" lives.
  • Medical device boundary. If it informs diagnosis or treatment decisions it may be SaMD (MHRA, FDA, EU MDR). Scope the product to stay clear of that line, or go through it deliberately.
  • Human oversight. A clinician accountable for anything clinical. The agent routes, informs and drafts; it doesn't decide.
  • Interoperability. FHIR via Cloud Healthcare API; consent and audit are part of the data layer.

Three design consequences follow, and the rest of this guide leans on them:

  1. Identity comes from the platform, never from the model. The customer or patient ID is set by your API layer from a verified token. A tool takes it from session state, not from an argument the LLM can fill in.
  2. Read and write are different trust levels. Reads can be autonomous inside a scope. Writes need an explicit, out-of-band confirmation from the user, and sometimes from a human at your end.
  3. Everything is reconstructable. Prompt version, model version, tool calls with arguments, guard verdicts, who approved. Stored somewhere that a regulator-facing person can query.
Not legal advice

The regimes above are pointers for the conversation with compliance, not a compliance analysis. Check each Google Cloud service you use against its current compliance scope (HIPAA BAA coverage, data residency, VPC Service Controls support) before the design is signed off. That list changes, and the answer for the model endpoint can differ from the answer for the session store.

4Architecture and agent graph

Decide how many agents you need before you write one.

Request path
Clientweb / app / IVR
→
API gatewayApigee · authN · quotas · WAF
→
Agent APICloud Run · FastAPI · ADK Runner
→
Input guardinjection · PII · scope
→
Conciergeroutes only
→
Spend insights read-only
Dispute flow reads, then 1 guarded write
Human handoff
→
Output guardclaims · PII · tone
Backends
Session storeFirestore / Cloud SQL
Core banking API / FHIR storeprivate connectivity, scoped SA
Policy corpusVertex AI Search
Audit logCloud Logging → BigQuery
TracesOpenTelemetry → Cloud Trace

One agent or many?

Most "multi-agent" systems should have been one agent with good tools. Split when one of these is true, not because the diagram looks impressive:

Reason to splitExample hereAlternative to consider
Different permissionsSpend insights can only read; dispute filer has one write toolNone. This is the strongest reason.
Different latency or model needsRouter on a small fast model, answerer on a larger oneOne agent with a model callback that swaps models
Prompt too large or contradictoryDispute policy is 3k tokens of rules that confuse small-talk handlingRetrieval of policy on demand as a tool
Independent steps that can run concurrentlyTransaction lookup and fraud-signal checkPlain asyncio.gather inside one tool
Separate ownershipCards team owns disputes, savings team owns insightsRemote agents over A2A

ADK's building blocks, in the order you will reach for them

PieceUse it forNotes
LlmAgentAnything needing language understanding or tool selectionModel decides what to call. Non-deterministic by nature.
SequentialAgentA fixed pipeline: gather → decide → actSub-agents share session state; pass data with output_key.
ParallelAgentIndependent lookups that should overlapBranches get isolated invocation paths; write to different state keys.
LoopAgentDraft → critique → revise, with a hard max_iterationsAlways cap iterations; each loop costs seconds.
sub_agents + transferLLM-driven routing to a specialistRouting quality depends on each agent's description.
AgentToolCall a specialist as a function and keep controlBetter than transfer when the parent must post-process the answer.
Callbacks / pluginsGuardrails, audit, caching, metricsPlugins apply across all agents in a Runner; callbacks are per agent.

5Project setup

A layout that survives a second team joining.

terminal
python3.11 -m venv .venv && source .venv/bin/activate
pip install "google-adk" fastapi "uvicorn[standard]" pydantic google-cloud-logging

# Vertex AI as the backend (not an API key): auth is a service account, quota and
# data-governance terms are your GCP project's.
export GOOGLE_GENAI_USE_VERTEXAI=TRUE
export GOOGLE_CLOUD_PROJECT=my-bank-agents-dev
export GOOGLE_CLOUD_LOCATION=europe-west2    # check model availability in your region
gcloud auth application-default login
layout
bank_assist/
├── __init__.py            # from . import agent   (ADK discovers root_agent here)
├── agent.py               # assembles root_agent
├── agents/
│   ├── concierge.py
│   ├── insights.py
│   └── disputes.py
├── tools/
│   ├── bank_api.py        # thin typed client for the core banking API
│   └── tools.py           # functions exposed to the model
├── guardrails.py          # callbacks: input, output, tool policy, audit
├── config.py              # model names, limits, prompt version
├── server.py              # FastAPI + Runner (the production entrypoint)
├── tests/                 # unit tests for tools + guards, evalsets for agents
└── Dockerfile

Run adk web from the parent directory for a local dev UI with event traces, and adk run bank_assist for a terminal loop. The dev UI is invaluable for seeing which sub-agent got the turn and what each tool returned, and it is not something to ship.

config.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    # Pin concrete GA model IDs; don't use "latest" aliases in a regulated system,
    # because behaviour changing under you breaks validation evidence.
    router_model: str = "gemini-2.5-flash-lite"
    answer_model: str = "gemini-2.5-flash"
    prompt_version: str = "2026-10-01.3"       # logged on every turn
    max_dispute_amount_gbp: float = 2_500.0    # business rule, enforced in code
    max_tool_calls_per_turn: int = 6

CFG = Config()

6Tools: where the real risk lives

The model proposes. Your function disposes.

ADK turns a typed Python function with a good docstring into a tool. The docstring is the prompt the model sees, so write it for the model: what it does, when to use it, what the arguments mean, what comes back. Three rules I would hold the line on:

  • No identity arguments. The customer ID isn't a parameter. It comes from tool_context.state, set by your server after token verification.
  • Return small, structured, pre-redacted data. Return the five fields the answer needs, not the 60-field record. Cheaper, faster, and nothing sensitive reaches the model by accident.
  • Return errors as data. {"status": "error", "message": ...} lets the model explain or escalate. An exception mid-turn gives the customer a stack trace or a hang.
tools/tools.py
from google.adk.tools.tool_context import ToolContext
from . import bank_api   # thin client; owns auth, timeouts, retries


async def get_recent_transactions(days: int, tool_context: ToolContext) -> dict:
    """Get the signed-in customer's card transactions for the last N days.

    Use for questions about spending, a specific payment, or before filing a dispute.

    Args:
        days: How far back to look. Between 1 and 90.

    Returns:
        {"status": "ok", "transactions": [{"id", "date", "merchant", "amount_gbp", "status"}]}
        or {"status": "error", "message": str}.
    """
    customer_id = tool_context.state["customer_id"]       # set server-side, not by the model
    days = max(1, min(days, 90))
    try:
        rows = await bank_api.transactions(customer_id, days=days)
    except bank_api.Unavailable:
        return {"status": "error", "message": "Transactions are temporarily unavailable."}
    # Mask on the way out: the model never needs a PAN or a full account number.
    return {"status": "ok", "transactions": [
        {"id": r.id, "date": r.date.isoformat(), "merchant": r.merchant,
         "amount_gbp": r.amount, "status": r.status} for r in rows[:50]]}


async def get_fraud_signals(transaction_id: str, tool_context: ToolContext) -> dict:
    """Check whether a transaction triggered any fraud-detection signals.

    Args:
        transaction_id: An id returned by get_recent_transactions.
    """
    customer_id = tool_context.state["customer_id"]
    sig = await bank_api.fraud_signals(customer_id, transaction_id)
    return {"status": "ok", "flagged": sig.flagged, "reason_codes": sig.reason_codes}


async def open_card_dispute(transaction_id: str, reason: str, tool_context: ToolContext) -> dict:
    """File a dispute for ONE transaction. Only call after the customer has confirmed in the UI.

    Args:
        transaction_id: The id of the disputed transaction.
        reason: One of "not_recognised", "duplicate", "not_received", "wrong_amount".
    """
    customer_id = tool_context.state["customer_id"]
    case = await bank_api.open_dispute(
        customer_id, transaction_id, reason,
        idempotency_key=f"{tool_context.state['session_key']}:{transaction_id}",  # retries can't double-file
    )
    return {"status": "ok", "case_id": case.id, "next_step": case.next_step}


def escalate_to_human(summary: str, tool_context: ToolContext) -> dict:
    """Hand the conversation to a human agent. Use when the customer asks, is distressed,
    reports vulnerability, or you cannot help safely.

    Args:
        summary: Two sentences for the human agent. No account numbers.
    """
    tool_context.actions.escalate = True
    tool_context.state["handoff_summary"] = summary
    return {"status": "ok", "message": "A colleague will pick this up."}
In practice

Most agent incidents I have seen trace back to a tool that was too powerful, not a prompt that was too weak: a "run query" tool, a "send message" tool with free-form recipients, or an identity argument the model could fill in. A narrow tool that can only do one thing is the cheapest control you will ever buy.

7Agents and workflows

Language where you need it, deterministic flow where you don't.

The specialists

agents/insights.py
from google.adk.agents import LlmAgent
from google.genai import types
from ..config import CFG
from ..tools.tools import get_recent_transactions
from ..guardrails import tool_policy, audit_after_tool

spend_insights = LlmAgent(
    name="spend_insights",
    model=CFG.answer_model,
    # The description is what the coordinator reads when routing. Write it like a job ad.
    description="Answers questions about the customer's spending, merchants and recent card payments. Read-only.",
    instruction="""You help a retail banking customer understand their card spending.
- Use get_recent_transactions for facts. Never state an amount or merchant you did not get from a tool.
- You are not a financial adviser. Do not recommend products, investments or borrowing.
- If the customer wants to dispute a payment, hand back to the coordinator.
- Be brief. Plain English. Amounts in pounds.""",
    tools=[get_recent_transactions],
    before_tool_callback=tool_policy,
    after_tool_callback=audit_after_tool,
    generate_content_config=types.GenerateContentConfig(temperature=0.2, max_output_tokens=600),
)

A pipeline with a parallel step

Disputes are a fixed process, so the control flow should be code, not a prompt. Gather evidence in parallel, then a final agent decides whether to propose a dispute. The write tool is only reachable at the last step.

agents/disputes.py
from google.adk.agents import LlmAgent, ParallelAgent, SequentialAgent
from ..config import CFG
from ..tools.tools import get_recent_transactions, get_fraud_signals, open_card_dispute
from ..guardrails import tool_policy, audit_after_tool

txn_lookup = LlmAgent(
    name="txn_lookup", model=CFG.router_model,
    description="Finds the transaction the customer is talking about.",
    instruction="Find the one transaction the customer means in the last 60 days. "
                "If several match, return the candidates. Output JSON only.",
    tools=[get_recent_transactions],
    output_key="candidate_txns",              # lands in session state
    before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)

fraud_check = LlmAgent(
    name="fraud_check", model=CFG.router_model,
    description="Checks fraud signals on a transaction.",
    instruction="For the transaction in the user's message, call get_fraud_signals. Output JSON only.",
    tools=[get_fraud_signals],
    output_key="fraud_result",
    before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)

gather_evidence = ParallelAgent(name="gather_evidence", sub_agents=[txn_lookup, fraud_check])

dispute_filer = LlmAgent(
    name="dispute_filer", model=CFG.answer_model,
    description="Explains findings and files a dispute once the customer has confirmed.",
    # {candidate_txns} and {fraud_result} are filled from session state.
    instruction="""Evidence so far:
transactions: {candidate_txns}
fraud signals: {fraud_result}

1. Tell the customer what you found, in two or three sentences.
2. If exactly one transaction matches and they want to dispute it, ask them to confirm
   with the on-screen button. Do not call open_card_dispute until the system says confirmed.
3. If anything is ambiguous, ask one clarifying question. Never guess a transaction.""",
    tools=[open_card_dispute],
    before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)

dispute_flow = SequentialAgent(
    name="dispute_flow",
    description="Handles a customer's request to dispute or query a specific card payment they don't recognise.",
    sub_agents=[gather_evidence, dispute_filer],
)

The coordinator

agent.py
from google.adk.agents import LlmAgent
from google.adk.tools import FunctionTool
from google.adk.planners import BuiltInPlanner
from google.genai import types
from .config import CFG
from .agents.insights import spend_insights
from .agents.disputes import dispute_flow
from .tools.tools import escalate_to_human
from .guardrails import input_guard, output_guard, prefetch_context

root_agent = LlmAgent(
    name="concierge",
    model=CFG.router_model,
    description="Front door for the retail banking assistant.",
    instruction="""You route. You do not answer account questions yourself.
- Spending, merchants, "what did I spend": transfer to spend_insights.
- A payment they don't recognise or want to dispute: transfer to dispute_flow.
- Asks for a person, is upset, mentions a bereavement, financial difficulty or fraud in progress: call escalate_to_human.
- Anything outside card spending and disputes: say what you can help with. No advice on products.""",
    sub_agents=[spend_insights, dispute_flow],
    tools=[FunctionTool(escalate_to_human)],
    planner=BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_budget=0)),   # routing needs no reasoning
    generate_content_config=types.GenerateContentConfig(temperature=0, max_output_tokens=256),
    before_agent_callback=[input_guard, prefetch_context],     # a list runs in order; stops at the first that returns content
    after_model_callback=output_guard,
)
Routing is your first failure mode

LLM routing works until two descriptions overlap. Write routing cases into your eval set on day one (guide 09: trajectory evals), and keep the agent count low enough that you can hold the whole routing table in your head. If a rule is obvious (a button click that means "dispute"), route on it in code and skip the model entirely.

Healthcare variant

Same graph, different tools and a tighter remit. For a post-discharge assistant: get_discharge_summary and get_medication_list (FHIR reads through Cloud Healthcare API with the patient's consent scope), search_patient_leaflets (retrieval over approved patient-information content only), and escalate_to_clinician. No tool that changes a medication, ever. The instruction states the boundary plainly: explain what is in the discharge plan; for anything about changing doses, new symptoms or "is this normal", route to a nurse line and give the red-flag advice the clinical team has approved. Red-flag symptom lists should be fixed text approved by your clinical safety officer, served by a tool, not generated.

8State, sessions and memory

Three scopes, three lifetimes, one retention policy to agree with legal.

ScopeKey prefixLifetimePut here
Turntemp:One invocationIntermediate values you don't want persisted
SessionnoneOne conversationcustomer_id, confirmation flags, evidence from the pipeline
Useruser:Across sessionsPreferences (language, accessibility needs). Not transcripts, not health details.
Appapp:GlobalFeature flags, banner text

InMemorySessionService is for tests. In production use DatabaseSessionService (SQLAlchemy URL to Cloud SQL or AlloyDB) or Vertex AI's managed session service if you deploy on Agent Engine. Whichever you pick, put the database in the same region as the Cloud Run service (a cross-region read per turn is 20-100ms you didn't need), encrypt with CMEK if your policy requires it, and set a TTL. Long-term memory is a separate decision: for banking I would store nothing the customer didn't ask us to remember, and for health I would not enable cross-session LLM memory without a documented lawful basis and clinical sign-off.

Session history is also your biggest latency and cost lever, covered in section 10. Keep it short on purpose.

9Controls: guardrails that the model can't talk its way past

Layered, mostly deterministic, and each one independently testable.

Where each control sits in a turn
1 · Edgeauth, rate limit, size cap, WAF
→
2 · Input guardinjection, PII, off-scope
→
3 · Modelinstruction + tools in scope
→
4 · Tool policyallow-list, args, caps, approval
→
5 · Output guardclaims, PII, advice
→
6 · Auditwho, what, why, version

Input guard and context prefetch

Screen the incoming message before it reaches a model. Google Cloud's Model Armor screens prompts and responses for injection, jailbreak and sensitive-data patterns, and Sensitive Data Protection (DLP) handles detection and redaction of PAN, NHS/SSN numbers and similar. The wrapper below is the seam: swap the stub for those clients, keep the callback shape.

guardrails.py
import asyncio, hashlib, json, logging, re
from typing import Optional
from google.adk.agents.callback_context import CallbackContext
from google.adk.models.llm_response import LlmResponse
from google.adk.tools.base_tool import BaseTool
from google.adk.tools.tool_context import ToolContext
from google.genai import types
from .config import CFG
from .tools import bank_api

log = logging.getLogger("audit")                     # shipped to Cloud Logging as JSON
PAN = re.compile(r"\b(?:\d[ -]?){13,19}\b")


def _say(text: str) -> types.Content:
    return types.Content(role="model", parts=[types.Part(text=text)])


async def screen(text: str) -> dict:
    """Seam for Model Armor sanitize_user_prompt + Sensitive Data Protection.
    Return {"blocked": bool, "reason": str, "redacted": str}."""
    redacted = PAN.sub("[card number removed]", text)
    return {"blocked": False, "reason": "", "redacted": redacted}


async def input_guard(callback_context: CallbackContext) -> Optional[types.Content]:
    """Runs before the coordinator. Returning Content short-circuits the agent."""
    text = "".join(p.text or "" for p in (callback_context.user_content.parts or []))
    if len(text) > 2_000:
        return _say("That message is too long for me to handle. Could you shorten it?")
    verdict = await screen(text)
    _audit("input_guard", callback_context, blocked=verdict["blocked"], reason=verdict["reason"])
    if verdict["blocked"]:
        return _say("I can't help with that request. I can help with your card spending and disputes.")
    if verdict["redacted"] != text:
        callback_context.state["input_had_pii"] = True
    return None


async def prefetch_context(callback_context: CallbackContext) -> Optional[types.Content]:
    """Fetch what almost every turn needs, concurrently, before any LLM call (see section 10)."""
    cid = callback_context.state["customer_id"]
    summary, vulnerable = await asyncio.gather(bank_api.summary(cid), bank_api.vulnerability_flag(cid))
    callback_context.state["account_summary"] = summary
    callback_context.state["vulnerable_customer"] = vulnerable     # drives tone and escalation rules
    return None

Tool policy: the control that matters most

guardrails.py (continued)
READ_ONLY = {"get_recent_transactions", "get_fraud_signals"}
WRITES = {"open_card_dispute"}
ALLOWED_BY_AGENT = {
    "spend_insights": {"get_recent_transactions"},
    "txn_lookup": {"get_recent_transactions"},
    "fraud_check": {"get_fraud_signals"},
    "dispute_filer": {"open_card_dispute"},
    "concierge": {"escalate_to_human"},
}
DISPUTE_REASONS = {"not_recognised", "duplicate", "not_received", "wrong_amount"}


def tool_policy(tool: BaseTool, args: dict, tool_context: ToolContext) -> Optional[dict]:
    """Return a dict to block the call and hand that dict to the model as the tool result."""
    agent = tool_context.agent_name
    if tool.name not in ALLOWED_BY_AGENT.get(agent, set()):
        return {"status": "error", "message": "Tool not permitted for this agent."}

    n = tool_context.state.get("tool_calls_this_turn", 0) + 1
    tool_context.state["tool_calls_this_turn"] = n
    if n > CFG.max_tool_calls_per_turn:
        return {"status": "error", "message": "Too many steps. Hand over to a colleague."}

    if tool.name == "open_card_dispute":
        # 1. Out-of-band approval. The UI button sets this via a state delta; the model can't.
        if tool_context.state.get("confirmed_txn") != args.get("transaction_id"):
            return {"status": "needs_confirmation",
                    "message": "Ask the customer to confirm with the on-screen button."}
        # 2. Argument validation in code.
        if args.get("reason") not in DISPUTE_REASONS:
            return {"status": "error", "message": f"reason must be one of {sorted(DISPUTE_REASONS)}"}
        # 3. Business limits the model doesn't get a vote on.
        if tool_context.state.get("txn_amount_gbp", 0) > CFG.max_dispute_amount_gbp:
            return {"status": "needs_human", "message": "Over the automated limit. Escalate to a colleague."}
    return None


def audit_after_tool(tool: BaseTool, args: dict, tool_context: ToolContext, tool_response: dict) -> Optional[dict]:
    _audit("tool_call", tool_context, tool=tool.name, status=tool_response.get("status"),
           args_hash=hashlib.sha256(json.dumps(args, sort_keys=True).encode()).hexdigest()[:16])
    return None


async def output_guard(callback_context: CallbackContext, llm_response: LlmResponse) -> Optional[LlmResponse]:
    """Last look at what the model wants to say."""
    if not llm_response.content or not llm_response.content.parts:
        return None
    text = "".join(p.text or "" for p in llm_response.content.parts)
    if PAN.search(text) or re.search(r"\b(guaranteed|risk[- ]free|you should (invest|borrow))\b", text, re.I):
        _audit("output_blocked", callback_context)
        return LlmResponse(content=_say("I'm not able to say that. A colleague can help; shall I connect you?"))
    return None


def _audit(event: str, ctx, **fields):
    """Structured, PII-free. Prompt version and model are what let you reproduce a decision later."""
    log.info(json.dumps({
        "event": event, "agent": getattr(ctx, "agent_name", ""),
        "session": ctx.state.get("session_key"), "customer": ctx.state.get("customer_ref"),   # pseudonymous ref
        "prompt_version": CFG.prompt_version, **fields}))
Why callbacks and not the prompt

A prompt says "never file a dispute without confirmation" and hopes. The callback makes it impossible: the tool function never runs. For anything a regulator or a customer's solicitor could ask about, the control belongs in code with a unit test beside it.

Human approval for the write

The confirmation flag must come from your UI, not from anything the model wrote. In the API layer, when the user taps "Yes, file this dispute", append a state delta to the session before the next turn:

server.py (excerpt)
import time
from google.adk.events import Event, EventActions

async def record_confirmation(session_service, session, txn_id: str, amount_gbp: float):
    await session_service.append_event(session, Event(
        invocation_id=f"confirm-{txn_id}", author="user", timestamp=time.time(),
        actions=EventActions(state_delta={"confirmed_txn": txn_id, "txn_amount_gbp": amount_gbp}),
    ))

For higher-risk actions, put a human at your end: the tool writes a pending case to a queue, an operations colleague approves, and the customer is told it is under review. ADK's long-running tool pattern fits that shape (the agent returns a ticket and resumes when the result arrives).

Controls checklist by layer

LayerControlBanking emphasisHealth emphasis
IdentityVerified JWT; ID set in session state; step-up auth for writesSCA / open-banking style step-upPatient proxy / carer consent rules
NetworkPrivate egress, VPC Service Controls perimeter, no public model endpoints from the agentPCI scope isolationPHI never leaves perimeter
DataMinimum fields, redaction on the way into prompts and logs, CMEK, retention TTLsNo PAN, mask account numbersDe-identify before analytics; no PHI in logs
ModelPinned model version, safety settings, output length capsNo product advice; suitability stays humanNo diagnosis or dosing; fixed red-flag text
ToolsAllow-list per agent, arg validation, caps, idempotency keysPer-action limits, dual control over thresholdRead-only by default; clinician approval for any write
BehaviourEscalation triggers, vulnerability handlingVulnerable customers (FCA FG21/1)Safeguarding, crisis language, self-harm routing
OperationsAudit trail, kill switch, budget alerts, circuit breaker, incident runbookComplaint handling, record-keepingClinical safety hazard log, incident reporting
AssuranceOffline evals gating release; sampled online evals; red-team before launchModel validation pack (SS1/23)Clinical safety case evidence
Kill switch

Put a feature flag in app: state (or Remote Config) that the API checks before every turn. Three levels worth having: degraded (disable writes), read-only answers from static FAQ, off and route to humans. You will want it the first time a provider incident or a bad prompt release happens on a Friday.

10Making it fast for consumers

Ranked by how much time each change usually saves. Do them in this order.

A consumer will wait about a second for something to start happening and a handful of seconds for the full answer. Agents burn latency in sequential LLM calls (routing call, tool-choice call, answer call), each with its own time-to-first-token plus generation. Controls add their own cost. So the target isn't "make the model faster", it's "make fewer model calls, make each smaller, and show progress while the rest happens".

#TechniqueTypical effectTrade-off
1Take LLM calls off the critical path. Route obvious intents in code; prefetch data you know you'll need.Removes 0.5-2s per avoided callMore code to maintain; test the deterministic paths
2Stream tokens. Send the first sentence as soon as it exists.Perceived latency drops from seconds to ~sub-secondOutput guard must work on chunks (buffer by sentence)
3Right-size models. Flash-Lite for routing and extraction, Flash for the answer, bigger models only where evals show you need them.2-5x faster on routed stepsEvals per tier; more model versions to pin
4Parallelise independent work. ParallelAgent, asyncio.gather, parallel function calls.Latency = slowest branch, not the sumShared-state key discipline; concurrency limits on core APIs
5Switch off thinking for simple turns. thinking_budget=0 on router and FAQ agents.Hundreds of ms to secondsLower quality on reasoning-heavy steps; measure
6Shrink the prompt. Trim history, include_contents="none" for stateless sub-agents, compact tool outputs, context caching for big static policy text.Lower TTFT and costLess memory of earlier turns; summarise instead of dropping
7Fix the infrastructure. Cloud Run min-instances plus startup CPU boost, same region for service, model, session store and core API, keep-alive HTTP clients.Removes cold starts (seconds) and cross-region hops (tens of ms each)Pays for idle instances
8Cache what's safe to cache. Non-personalised FAQ answers, policy lookups, embeddings.Near-zero latency on hitsNever cache personalised or account-bound answers
9Stabilise the tail. Provisioned Throughput for predictable capacity, timeouts, one retry, fall back to a smaller model or a static answer.p99 improves more than p50Cost; fallback needs its own evals

Illustrative latency budget

Budgets make the trade-offs visible. These are targets to argue about, not measurements:

StageBudgetHow
Edge + auth~50 msGateway validates JWT; no extra hop
Input guard~100 msRun concurrently with prefetch; fail closed with a short timeout
Routing~300 msSmall model, thinking off, 256 output tokens max, or skip via code
Tools~250 msParallel; core API p95 is your real constraint
Answer, first token~500 msStream; compact context
Answer, complete2-4 sShort answers by instruction; cap max tokens

Code: tiering, thinking and prompt trimming

agents/latency.py
from google.adk.agents import LlmAgent
from google.adk.agents.callback_context import CallbackContext
from google.adk.models.llm_request import LlmRequest
from google.adk.planners import BuiltInPlanner
from google.genai import types

NO_THINKING = BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_budget=0))
MAX_HISTORY_TURNS = 6


def trim_history(callback_context: CallbackContext, llm_request: LlmRequest):
    """Keep the system prompt and the last few turns. Summarise older ones elsewhere if needed."""
    contents = llm_request.contents
    if len(contents) > MAX_HISTORY_TURNS * 2:
        llm_request.contents = contents[-MAX_HISTORY_TURNS * 2:]
    return None                                     # None means: continue to the model


faq_agent = LlmAgent(
    name="faq", model="gemini-2.5-flash-lite",
    description="Answers general, non-personal questions about card fees and opening hours.",
    instruction="Answer only from the provided policy snippets. If unsure, say so.",
    planner=NO_THINKING,
    include_contents="none",                         # stateless: don't resend the conversation
    before_model_callback=trim_history,
    generate_content_config=types.GenerateContentConfig(temperature=0, max_output_tokens=300),
)

Streaming without skipping the output guard

The part people get wrong. If you stream raw partial events to the browser you have bypassed your output controls, because the check ran (or didn't, depending on streaming mode and ADK version) on the aggregated text. Verify the behaviour in your version with a test that tries to leak a card number. A pattern that works: stream into a sentence buffer in the API layer, run the cheap deterministic checks (PAN regex, blocklist) on each completed sentence, and flush only the ones that pass. For healthcare, add a rule that clinical statements are only released if they came from an approved tool result or retrieved leaflet.

Perceived speed is half the game

While tools run, send a status event ("Checking your recent payments…"). That costs nothing, and users tolerate a 3-second wait with progress better than a 1.5-second silence. Emit it from before_tool_callback into the same SSE stream.

11Exposing it

A streaming HTTP API in front of the Runner, with identity verified before ADK sees anything.

server.py
import json, uuid
from fastapi import Depends, FastAPI, HTTPException, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from google.adk.runners import Runner
from google.adk.agents.run_config import RunConfig, StreamingMode
from google.adk.sessions import DatabaseSessionService
from google.genai import types
from bank_assist.agent import root_agent

APP = "bank_assist"
sessions = DatabaseSessionService(db_url="postgresql+asyncpg://...")   # Cloud SQL via the connector
runner = Runner(agent=root_agent, app_name=APP, session_service=sessions)
app = FastAPI()


class ChatIn(BaseModel):
    session_id: str | None = None
    message: str = Field(max_length=2000)


async def verified_user(request: Request) -> dict:
    """Validate the bearer token (Identity Platform / your IdP) and return claims.
    Reject on any doubt. The agent never sees an unverified identity."""
    token = request.headers.get("authorization", "").removeprefix("Bearer ").strip()
    claims = await verify_jwt(token)                 # google.oauth2.id_token / your IdP SDK
    if not claims:
        raise HTTPException(401)
    return claims


@app.post("/v1/chat")
async def chat(body: ChatIn, user: dict = Depends(verified_user)):
    uid = user["sub"]
    sid = body.session_id or str(uuid.uuid4())
    session = await sessions.get_session(app_name=APP, user_id=uid, session_id=sid)
    if session is None:
        session = await sessions.create_session(
            app_name=APP, user_id=uid, session_id=sid,
            state={"customer_id": user["customer_id"],          # from the token, not the request body
                   "customer_ref": user["pseudo_id"], "session_key": sid})

    async def events():
        msg = types.Content(role="user", parts=[types.Part(text=body.message)])
        async for ev in runner.run_async(user_id=uid, session_id=sid, new_message=msg,
                                         run_config=RunConfig(streaming_mode=StreamingMode.SSE)):
            if ev.partial and ev.content and ev.content.parts and ev.content.parts[0].text:
                yield f"data: {json.dumps({'delta': ev.content.parts[0].text})}\n\n"
            elif ev.is_final_response():
                yield f"data: {json.dumps({'done': True, 'session_id': sid})}\n\n"

    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"})


@app.get("/healthz")
def healthz(): return {"ok": True}

Choices for the front door

OptionGood forWatch for
Cloud Run + API gateway (Apigee or API Gateway)Full control of auth, quotas, SSE, custom controls. My default for regulated consumer products.You own scaling config, patching and the HTTP surface
Vertex AI Agent EngineManaged runtime, sessions and memory; less opsLess control over the HTTP layer; confirm compliance scope and network options for your case
GKEYou already run a platform team and need sidecars, custom networkingMore to run for the same agent
A2A protocolExposing a specialist to other teams' agents, or consuming theirsTreat remote agents as untrusted input; same tool-policy rules apply
MCP serversReusing tools across agents and vendorsAn MCP tool is still a tool: allow-list, scope credentials, log calls

The edge, concretely

  • Authentication: OIDC from your IdP, validated at the gateway and again in the service. Cloud Run set to require authentication, ingress limited to the load balancer.
  • Per-user quotas: requests per minute and tokens per day, keyed to the verified user, enforced at the gateway. Prevents one account (or one prompt-injection loop) from becoming a bill.
  • Abuse controls: Cloud Armor rules, bot detection, input size caps, concurrency caps per session (a customer shouldn't have three turns in flight).
  • Response hygiene: no-store on chat responses, no model reasoning or tool arguments in the stream, structured error codes that don't leak internals.
  • Channel parity: the voice, in-app and web channels all call the same API so the controls apply once.

12Deploy and operate

Boring on purpose.

Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY bank_assist ./bank_assist
COPY server.py .
ENV PYTHONUNBUFFERED=1 GOOGLE_GENAI_USE_VERTEXAI=TRUE
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1"]
terminal
gcloud run deploy bank-assist \
  --source . --region europe-west2 \
  --service-account bank-assist-runtime@$PROJECT.iam.gserviceaccount.com \
  --no-allow-unauthenticated --ingress internal-and-cloud-load-balancing \
  --min-instances 2 --cpu-boost --concurrency 20 --timeout 60 \
  --vpc-egress all-traffic --network default --subnet agents-subnet \
  --set-env-vars GOOGLE_CLOUD_PROJECT=$PROJECT,GOOGLE_CLOUD_LOCATION=europe-west2
  • Least-privilege service account. It can call Vertex AI, read its secrets and reach the specific backend APIs. Nothing else. Separate accounts for the agent runtime and for the CI deployer.
  • Perimeter. Put the project in a VPC Service Controls perimeter, use Private Service Connect to the model endpoint where available, and deny egress by default. A prompt-injected agent that can't reach the internet can't exfiltrate to it.
  • Release. Build once, promote the same image dev → staging → prod. Gate promotion on the eval suite (guide 09) and a signed-off prompt version. Roll out by traffic percentage with automatic rollback on guard-trigger rate, error rate or escalation rate moving.
  • Region and residency. Check where inference actually runs for your chosen model and endpoint, not just where your service lives. For UK/EU data this is a design-time question, not a deployment-day one.

What to watch in production

Service health

Latency p50/p95/p99 per stage (TTFT, tool, full), error rate, timeouts, fallback usage, model quota errors.

Behaviour

Tool-call mix, transfers per agent, guard block rate, escalation rate, "confirmation requested vs completed", conversation length.

Quality & cost

Sampled online eval scores (guide 09), thumbs and complaints (guide 10), tokens and cost per resolved conversation.

ADK emits OpenTelemetry spans for agent runs, model calls and tool calls. Export them to Cloud Trace and you get a waterfall per turn, which is the fastest way to find out which of your nine latency techniques you actually need. Redact before export: spans can carry prompt text.

13Testing before it reaches a customer

Three layers, cheapest first.

  1. Unit tests, no model. Tools against a fake bank API; tool_policy with a table of allowed and blocked calls; output_guard with leak attempts. These run in seconds and catch most control regressions.
  2. Trajectory evals. For a set of scripted conversations, assert the right agent got the turn and the right tools were called with the right arguments. ADK's evalsets and adk eval do this. Detail in guide 09.
  3. Adversarial and scenario testing. Prompt injection via transaction descriptions (an attacker can pick a merchant name!), identity-swap attempts ("show me my husband's account"), requests for advice, a distressed customer, a patient describing chest pain. Keep every one you find as a permanent test.
tests/test_tool_policy.py
import pytest
from types import SimpleNamespace
from bank_assist.guardrails import tool_policy

def ctx(agent, **state):
    return SimpleNamespace(agent_name=agent, state=dict(state))

TOOL = lambda name: SimpleNamespace(name=name)

def test_dispute_blocked_without_confirmation():
    r = tool_policy(TOOL("open_card_dispute"),
                    {"transaction_id": "t1", "reason": "duplicate"}, ctx("dispute_filer"))
    assert r["status"] == "needs_confirmation"

def test_dispute_allowed_after_ui_confirmation():
    c = ctx("dispute_filer", confirmed_txn="t1", txn_amount_gbp=40)
    assert tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t1", "reason": "duplicate"}, c) is None

def test_wrong_agent_cannot_call_write_tool():
    r = tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t1", "reason": "duplicate"}, ctx("spend_insights"))
    assert r["status"] == "error"

def test_model_cannot_confirm_for_a_different_transaction():
    c = ctx("dispute_filer", confirmed_txn="t1")
    r = tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t2", "reason": "duplicate"}, c)
    assert r["status"] == "needs_confirmation"

14Production readiness checklist

If you can't tick these, you have a demo.

Design
  • Each agent justified by a permission, latency or ownership reason
  • Identity from verified token, never from model arguments
  • Writes gated by out-of-band confirmation and code-level limits
  • Fixed, approved text for red flags, disclosures and refusals
  • Hazard log or model-risk entry written and owned
Platform
  • Pinned model and ADK versions; prompt version logged per turn
  • Private networking, VPC-SC, CMEK where required, TTLs on sessions
  • Min instances, same-region data path, timeouts and fallbacks
  • Per-user quotas, concurrency caps, budget alerts, kill switch
  • PII-free structured audit trail queryable in BigQuery
Quality
  • Unit tests on tools and guards in CI
  • Trajectory and response evals gate every release
  • Red-team set including injection through data fields
  • Sampled online evals and human review on live traffic
  • Rollback rehearsed, not just documented
Speed
  • Per-stage latency traced; budget agreed with product
  • Streaming on, with sentence-level output checks
  • Router on small model, thinking off, or no model at all
  • Parallel reads, compact tool outputs, trimmed history
  • No caching of personalised answers
Takeaway

The agent code is the easy 20%. The work that makes it shippable in a bank or a health system is identity, tool scoping, approval, audit and latency discipline, and almost all of it is ordinary software engineering applied to a component that happens to be non-deterministic. Build the controls so that the worst thing the model can do is say something unhelpful, then use evals (next guide) to find out how often it does.