1The short version
What you are building, what to decide first, and what I would not skip.
Build the smallest agent graph that works (one coordinator, two or three specialists, deterministic code wherever a decision doesn't need language), put every control outside the model where it can't be talked out of its job, and treat latency as a design constraint from day one rather than a tuning exercise in month four.
Google's Agent Development Kit (ADK) gives you the pieces: LlmAgent for reasoning, workflow agents (SequentialAgent, ParallelAgent, LoopAgent) for control flow you want to be deterministic, a Runner and session service for state, and callbacks that run before and after every model call, tool call and agent. It is code-first Python, which matters in regulated work: your control logic lives in version control, gets code review and has unit tests.
We build one worked example end to end: a retail-bank assistant that answers spending questions and files card disputes. A healthcare variant (post-discharge medication questions, FHIR-backed) swaps the tools and tightens the controls; the architecture is the same. The code is deliberately compact so the shape is visible, and you will need to harden each piece for your own environment.
What you'll build
A coordinator with three specialists, a parallel evidence-gathering step, guardrails in callbacks, a streaming API and a Cloud Run deployment.
Where it gets hard
Not the agent code. Identity propagation, write-action approval, audit, and keeping p95 latency tolerable with guardrails switched on.
Assumptions
Vertex AI as the model backend, Python 3.11+, Cloud Run as the runtime, a UK or EU region. Model names move fast, so pin whatever is GA when you read this.
ADK ships often and some APIs here (plugins, tool confirmation, streaming behaviour inside callbacks) have changed between minor versions. Pin google-adk in your lockfile, read the release notes before upgrading, and let your eval suite (guide 09) be the thing that tells you an upgrade is safe.
2Lifecycle and ownership at a glance
6 phases, 45 activities, who owns each. Jump to a phase:
| Activity | Domain | Data science | Platform | Risk & compl. | SRE / support | Output |
|---|---|---|---|---|---|---|
| 1 · Design Decide what to build, how risky it is and how it is shaped | ||||||
| Define use case, success metrics, in/out of scope | A | C | I | C | I | Scope doc, KPIs |
| Risk tier and regulatory mapping (SS1/23, HIPAA, DCB0129, AI Act) | R | C | C | A | I | Tier, control set |
| Agent graph and routing (one agent or many) | C | A | C | I | Agent graph, routing table | |
| Tool catalogue: read/write split, approval points, limits | R | R | A | C | I | Tool contracts |
| Data access, privacy and residency (DPIA, retention, PHI/PCI) | C | C | R | A | DPIA, data-flow diagram | |
| Escalation, handoff and fallback journeys | A | C | I | C | C | Handoff rules, fixed safety text |
| Latency, cost and capacity budget | C | R | A | C | Per-stage budget | |
| Clinical safety case / model-risk entry and hazard log | R | R | C | A | I | Safety case, hazard log |
| 2 · Build Write the agent, tools, controls and infrastructure | ||||||
| Prompts and instructions, versioned in the repo | C | A | I | I | Prompt versions | |
| ADK agents, workflows, state and sessions | I | A | C | Agent code | ||
| Tool wrappers and backend integrations | A | R | C | I | Typed tools, fakes for tests | |
| Knowledge and retrieval content (approved leaflets, policy) | A | C | I | C | Curated, versioned corpus | |
| Guardrail plugins and callbacks (input, output, tool policy) | I | C | A | C | Shared guard library | |
| Identity, API exposure, gateway, quotas | I | A | C | C | Authenticated streaming API | |
| Infrastructure as code: Cloud Run, IAM, VPC-SC, CMEK, logging | A | C | C | Terraform module call | ||
| Model, library and config pinning; secrets | R | A | I | I | Lockfiles, pinned model IDs | |
| 3 · Test Prove it behaves before a customer or patient sees it | ||||||
| Unit tests for tools and guards (no model) | A | C | Fast CI tests | |||
| Golden dataset and domain eval cases from real transcripts | A | R | I | C | Versioned datasets | |
| Eval harness, baseline suites, judge calibration | C | R | A | C | Eval runner, calibrated judges | |
| Trajectory and routing evals | C | A | C | Routing confusion matrix | ||
| Red-team: injection via data fields, impersonation, scope creep | C | R | R | A | Attack set, findings | |
| Latency and load test against the budget | C | R | A | Perf report | ||
| SME review of sampled outputs (clinicians, claims, compliance) | A | C | C | Labelled sample | ||
| Release gate sign-off by tier | R | R | R | A | I | Signed gate record |
| 4 · Deploy Ship safely and be able to undo it | ||||||
| CI/CD pipeline: evals, plan, policy checks, attestation | C | A | C | I | Reusable pipeline | |
| Release approval and change record | A | R | R | C | I | Approved change |
| Shadow, canary and rollback plan | C | R | R | A | Rollout plan, rollback tested | |
| Runbook, on-call rota, escalation contacts | R | C | C | A | Runbook | |
| Staff and user communication, training, disclosures | A | C | C | I | Training, notices | |
| 5 · Operate and observe Keep it healthy, visible and auditable | ||||||
| Tracing, structured logs, dashboards (OpenTelemetry, Cloud Trace, BigQuery) | C | A | I | R | Per-turn traces | |
| SLOs and alerts: latency, errors, guard triggers, escalations, cost | C | C | R | A | Alert policy | |
| Incident response: page, triage, kill switch, post-incident review | R | R | R | C | A | Incident record |
| Cost, quota and capacity management | I | C | A | R | Budget reports | |
| Model and provider changes, deprecations, upgrades | I | R | A | C | I | Upgrade plan, re-run evals |
| Security monitoring, access reviews, key rotation | R | A | C | Review records | ||
| Audit trail retention and regulator or customer requests | C | R | A | I | Queryable audit store | |
| 6 · Feedback and improve Turn live behaviour into fixes, tests and better models | ||||||
| Capture feedback: thumbs, edits, escalations, nurse or agent tags, outcomes | C | C | A | I | Feedback events | |
| Sampled online evals with LLM judge on live traffic | I | A | R | C | Online scores by version | |
| Human review queue for flagged and sampled conversations | A | C | C | Labels, corrected replies | ||
| Judge calibration against human labels (weekly or monthly) | R | A | C | TPR/TNR report | ||
| Triage failures into fixes: content, prompt, tool or guard | R | A | C | I | Fix backlog | |
| Turn every confirmed failure into an eval case | R | A | C | New regression cases | ||
| Preference data, tuning or RL candidates behind the release gate | C | A | R | C | Candidate model, eval evidence | |
| Drift review and quarterly threshold and metric review | A | R | C | C | I | Updated thresholds |
| Re-tier, expand scope or retire the agent | A | C | C | R | I | Decision record |
A accountable: owns the outcome, one per row, usually does the work. R does part of the work. C consulted before it is done. I told afterwards. Blank means no role.
- Domain is the business or product owner plus subject-matter experts (clinicians, claims, relationship managers, operations reviewers).
- Data science covers ML and AI engineers who build and evaluate the agent. Platform is the shared engineering team (cloud, CI/CD, guard library). SRE / support runs it in production and carries the pager.
- Small teams collapse columns (one team may be Data science and Platform). Keep the single accountable owner per row even then.
- For tier-1 agents, Risk & compliance becomes accountable for release sign-off and for any change to thresholds. Everyone else's role stays as shown.
3What regulated industries change
The same agent, a different set of questions from your risk function.
In an unregulated product the question is "does it work?". In banking and health it becomes "can you prove what it did, why, for whom, and that it stayed inside its remit?" That moves a handful of design decisions early.
Banking
- Model risk management. UK PRA SS1/23, US SR 11-7. An LLM agent is a model: inventory it, tier it, validate it, monitor it.
- Consumer Duty / conduct. Outcomes for retail customers must be good and evidenced. No steering, no advice you aren't authorised to give.
- Operational resilience. DORA in the EU, PRA/FCA rules in the UK: tolerances, exit plans, third-party concentration on the model provider.
- PCI DSS, GDPR/UK GDPR. Card numbers never reach the model or the logs. Lawful basis and retention for chat transcripts.
- EU AI Act. Creditworthiness assessment is classed high-risk; a servicing assistant usually isn't, but confirm with legal.
Healthcare
- PHI handling. HIPAA (US) or UK GDPR special-category data plus NHS DSPT. Sign the BAA, use only covered services, minimum necessary data.
- Clinical safety. In England, DCB0129/DCB0160 with a named Clinical Safety Officer and a hazard log. A hazard log is where "agent gives wrong dose advice" lives.
- Medical device boundary. If it informs diagnosis or treatment decisions it may be SaMD (MHRA, FDA, EU MDR). Scope the product to stay clear of that line, or go through it deliberately.
- Human oversight. A clinician accountable for anything clinical. The agent routes, informs and drafts; it doesn't decide.
- Interoperability. FHIR via Cloud Healthcare API; consent and audit are part of the data layer.
Three design consequences follow, and the rest of this guide leans on them:
- Identity comes from the platform, never from the model. The customer or patient ID is set by your API layer from a verified token. A tool takes it from session state, not from an argument the LLM can fill in.
- Read and write are different trust levels. Reads can be autonomous inside a scope. Writes need an explicit, out-of-band confirmation from the user, and sometimes from a human at your end.
- Everything is reconstructable. Prompt version, model version, tool calls with arguments, guard verdicts, who approved. Stored somewhere that a regulator-facing person can query.
The regimes above are pointers for the conversation with compliance, not a compliance analysis. Check each Google Cloud service you use against its current compliance scope (HIPAA BAA coverage, data residency, VPC Service Controls support) before the design is signed off. That list changes, and the answer for the model endpoint can differ from the answer for the session store.
4Architecture and agent graph
Decide how many agents you need before you write one.
One agent or many?
Most "multi-agent" systems should have been one agent with good tools. Split when one of these is true, not because the diagram looks impressive:
| Reason to split | Example here | Alternative to consider |
|---|---|---|
| Different permissions | Spend insights can only read; dispute filer has one write tool | None. This is the strongest reason. |
| Different latency or model needs | Router on a small fast model, answerer on a larger one | One agent with a model callback that swaps models |
| Prompt too large or contradictory | Dispute policy is 3k tokens of rules that confuse small-talk handling | Retrieval of policy on demand as a tool |
| Independent steps that can run concurrently | Transaction lookup and fraud-signal check | Plain asyncio.gather inside one tool |
| Separate ownership | Cards team owns disputes, savings team owns insights | Remote agents over A2A |
ADK's building blocks, in the order you will reach for them
| Piece | Use it for | Notes |
|---|---|---|
LlmAgent | Anything needing language understanding or tool selection | Model decides what to call. Non-deterministic by nature. |
SequentialAgent | A fixed pipeline: gather → decide → act | Sub-agents share session state; pass data with output_key. |
ParallelAgent | Independent lookups that should overlap | Branches get isolated invocation paths; write to different state keys. |
LoopAgent | Draft → critique → revise, with a hard max_iterations | Always cap iterations; each loop costs seconds. |
sub_agents + transfer | LLM-driven routing to a specialist | Routing quality depends on each agent's description. |
AgentTool | Call a specialist as a function and keep control | Better than transfer when the parent must post-process the answer. |
| Callbacks / plugins | Guardrails, audit, caching, metrics | Plugins apply across all agents in a Runner; callbacks are per agent. |
5Project setup
A layout that survives a second team joining.
python3.11 -m venv .venv && source .venv/bin/activate
pip install "google-adk" fastapi "uvicorn[standard]" pydantic google-cloud-logging
# Vertex AI as the backend (not an API key): auth is a service account, quota and
# data-governance terms are your GCP project's.
export GOOGLE_GENAI_USE_VERTEXAI=TRUE
export GOOGLE_CLOUD_PROJECT=my-bank-agents-dev
export GOOGLE_CLOUD_LOCATION=europe-west2 # check model availability in your region
gcloud auth application-default loginbank_assist/
├── __init__.py # from . import agent (ADK discovers root_agent here)
├── agent.py # assembles root_agent
├── agents/
│ ├── concierge.py
│ ├── insights.py
│ └── disputes.py
├── tools/
│ ├── bank_api.py # thin typed client for the core banking API
│ └── tools.py # functions exposed to the model
├── guardrails.py # callbacks: input, output, tool policy, audit
├── config.py # model names, limits, prompt version
├── server.py # FastAPI + Runner (the production entrypoint)
├── tests/ # unit tests for tools + guards, evalsets for agents
└── DockerfileRun adk web from the parent directory for a local dev UI with event traces, and adk run bank_assist for a terminal loop. The dev UI is invaluable for seeing which sub-agent got the turn and what each tool returned, and it is not something to ship.
from dataclasses import dataclass
@dataclass(frozen=True)
class Config:
# Pin concrete GA model IDs; don't use "latest" aliases in a regulated system,
# because behaviour changing under you breaks validation evidence.
router_model: str = "gemini-2.5-flash-lite"
answer_model: str = "gemini-2.5-flash"
prompt_version: str = "2026-10-01.3" # logged on every turn
max_dispute_amount_gbp: float = 2_500.0 # business rule, enforced in code
max_tool_calls_per_turn: int = 6
CFG = Config()6Tools: where the real risk lives
The model proposes. Your function disposes.
ADK turns a typed Python function with a good docstring into a tool. The docstring is the prompt the model sees, so write it for the model: what it does, when to use it, what the arguments mean, what comes back. Three rules I would hold the line on:
- No identity arguments. The customer ID isn't a parameter. It comes from
tool_context.state, set by your server after token verification. - Return small, structured, pre-redacted data. Return the five fields the answer needs, not the 60-field record. Cheaper, faster, and nothing sensitive reaches the model by accident.
- Return errors as data.
{"status": "error", "message": ...}lets the model explain or escalate. An exception mid-turn gives the customer a stack trace or a hang.
from google.adk.tools.tool_context import ToolContext
from . import bank_api # thin client; owns auth, timeouts, retries
async def get_recent_transactions(days: int, tool_context: ToolContext) -> dict:
"""Get the signed-in customer's card transactions for the last N days.
Use for questions about spending, a specific payment, or before filing a dispute.
Args:
days: How far back to look. Between 1 and 90.
Returns:
{"status": "ok", "transactions": [{"id", "date", "merchant", "amount_gbp", "status"}]}
or {"status": "error", "message": str}.
"""
customer_id = tool_context.state["customer_id"] # set server-side, not by the model
days = max(1, min(days, 90))
try:
rows = await bank_api.transactions(customer_id, days=days)
except bank_api.Unavailable:
return {"status": "error", "message": "Transactions are temporarily unavailable."}
# Mask on the way out: the model never needs a PAN or a full account number.
return {"status": "ok", "transactions": [
{"id": r.id, "date": r.date.isoformat(), "merchant": r.merchant,
"amount_gbp": r.amount, "status": r.status} for r in rows[:50]]}
async def get_fraud_signals(transaction_id: str, tool_context: ToolContext) -> dict:
"""Check whether a transaction triggered any fraud-detection signals.
Args:
transaction_id: An id returned by get_recent_transactions.
"""
customer_id = tool_context.state["customer_id"]
sig = await bank_api.fraud_signals(customer_id, transaction_id)
return {"status": "ok", "flagged": sig.flagged, "reason_codes": sig.reason_codes}
async def open_card_dispute(transaction_id: str, reason: str, tool_context: ToolContext) -> dict:
"""File a dispute for ONE transaction. Only call after the customer has confirmed in the UI.
Args:
transaction_id: The id of the disputed transaction.
reason: One of "not_recognised", "duplicate", "not_received", "wrong_amount".
"""
customer_id = tool_context.state["customer_id"]
case = await bank_api.open_dispute(
customer_id, transaction_id, reason,
idempotency_key=f"{tool_context.state['session_key']}:{transaction_id}", # retries can't double-file
)
return {"status": "ok", "case_id": case.id, "next_step": case.next_step}
def escalate_to_human(summary: str, tool_context: ToolContext) -> dict:
"""Hand the conversation to a human agent. Use when the customer asks, is distressed,
reports vulnerability, or you cannot help safely.
Args:
summary: Two sentences for the human agent. No account numbers.
"""
tool_context.actions.escalate = True
tool_context.state["handoff_summary"] = summary
return {"status": "ok", "message": "A colleague will pick this up."}Most agent incidents I have seen trace back to a tool that was too powerful, not a prompt that was too weak: a "run query" tool, a "send message" tool with free-form recipients, or an identity argument the model could fill in. A narrow tool that can only do one thing is the cheapest control you will ever buy.
7Agents and workflows
Language where you need it, deterministic flow where you don't.
The specialists
from google.adk.agents import LlmAgent
from google.genai import types
from ..config import CFG
from ..tools.tools import get_recent_transactions
from ..guardrails import tool_policy, audit_after_tool
spend_insights = LlmAgent(
name="spend_insights",
model=CFG.answer_model,
# The description is what the coordinator reads when routing. Write it like a job ad.
description="Answers questions about the customer's spending, merchants and recent card payments. Read-only.",
instruction="""You help a retail banking customer understand their card spending.
- Use get_recent_transactions for facts. Never state an amount or merchant you did not get from a tool.
- You are not a financial adviser. Do not recommend products, investments or borrowing.
- If the customer wants to dispute a payment, hand back to the coordinator.
- Be brief. Plain English. Amounts in pounds.""",
tools=[get_recent_transactions],
before_tool_callback=tool_policy,
after_tool_callback=audit_after_tool,
generate_content_config=types.GenerateContentConfig(temperature=0.2, max_output_tokens=600),
)A pipeline with a parallel step
Disputes are a fixed process, so the control flow should be code, not a prompt. Gather evidence in parallel, then a final agent decides whether to propose a dispute. The write tool is only reachable at the last step.
from google.adk.agents import LlmAgent, ParallelAgent, SequentialAgent
from ..config import CFG
from ..tools.tools import get_recent_transactions, get_fraud_signals, open_card_dispute
from ..guardrails import tool_policy, audit_after_tool
txn_lookup = LlmAgent(
name="txn_lookup", model=CFG.router_model,
description="Finds the transaction the customer is talking about.",
instruction="Find the one transaction the customer means in the last 60 days. "
"If several match, return the candidates. Output JSON only.",
tools=[get_recent_transactions],
output_key="candidate_txns", # lands in session state
before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)
fraud_check = LlmAgent(
name="fraud_check", model=CFG.router_model,
description="Checks fraud signals on a transaction.",
instruction="For the transaction in the user's message, call get_fraud_signals. Output JSON only.",
tools=[get_fraud_signals],
output_key="fraud_result",
before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)
gather_evidence = ParallelAgent(name="gather_evidence", sub_agents=[txn_lookup, fraud_check])
dispute_filer = LlmAgent(
name="dispute_filer", model=CFG.answer_model,
description="Explains findings and files a dispute once the customer has confirmed.",
# {candidate_txns} and {fraud_result} are filled from session state.
instruction="""Evidence so far:
transactions: {candidate_txns}
fraud signals: {fraud_result}
1. Tell the customer what you found, in two or three sentences.
2. If exactly one transaction matches and they want to dispute it, ask them to confirm
with the on-screen button. Do not call open_card_dispute until the system says confirmed.
3. If anything is ambiguous, ask one clarifying question. Never guess a transaction.""",
tools=[open_card_dispute],
before_tool_callback=tool_policy, after_tool_callback=audit_after_tool,
)
dispute_flow = SequentialAgent(
name="dispute_flow",
description="Handles a customer's request to dispute or query a specific card payment they don't recognise.",
sub_agents=[gather_evidence, dispute_filer],
)The coordinator
from google.adk.agents import LlmAgent
from google.adk.tools import FunctionTool
from google.adk.planners import BuiltInPlanner
from google.genai import types
from .config import CFG
from .agents.insights import spend_insights
from .agents.disputes import dispute_flow
from .tools.tools import escalate_to_human
from .guardrails import input_guard, output_guard, prefetch_context
root_agent = LlmAgent(
name="concierge",
model=CFG.router_model,
description="Front door for the retail banking assistant.",
instruction="""You route. You do not answer account questions yourself.
- Spending, merchants, "what did I spend": transfer to spend_insights.
- A payment they don't recognise or want to dispute: transfer to dispute_flow.
- Asks for a person, is upset, mentions a bereavement, financial difficulty or fraud in progress: call escalate_to_human.
- Anything outside card spending and disputes: say what you can help with. No advice on products.""",
sub_agents=[spend_insights, dispute_flow],
tools=[FunctionTool(escalate_to_human)],
planner=BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_budget=0)), # routing needs no reasoning
generate_content_config=types.GenerateContentConfig(temperature=0, max_output_tokens=256),
before_agent_callback=[input_guard, prefetch_context], # a list runs in order; stops at the first that returns content
after_model_callback=output_guard,
)LLM routing works until two descriptions overlap. Write routing cases into your eval set on day one (guide 09: trajectory evals), and keep the agent count low enough that you can hold the whole routing table in your head. If a rule is obvious (a button click that means "dispute"), route on it in code and skip the model entirely.
Healthcare variant
Same graph, different tools and a tighter remit. For a post-discharge assistant: get_discharge_summary and get_medication_list (FHIR reads through Cloud Healthcare API with the patient's consent scope), search_patient_leaflets (retrieval over approved patient-information content only), and escalate_to_clinician. No tool that changes a medication, ever. The instruction states the boundary plainly: explain what is in the discharge plan; for anything about changing doses, new symptoms or "is this normal", route to a nurse line and give the red-flag advice the clinical team has approved. Red-flag symptom lists should be fixed text approved by your clinical safety officer, served by a tool, not generated.
8State, sessions and memory
Three scopes, three lifetimes, one retention policy to agree with legal.
| Scope | Key prefix | Lifetime | Put here |
|---|---|---|---|
| Turn | temp: | One invocation | Intermediate values you don't want persisted |
| Session | none | One conversation | customer_id, confirmation flags, evidence from the pipeline |
| User | user: | Across sessions | Preferences (language, accessibility needs). Not transcripts, not health details. |
| App | app: | Global | Feature flags, banner text |
InMemorySessionService is for tests. In production use DatabaseSessionService (SQLAlchemy URL to Cloud SQL or AlloyDB) or Vertex AI's managed session service if you deploy on Agent Engine. Whichever you pick, put the database in the same region as the Cloud Run service (a cross-region read per turn is 20-100ms you didn't need), encrypt with CMEK if your policy requires it, and set a TTL. Long-term memory is a separate decision: for banking I would store nothing the customer didn't ask us to remember, and for health I would not enable cross-session LLM memory without a documented lawful basis and clinical sign-off.
Session history is also your biggest latency and cost lever, covered in section 10. Keep it short on purpose.
9Controls: guardrails that the model can't talk its way past
Layered, mostly deterministic, and each one independently testable.
Input guard and context prefetch
Screen the incoming message before it reaches a model. Google Cloud's Model Armor screens prompts and responses for injection, jailbreak and sensitive-data patterns, and Sensitive Data Protection (DLP) handles detection and redaction of PAN, NHS/SSN numbers and similar. The wrapper below is the seam: swap the stub for those clients, keep the callback shape.
import asyncio, hashlib, json, logging, re
from typing import Optional
from google.adk.agents.callback_context import CallbackContext
from google.adk.models.llm_response import LlmResponse
from google.adk.tools.base_tool import BaseTool
from google.adk.tools.tool_context import ToolContext
from google.genai import types
from .config import CFG
from .tools import bank_api
log = logging.getLogger("audit") # shipped to Cloud Logging as JSON
PAN = re.compile(r"\b(?:\d[ -]?){13,19}\b")
def _say(text: str) -> types.Content:
return types.Content(role="model", parts=[types.Part(text=text)])
async def screen(text: str) -> dict:
"""Seam for Model Armor sanitize_user_prompt + Sensitive Data Protection.
Return {"blocked": bool, "reason": str, "redacted": str}."""
redacted = PAN.sub("[card number removed]", text)
return {"blocked": False, "reason": "", "redacted": redacted}
async def input_guard(callback_context: CallbackContext) -> Optional[types.Content]:
"""Runs before the coordinator. Returning Content short-circuits the agent."""
text = "".join(p.text or "" for p in (callback_context.user_content.parts or []))
if len(text) > 2_000:
return _say("That message is too long for me to handle. Could you shorten it?")
verdict = await screen(text)
_audit("input_guard", callback_context, blocked=verdict["blocked"], reason=verdict["reason"])
if verdict["blocked"]:
return _say("I can't help with that request. I can help with your card spending and disputes.")
if verdict["redacted"] != text:
callback_context.state["input_had_pii"] = True
return None
async def prefetch_context(callback_context: CallbackContext) -> Optional[types.Content]:
"""Fetch what almost every turn needs, concurrently, before any LLM call (see section 10)."""
cid = callback_context.state["customer_id"]
summary, vulnerable = await asyncio.gather(bank_api.summary(cid), bank_api.vulnerability_flag(cid))
callback_context.state["account_summary"] = summary
callback_context.state["vulnerable_customer"] = vulnerable # drives tone and escalation rules
return NoneTool policy: the control that matters most
READ_ONLY = {"get_recent_transactions", "get_fraud_signals"}
WRITES = {"open_card_dispute"}
ALLOWED_BY_AGENT = {
"spend_insights": {"get_recent_transactions"},
"txn_lookup": {"get_recent_transactions"},
"fraud_check": {"get_fraud_signals"},
"dispute_filer": {"open_card_dispute"},
"concierge": {"escalate_to_human"},
}
DISPUTE_REASONS = {"not_recognised", "duplicate", "not_received", "wrong_amount"}
def tool_policy(tool: BaseTool, args: dict, tool_context: ToolContext) -> Optional[dict]:
"""Return a dict to block the call and hand that dict to the model as the tool result."""
agent = tool_context.agent_name
if tool.name not in ALLOWED_BY_AGENT.get(agent, set()):
return {"status": "error", "message": "Tool not permitted for this agent."}
n = tool_context.state.get("tool_calls_this_turn", 0) + 1
tool_context.state["tool_calls_this_turn"] = n
if n > CFG.max_tool_calls_per_turn:
return {"status": "error", "message": "Too many steps. Hand over to a colleague."}
if tool.name == "open_card_dispute":
# 1. Out-of-band approval. The UI button sets this via a state delta; the model can't.
if tool_context.state.get("confirmed_txn") != args.get("transaction_id"):
return {"status": "needs_confirmation",
"message": "Ask the customer to confirm with the on-screen button."}
# 2. Argument validation in code.
if args.get("reason") not in DISPUTE_REASONS:
return {"status": "error", "message": f"reason must be one of {sorted(DISPUTE_REASONS)}"}
# 3. Business limits the model doesn't get a vote on.
if tool_context.state.get("txn_amount_gbp", 0) > CFG.max_dispute_amount_gbp:
return {"status": "needs_human", "message": "Over the automated limit. Escalate to a colleague."}
return None
def audit_after_tool(tool: BaseTool, args: dict, tool_context: ToolContext, tool_response: dict) -> Optional[dict]:
_audit("tool_call", tool_context, tool=tool.name, status=tool_response.get("status"),
args_hash=hashlib.sha256(json.dumps(args, sort_keys=True).encode()).hexdigest()[:16])
return None
async def output_guard(callback_context: CallbackContext, llm_response: LlmResponse) -> Optional[LlmResponse]:
"""Last look at what the model wants to say."""
if not llm_response.content or not llm_response.content.parts:
return None
text = "".join(p.text or "" for p in llm_response.content.parts)
if PAN.search(text) or re.search(r"\b(guaranteed|risk[- ]free|you should (invest|borrow))\b", text, re.I):
_audit("output_blocked", callback_context)
return LlmResponse(content=_say("I'm not able to say that. A colleague can help; shall I connect you?"))
return None
def _audit(event: str, ctx, **fields):
"""Structured, PII-free. Prompt version and model are what let you reproduce a decision later."""
log.info(json.dumps({
"event": event, "agent": getattr(ctx, "agent_name", ""),
"session": ctx.state.get("session_key"), "customer": ctx.state.get("customer_ref"), # pseudonymous ref
"prompt_version": CFG.prompt_version, **fields}))A prompt says "never file a dispute without confirmation" and hopes. The callback makes it impossible: the tool function never runs. For anything a regulator or a customer's solicitor could ask about, the control belongs in code with a unit test beside it.
Human approval for the write
The confirmation flag must come from your UI, not from anything the model wrote. In the API layer, when the user taps "Yes, file this dispute", append a state delta to the session before the next turn:
import time
from google.adk.events import Event, EventActions
async def record_confirmation(session_service, session, txn_id: str, amount_gbp: float):
await session_service.append_event(session, Event(
invocation_id=f"confirm-{txn_id}", author="user", timestamp=time.time(),
actions=EventActions(state_delta={"confirmed_txn": txn_id, "txn_amount_gbp": amount_gbp}),
))For higher-risk actions, put a human at your end: the tool writes a pending case to a queue, an operations colleague approves, and the customer is told it is under review. ADK's long-running tool pattern fits that shape (the agent returns a ticket and resumes when the result arrives).
Controls checklist by layer
| Layer | Control | Banking emphasis | Health emphasis |
|---|---|---|---|
| Identity | Verified JWT; ID set in session state; step-up auth for writes | SCA / open-banking style step-up | Patient proxy / carer consent rules |
| Network | Private egress, VPC Service Controls perimeter, no public model endpoints from the agent | PCI scope isolation | PHI never leaves perimeter |
| Data | Minimum fields, redaction on the way into prompts and logs, CMEK, retention TTLs | No PAN, mask account numbers | De-identify before analytics; no PHI in logs |
| Model | Pinned model version, safety settings, output length caps | No product advice; suitability stays human | No diagnosis or dosing; fixed red-flag text |
| Tools | Allow-list per agent, arg validation, caps, idempotency keys | Per-action limits, dual control over threshold | Read-only by default; clinician approval for any write |
| Behaviour | Escalation triggers, vulnerability handling | Vulnerable customers (FCA FG21/1) | Safeguarding, crisis language, self-harm routing |
| Operations | Audit trail, kill switch, budget alerts, circuit breaker, incident runbook | Complaint handling, record-keeping | Clinical safety hazard log, incident reporting |
| Assurance | Offline evals gating release; sampled online evals; red-team before launch | Model validation pack (SS1/23) | Clinical safety case evidence |
Put a feature flag in app: state (or Remote Config) that the API checks before every turn. Three levels worth having: degraded (disable writes), read-only answers from static FAQ, off and route to humans. You will want it the first time a provider incident or a bad prompt release happens on a Friday.
10Making it fast for consumers
Ranked by how much time each change usually saves. Do them in this order.
A consumer will wait about a second for something to start happening and a handful of seconds for the full answer. Agents burn latency in sequential LLM calls (routing call, tool-choice call, answer call), each with its own time-to-first-token plus generation. Controls add their own cost. So the target isn't "make the model faster", it's "make fewer model calls, make each smaller, and show progress while the rest happens".
| # | Technique | Typical effect | Trade-off |
|---|---|---|---|
| 1 | Take LLM calls off the critical path. Route obvious intents in code; prefetch data you know you'll need. | Removes 0.5-2s per avoided call | More code to maintain; test the deterministic paths |
| 2 | Stream tokens. Send the first sentence as soon as it exists. | Perceived latency drops from seconds to ~sub-second | Output guard must work on chunks (buffer by sentence) |
| 3 | Right-size models. Flash-Lite for routing and extraction, Flash for the answer, bigger models only where evals show you need them. | 2-5x faster on routed steps | Evals per tier; more model versions to pin |
| 4 | Parallelise independent work. ParallelAgent, asyncio.gather, parallel function calls. | Latency = slowest branch, not the sum | Shared-state key discipline; concurrency limits on core APIs |
| 5 | Switch off thinking for simple turns. thinking_budget=0 on router and FAQ agents. | Hundreds of ms to seconds | Lower quality on reasoning-heavy steps; measure |
| 6 | Shrink the prompt. Trim history, include_contents="none" for stateless sub-agents, compact tool outputs, context caching for big static policy text. | Lower TTFT and cost | Less memory of earlier turns; summarise instead of dropping |
| 7 | Fix the infrastructure. Cloud Run min-instances plus startup CPU boost, same region for service, model, session store and core API, keep-alive HTTP clients. | Removes cold starts (seconds) and cross-region hops (tens of ms each) | Pays for idle instances |
| 8 | Cache what's safe to cache. Non-personalised FAQ answers, policy lookups, embeddings. | Near-zero latency on hits | Never cache personalised or account-bound answers |
| 9 | Stabilise the tail. Provisioned Throughput for predictable capacity, timeouts, one retry, fall back to a smaller model or a static answer. | p99 improves more than p50 | Cost; fallback needs its own evals |
Illustrative latency budget
Budgets make the trade-offs visible. These are targets to argue about, not measurements:
| Stage | Budget | How |
|---|---|---|
| Edge + auth | ~50 ms | Gateway validates JWT; no extra hop |
| Input guard | ~100 ms | Run concurrently with prefetch; fail closed with a short timeout |
| Routing | ~300 ms | Small model, thinking off, 256 output tokens max, or skip via code |
| Tools | ~250 ms | Parallel; core API p95 is your real constraint |
| Answer, first token | ~500 ms | Stream; compact context |
| Answer, complete | 2-4 s | Short answers by instruction; cap max tokens |
Code: tiering, thinking and prompt trimming
from google.adk.agents import LlmAgent
from google.adk.agents.callback_context import CallbackContext
from google.adk.models.llm_request import LlmRequest
from google.adk.planners import BuiltInPlanner
from google.genai import types
NO_THINKING = BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_budget=0))
MAX_HISTORY_TURNS = 6
def trim_history(callback_context: CallbackContext, llm_request: LlmRequest):
"""Keep the system prompt and the last few turns. Summarise older ones elsewhere if needed."""
contents = llm_request.contents
if len(contents) > MAX_HISTORY_TURNS * 2:
llm_request.contents = contents[-MAX_HISTORY_TURNS * 2:]
return None # None means: continue to the model
faq_agent = LlmAgent(
name="faq", model="gemini-2.5-flash-lite",
description="Answers general, non-personal questions about card fees and opening hours.",
instruction="Answer only from the provided policy snippets. If unsure, say so.",
planner=NO_THINKING,
include_contents="none", # stateless: don't resend the conversation
before_model_callback=trim_history,
generate_content_config=types.GenerateContentConfig(temperature=0, max_output_tokens=300),
)Streaming without skipping the output guard
The part people get wrong. If you stream raw partial events to the browser you have bypassed your output controls, because the check ran (or didn't, depending on streaming mode and ADK version) on the aggregated text. Verify the behaviour in your version with a test that tries to leak a card number. A pattern that works: stream into a sentence buffer in the API layer, run the cheap deterministic checks (PAN regex, blocklist) on each completed sentence, and flush only the ones that pass. For healthcare, add a rule that clinical statements are only released if they came from an approved tool result or retrieved leaflet.
While tools run, send a status event ("Checking your recent payments…"). That costs nothing, and users tolerate a 3-second wait with progress better than a 1.5-second silence. Emit it from before_tool_callback into the same SSE stream.
11Exposing it
A streaming HTTP API in front of the Runner, with identity verified before ADK sees anything.
import json, uuid
from fastapi import Depends, FastAPI, HTTPException, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from google.adk.runners import Runner
from google.adk.agents.run_config import RunConfig, StreamingMode
from google.adk.sessions import DatabaseSessionService
from google.genai import types
from bank_assist.agent import root_agent
APP = "bank_assist"
sessions = DatabaseSessionService(db_url="postgresql+asyncpg://...") # Cloud SQL via the connector
runner = Runner(agent=root_agent, app_name=APP, session_service=sessions)
app = FastAPI()
class ChatIn(BaseModel):
session_id: str | None = None
message: str = Field(max_length=2000)
async def verified_user(request: Request) -> dict:
"""Validate the bearer token (Identity Platform / your IdP) and return claims.
Reject on any doubt. The agent never sees an unverified identity."""
token = request.headers.get("authorization", "").removeprefix("Bearer ").strip()
claims = await verify_jwt(token) # google.oauth2.id_token / your IdP SDK
if not claims:
raise HTTPException(401)
return claims
@app.post("/v1/chat")
async def chat(body: ChatIn, user: dict = Depends(verified_user)):
uid = user["sub"]
sid = body.session_id or str(uuid.uuid4())
session = await sessions.get_session(app_name=APP, user_id=uid, session_id=sid)
if session is None:
session = await sessions.create_session(
app_name=APP, user_id=uid, session_id=sid,
state={"customer_id": user["customer_id"], # from the token, not the request body
"customer_ref": user["pseudo_id"], "session_key": sid})
async def events():
msg = types.Content(role="user", parts=[types.Part(text=body.message)])
async for ev in runner.run_async(user_id=uid, session_id=sid, new_message=msg,
run_config=RunConfig(streaming_mode=StreamingMode.SSE)):
if ev.partial and ev.content and ev.content.parts and ev.content.parts[0].text:
yield f"data: {json.dumps({'delta': ev.content.parts[0].text})}\n\n"
elif ev.is_final_response():
yield f"data: {json.dumps({'done': True, 'session_id': sid})}\n\n"
return StreamingResponse(events(), media_type="text/event-stream",
headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"})
@app.get("/healthz")
def healthz(): return {"ok": True}Choices for the front door
| Option | Good for | Watch for |
|---|---|---|
| Cloud Run + API gateway (Apigee or API Gateway) | Full control of auth, quotas, SSE, custom controls. My default for regulated consumer products. | You own scaling config, patching and the HTTP surface |
| Vertex AI Agent Engine | Managed runtime, sessions and memory; less ops | Less control over the HTTP layer; confirm compliance scope and network options for your case |
| GKE | You already run a platform team and need sidecars, custom networking | More to run for the same agent |
| A2A protocol | Exposing a specialist to other teams' agents, or consuming theirs | Treat remote agents as untrusted input; same tool-policy rules apply |
| MCP servers | Reusing tools across agents and vendors | An MCP tool is still a tool: allow-list, scope credentials, log calls |
The edge, concretely
- Authentication: OIDC from your IdP, validated at the gateway and again in the service. Cloud Run set to require authentication, ingress limited to the load balancer.
- Per-user quotas: requests per minute and tokens per day, keyed to the verified user, enforced at the gateway. Prevents one account (or one prompt-injection loop) from becoming a bill.
- Abuse controls: Cloud Armor rules, bot detection, input size caps, concurrency caps per session (a customer shouldn't have three turns in flight).
- Response hygiene:
no-storeon chat responses, no model reasoning or tool arguments in the stream, structured error codes that don't leak internals. - Channel parity: the voice, in-app and web channels all call the same API so the controls apply once.
12Deploy and operate
Boring on purpose.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY bank_assist ./bank_assist
COPY server.py .
ENV PYTHONUNBUFFERED=1 GOOGLE_GENAI_USE_VERTEXAI=TRUE
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1"]gcloud run deploy bank-assist \
--source . --region europe-west2 \
--service-account bank-assist-runtime@$PROJECT.iam.gserviceaccount.com \
--no-allow-unauthenticated --ingress internal-and-cloud-load-balancing \
--min-instances 2 --cpu-boost --concurrency 20 --timeout 60 \
--vpc-egress all-traffic --network default --subnet agents-subnet \
--set-env-vars GOOGLE_CLOUD_PROJECT=$PROJECT,GOOGLE_CLOUD_LOCATION=europe-west2- Least-privilege service account. It can call Vertex AI, read its secrets and reach the specific backend APIs. Nothing else. Separate accounts for the agent runtime and for the CI deployer.
- Perimeter. Put the project in a VPC Service Controls perimeter, use Private Service Connect to the model endpoint where available, and deny egress by default. A prompt-injected agent that can't reach the internet can't exfiltrate to it.
- Release. Build once, promote the same image dev → staging → prod. Gate promotion on the eval suite (guide 09) and a signed-off prompt version. Roll out by traffic percentage with automatic rollback on guard-trigger rate, error rate or escalation rate moving.
- Region and residency. Check where inference actually runs for your chosen model and endpoint, not just where your service lives. For UK/EU data this is a design-time question, not a deployment-day one.
What to watch in production
Service health
Latency p50/p95/p99 per stage (TTFT, tool, full), error rate, timeouts, fallback usage, model quota errors.
Behaviour
Tool-call mix, transfers per agent, guard block rate, escalation rate, "confirmation requested vs completed", conversation length.
Quality & cost
Sampled online eval scores (guide 09), thumbs and complaints (guide 10), tokens and cost per resolved conversation.
ADK emits OpenTelemetry spans for agent runs, model calls and tool calls. Export them to Cloud Trace and you get a waterfall per turn, which is the fastest way to find out which of your nine latency techniques you actually need. Redact before export: spans can carry prompt text.
13Testing before it reaches a customer
Three layers, cheapest first.
- Unit tests, no model. Tools against a fake bank API;
tool_policywith a table of allowed and blocked calls;output_guardwith leak attempts. These run in seconds and catch most control regressions. - Trajectory evals. For a set of scripted conversations, assert the right agent got the turn and the right tools were called with the right arguments. ADK's evalsets and
adk evaldo this. Detail in guide 09. - Adversarial and scenario testing. Prompt injection via transaction descriptions (an attacker can pick a merchant name!), identity-swap attempts ("show me my husband's account"), requests for advice, a distressed customer, a patient describing chest pain. Keep every one you find as a permanent test.
import pytest
from types import SimpleNamespace
from bank_assist.guardrails import tool_policy
def ctx(agent, **state):
return SimpleNamespace(agent_name=agent, state=dict(state))
TOOL = lambda name: SimpleNamespace(name=name)
def test_dispute_blocked_without_confirmation():
r = tool_policy(TOOL("open_card_dispute"),
{"transaction_id": "t1", "reason": "duplicate"}, ctx("dispute_filer"))
assert r["status"] == "needs_confirmation"
def test_dispute_allowed_after_ui_confirmation():
c = ctx("dispute_filer", confirmed_txn="t1", txn_amount_gbp=40)
assert tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t1", "reason": "duplicate"}, c) is None
def test_wrong_agent_cannot_call_write_tool():
r = tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t1", "reason": "duplicate"}, ctx("spend_insights"))
assert r["status"] == "error"
def test_model_cannot_confirm_for_a_different_transaction():
c = ctx("dispute_filer", confirmed_txn="t1")
r = tool_policy(TOOL("open_card_dispute"), {"transaction_id": "t2", "reason": "duplicate"}, c)
assert r["status"] == "needs_confirmation"14Production readiness checklist
If you can't tick these, you have a demo.
Design
- Each agent justified by a permission, latency or ownership reason
- Identity from verified token, never from model arguments
- Writes gated by out-of-band confirmation and code-level limits
- Fixed, approved text for red flags, disclosures and refusals
- Hazard log or model-risk entry written and owned
Platform
- Pinned model and ADK versions; prompt version logged per turn
- Private networking, VPC-SC, CMEK where required, TTLs on sessions
- Min instances, same-region data path, timeouts and fallbacks
- Per-user quotas, concurrency caps, budget alerts, kill switch
- PII-free structured audit trail queryable in BigQuery
Quality
- Unit tests on tools and guards in CI
- Trajectory and response evals gate every release
- Red-team set including injection through data fields
- Sampled online evals and human review on live traffic
- Rollback rehearsed, not just documented
Speed
- Per-stage latency traced; budget agreed with product
- Streaming on, with sentence-level output checks
- Router on small model, thinking off, or no model at all
- Parallel reads, compact tool outputs, trimmed history
- No caching of personalised answers
The agent code is the easy 20%. The work that makes it shippable in a bank or a health system is identity, tool scoping, approval, audit and latency discipline, and almost all of it is ordinary software engineering applied to a component that happens to be non-deterministic. Build the controls so that the worst thing the model can do is say something unhelpful, then use evals (next guide) to find out how often it does.