Databricks as a system: data, AI, Lakebase and Lakewatch

Start with a request, then draw its boundaries

Imagine a support representative asking, “Why did this customer receive two invoices, and may I issue a credit?” This is an original design exercise. There is no deployed customer system or measured result behind it. A useful answer needs reliable business data, an explanation with evidence, authorization to perform the credit, durable transaction state, and a record of who acted. Buying a model endpoint settles only one of these questions.

Separate four responsibilities: analytical history, transactional state, AI reasoning and tools, and security investigation. Draw identity and governance across all four. A table, an agent memory record and a security alert can refer to the same customer while having different readers, retention policies and mutation rules. Sharing a platform does not make them one transaction or one permission boundary.

Scroll horizontally to read the diagram.

This is an original explanatory diagram, not a Databricks architecture drawing, screenshot or performance measurement. Open the full-size diagram. The arrows describe intended application relationships; they do not promise a native connector between every pair.

Give each service one job

Responsibility	What it contributes	What the application must still decide
Governed analytical data	Historical evidence for reporting and retrieval	Grain, business definitions, freshness and sensitive columns
Unity Catalog	Asset discovery, grants, lineage and auditing	Which identity executes each operation and how access is provisioned
AI services and agent tools	Reasoning over evidence and invoking selected tools	Allowed actions, evaluation criteria, confirmation and idempotency
Lakebase	Postgres state for application transactions or agent state	Schema, transaction boundaries, isolation, connection behavior and retention
AI/BI	Dashboards, conversational analysis and reusable semantics	Which measure is authoritative and how an answer is checked
Lakewatch	Security telemetry, detection and investigation	Enablement, response authority and operational incident procedures

Unity Catalog documents a catalog/schema/object hierarchy for many securable assets. Managed and external assets have different storage-lifecycle responsibilities. A governed object name is therefore not merely a folder name. In our exercise, label the analytical invoice table, the function that reads it and the principal that runs the function separately; then list the privilege required at each edge.

Agent documentation separates model access, authoring, tools, deployment and evaluation. MLflow tracing can expose the steps of an agent; evaluation can inspect quality, cost and latency. Neither a fluent answer nor the existence of a trace proves that issuing the credit was authorized. Design the write operation as a bounded application function with a request ID, a deterministic validation step and a recorded result.

Model the data before adding the assistant

Choose the grain of each record. For the exercise, an invoice line is not an invoice, a payment attempt is not a settled payment, and a customer is not an account. Draw customer → invoice → invoice_line and invoice → payment_attempt as separate relationships. State whether a refund belongs to the invoice, payment or line. Decide the meaning of “duplicate” before asking a model to explain it.

1. 1Source events
2. 2Validated invoice grain
3. 3Shared business measure
4. 4BI answer

1. 1Question
2. 2Authorized evidence tool
3. 3Proposed action
4. 4Application validation
5. 5Recorded transaction

Build a small paper fixture with two customers, three invoices and four payment attempts. Include a retried payment, a late event and a missing customer reference. Write the expected outstanding balance by hand. If two tools use different definitions of “paid”, fix the definition first. The platform's AI/BI documentation describes dashboards, Genie Agents and Unity Catalog semantics as related components. Our fixture adds the business acceptance test that their presence alone cannot supply.

Lakebase connects operational state to analytical use

Lakebase's overview describes managed Postgres, serving lakehouse data through synced tables, and storing Postgres changes as Delta for downstream use. The latter path is explicitly Public Preview in the reviewed overview. Draw each direction and its freshness contract. Do not assume that an arbitrary Postgres write appears immediately in every analytical table.

For this exercise, keep credit_request(request_id, customer_id, status) as operational state and an analytical credit history as a distinct data product. The proposal may be accepted, rejected or still unknown after a network interruption. Specify how a retry looks up the request ID instead of creating another credit. This is application design, not a claim that the following table is a built-in Lakebase configuration.

Choice or setting	Effect on the design	Tradeoff to evaluate
Autoscaling minimum and maximum CU	Bound active compute capacity	A low minimum may constrain the working set; a high maximum permits more compute spend
Scale-to-zero timeout	Suspend after inactivity	More idle savings can mean more wake-ups and lost session context
Development branch	Isolate a schema experiment	A branch is not a continuous subscription to the parent's later changes
Serving synced analytical tables	Give the application a serving path	Observe replication freshness rather than treating “synced” as synchronous
Application request ID	Identify one intended mutation	Requires a designed uniqueness and retry contract

The reviewed autoscaling page limits the autoscaling range difference to 16 CU and scale-to-zero eligibility to a maximum of 32 CU. Autoscaling with HA has additional conditions, including no scale to zero. These are current AWS reference conditions, not universal cloud guarantees. A paper choice of 2–8 CU fits that range; it does not establish a capacity target or a price.

The scale-to-zero reference allows an inactivity timeout from 60 seconds to seven days and describes session-context reset after suspension. Suppose our internal support app is unused overnight. Compare a short timeout with keeping the compute active, and list what a cold request must reinitialize. Measure first-request latency and connection errors in an authorized environment before choosing a production timeout; this exercise has not measured them.

Project documentation says a child branch inherits the parent's data at creation, with later changes not automatically propagating. Use a development branch to test an invoice schema migration, then separately design deployment and data migration. A branch does not make it safe to expose production customer data to a broader development group.

External access: query now or ingest first?

Federation documentation distinguishes read-only query federation over remote database compute from catalog federation over object storage on Databricks compute. It also distinguishes these from Spark data sources when writing or more execution control is required.

For the invoice investigation, occasional read-only access to an external billing system may suit federation. A repeated historical workload may justify ingestion with an explicit freshness target. Compare remote database load, transfer and compute cost, credential ownership, schema drift and failure diagnosis. “No copy” saves one movement path; it does not remove the source database's availability dependency. Draw a timeout at the remote boundary and decide whether the assistant should say “evidence unavailable” instead of inventing a balance.

Lakewatch is a security product with an enablement boundary

The 24 March 2026 launch describes Lakewatch as an agentic SIEM combining security, IT and business data, and explicitly calls its launch Private Preview. That launch state is the dated evidence retained here. Current GA, regional endpoints, account entitlement and actual response behavior were not verified; a marketing description is not a tested availability matrix.

Our design example places security telemetry on a separate investigation path. A model trace explains an answer; an audit event explains an access; a detection evaluates a security hypothesis. These records may join during an investigation, but they answer different questions. Ask whether a sequence of denied data reads followed by a tool invocation should open a case. Keep the first exercise read-only: specify the signals, expected evidence and reviewer decision without wiring any containment action.

Read changes as evidence for design decisions

The reviewed September release page dates managed agent memory/sessions as Beta on 16 September and Agent Bricks CLI as Beta on 29 September. The October page dates JDBC Unity Catalog connections GA on 2 October with compute-version conditions; OAuth M2M remains Beta. Those entries change integration choices, but do not show that this account has received the staged rollout. The 5 October group-run pipeline entry is future-dated at our 4 October review and is excluded from released capabilities.

A 7 May community article by Brahmareddy reports enthusiasm about combining applications and analytics. It is an individual account, not a product contract or our measurement. Its broad “no sync” framing should not replace the official overview's explicit sync and change-feed paths. Use it to ask a sharper question: which exact path, freshness and product generation did the author observe?

Exercise: defend one design

This offline exercise is unexecuted. On one page, draw the invoice evidence path, the credit transaction path and the investigation path. Give every arrow an input, output, identity and failure state. Choose a table grain, an application request ID, a serving freshness target and a scale-to-zero assumption. Record cost drivers rather than invented prices.

Then test three counterexamples on paper: the source times out; the proposed credit is repeated; the agent can read invoices but cannot mutate them. Define what the user should see and which record proves the result. Explain when you would keep a conventional external Postgres service or ingest data instead of using federation. Continue with Auto Loader design and the governed-agent case study. Certification practice is a separate path from the Databricks learning hub; an exam score does not validate this design.