Durable Execution · Temporal · 2026

Temporal, from the inside — and when you actually need to deploy it on GCP

Temporal turns fragile, multi-step business logic into code that survives crashes, restarts, and week-long waits. This walkthrough opens the box: how an append-only event history makes execution durable, the four server services that run the show, how a task threads through them, and the database underneath — then the practical decision most teams get wrong: self-host on Google Cloud, or reach for Temporal Cloud?

Topic: Temporal Server architecture + self-hosting on GKE & Cloud SQL · ~20 min
0 / 6 sections
1
The core idea: durable execution via an event history
crash the worker mid-workflow — the state comes back

A normal program keeps its state in memory. Kill the process and that state is gone — the half-charged order, the "waiting 3 days for the trial to end" timer, all of it. Temporal's whole reason for existing is to make that not happen.

The trick is event sourcing. Your Workflow code doesn't store state in variables that live only in RAM — every state-changing decision it makes (an activity was scheduled, an activity completed, a timer started, a signal arrived) is written as an event to an append-only Event History in a database. If the worker running your code dies, another worker picks up the workflow, replays the history to deterministically rebuild the exact in-memory state, and continues as if nothing happened.

Two rules fall out of this: your Workflow code must be deterministic (replaying the same history must produce the same decisions), and anything non-deterministic or with side effects — an API call, a DB write, reading the clock — must live in an Activity. Activities are recorded by their result, so on replay they are not re-run; the recorded result is handed back.

Demo — run a workflow, kill the worker, watch it recover
Event History 0 events
Worker (in-memory state)
Payment charged
Fraud check passed
Inventory reserved
Order shipped
Workflow complete
Idle. Press Start.
The left log is persisted in the database. The right panel lives only in the worker's memory — that's what a crash destroys.
The payoff: the durability lives in the history, not the process. Any worker, anywhere, can rebuild a workflow's state from its event log — so a crash, a deploy, or a machine dying becomes a non-event.
When a worker crashes mid-workflow, how does Temporal recover the state?
2
The four server services
frontend · history · matching · worker — each scales on its own

The "Temporal Server" ships as a single binary, but inside it are four independent services that can be scaled and even deployed separately. Understanding what each one does is the whole game — it tells you where load goes and what to size when you self-host.

One distinction to hold onto: the Worker service below is the server's internal worker (replication, archival, scheduled/system workflows). It is not your application Workers — those are separate processes running your Workflow and Activity code, which connect in from outside.

Demo — hover each service to see what it owns
🚪
Frontend Service
stateless gRPC gateway — the only door in
stateless
Every call from clients, your Workers, and the CLI/UI hits Frontend first. It handles rate limiting, authentication, request validation, and routing to the right internal service. Stateless, so you scale it horizontally for throughput.
📜
History Service
owns workflow state, event history, timers, retries
the engine
The heart of Temporal. It persists each workflow's event history and mutable state, drives timers and activity retries, and generates the tasks that move a workflow forward. Partitioned into history shards for horizontal scale — this is the service you size most carefully.
🔀
Matching Service
hosts task queues, hands tasks to workers
task queues
Hosts the user-facing Task Queues. When History creates a task, Matching holds it until a Worker polls for work, then dispatches it. Supports sticky queues (route a workflow back to the worker that already has its state cached) to avoid re-replaying long histories.
⚙️
Worker Service (internal)
replication, archival, scheduled & system workflows
background
The server's own background worker: it runs system workflows (batch operations, Schedules), handles multi-cluster replication, and archival. Do not confuse it with your application Workers, which run your code in separate processes.
All four talk to the same persistence layer underneath. Frontend is stateless; History holds the durable state; Matching brokers tasks; the internal Worker runs housekeeping.
Sizing intuition: Frontend scales with request volume, Matching with task-dispatch throughput, and History with the number of concurrent open workflows and their write rate. When self-hosting bites, it's almost always History (and the database behind it).
In Temporal's server, what does the "Worker Service" refer to?
3
How one task threads through the system
workers poll — they are never pushed to

Follow a single step of a workflow and the architecture clicks into place. The key surprise for newcomers: Temporal never pushes work to your Workers. Workers long-poll a Task Queue and pull tasks when they're ready. That's what lets you scale workers up and down freely, deploy them independently, and survive them all being offline for a while — the tasks just wait.

Demo — walk (or Play) one workflow task, hop by hop
Click a step to inspect it — or press Play to walk the path.
Why polling wins: because Workers pull, the server needs no connection back to them, no knowledge of their addresses, and no re-delivery machinery. A task sits safely in the queue until some worker is ready — that's the backbone of Temporal's at-least-once, never-lost task delivery.
How do your Workers get work from the Temporal Server?
4
Persistence & history shards
the database is pluggable — the shard count is forever

Temporal is stateless in its services and stateful in its database. The persistence layer is pluggable, with two logical stores:

The two stores

Default storeevent history, mutable state, task queues, timers
Visibility storelist / search workflows

The default store runs on Cassandra, PostgreSQL, or MySQL. The visibility store can share that same SQL/Cassandra database for basic listing, or point at Elasticsearch / OpenSearch for Advanced Visibility — rich search over custom attributes. Pick your database by scale: SQL (often managed, like Cloud SQL) is simplest; Cassandra is the path for very high write throughput.

History shards — the one knob you can't take back

The History service partitions all workflows across a fixed number of history shards. Each workflow execution is hashed by its ID to exactly one shard, and each shard is processed by one owner at a time — so the shard count is your ceiling on concurrent throughput. It's set once at cluster creation and cannot be changed afterward. Too few and you cap your scale; too many and you pay coordination overhead on a small cluster.

Demo — see how a workflow ID maps to a shard
Type a different ID — the same ID always lands on the same shard (deterministic hashing). Change the shard count and the mapping shifts entirely, which is exactly why you can't resize it on a live cluster.
Plan the shard count up front. Common production clusters use 512–4096+ shards. Since you can't change it later, size for where you're going, not where you start — a fresh cluster + migration is the only way to change it.
Why does the number of history shards matter so much?
5
When do you actually need to deploy it yourself?
Temporal Cloud vs self-hosting — be honest about the tradeoff

Here's the part teams get wrong: they self-host by default because it's open source, then discover they've signed up to operate a sharded, database-backed distributed system 24/7. Running Temporal well means owning the database, the shards, upgrades, backups, and on-call. Temporal Cloud takes all of that off your plate for a consumption price (per "Action").

Self-hosting genuinely wins in a few situations — regulated data that must stay in your VPC or region, strict network isolation, a strong platform team that already runs Kubernetes and Cassandra, or very high volume where the all-in cost finally tips in your favor. Use the finder to pressure-test your own case.

Demo — answer four questions, get a recommendation
Data must stay in your own VPC / region (residency, sovereignty)?
Air-gapped / strict network isolation?
Do you have a platform team already running Kubernetes + a scalable DB?
Scale (rough Actions / month)
< 5M
The scale threshold is a rough heuristic, not a hard line — self-hosting only tends to become cost-competitive at high volume (often cited around ~30–50M Actions/month) and only if you already operate the infrastructure. Below that, Cloud is usually cheaper all-in once you count the ops time.
Default position: start on Temporal Cloud unless a hard requirement (residency, isolation) or a genuine scale-plus-platform-team case pushes you to self-host. Migration paths exist in both directions, so this isn't a one-way door.
Which is the strongest reason to self-host Temporal instead of using Temporal Cloud?
6
Deploying self-hosted Temporal on GCP
GKE + Cloud SQL (or Cassandra) via the official Helm chart

If you've decided to self-host, here's the concrete GCP shape. The server services and your Workers run on GKE; the durable state lives in Cloud SQL for PostgreSQL (or self-managed Cassandra for very high scale); and the official temporalio/helm-charts wire it together. Click a step to see what it involves — or run through the whole sequence.

Demo — the GCP deployment sequence
Click a step to see its detail — or run the whole sequence.

The moving parts on GCP

ComputeGKE — services + your Workers
Default storeCloud SQL (PostgreSQL/MySQL) or Cassandra
VisibilityElasticsearch / OpenSearch
DB accessCloud SQL Auth Proxy + Workload Identity
Ingressinternal gRPC LB; UI behind IAP
ObservabilityPrometheus / Cloud Monitoring + Grafana
The Cloud-native shortcuts: use the Cloud SQL Auth Proxy as a sidecar with Workload Identity so no static DB password lives in the cluster — recent Temporal versions even support a passwordCommand to fetch short-lived Cloud SQL IAM tokens per connection. Keep the frontend on an internal load balancer and secure it with mTLS; don't expose it to the public internet.
Don't skip the schema job. Temporal will not start against an empty or unversioned database. The Helm chart's schema-setup job runs temporal-sql-tool to create and version both the default and visibility schemas before the services come up.
On GKE, what's the recommended way to let Temporal reach a private Cloud SQL instance without a static password?

🎉 You've got the whole picture

Durable execution comes from an append-only event history that any worker can replay. Four services run it — frontend (gateway), history (the engine, sharded), matching (task queues), and the internal worker — all over a pluggable database whose shard count is fixed for life. And the deploy decision is a real one: reach for Temporal Cloud unless residency, isolation, or scale-plus-a-platform-team push you to self-host on GKE + Cloud SQL.

Learning Reference · Temporal Service docs · Self-hosted guide · temporalio/helm-charts

Get new posts in your inbox

No spam — just new posts and learning pages when they ship.

Share X LinkedIn Reddit Hacker News