Chapter 6: Memory Configuration — Getting It Right

Memory configuration is the single highest-impact tuning dimension for Starburst Enterprise. Get it wrong and you get either constant OOM failures (configured too tight) or wasted capacity (configured too loose with too-generous per-query limits). Get it right and the cluster runs predictably under heavy concurrent load.

Before configuring anything, you need to understand what is actually consuming memory on each node.

6.1 The Memory Hierarchy — Every Layer Accounted For

A common assumption is that assigning 128GB of RAM to a Trino worker means 128GB is available for queries. The reality is more layered:

6.2 The Memory Allocation Formula

The official Starburst documentation states: ‘Typically, values representing 70 to 85 percent of the total available memory is recommended’ for the JVM heap. For larger nodes, the percentage can be lower because absolute headroom grows. Here is how I think about allocation on a 256 GB worker:

6.3 Correct Configuration Properties (SEP 453-e and Later)

This is where I want to be very precise, because several property names have changed across SEP versions and incorrect properties cause the cluster to fail to start. The following are correct for SEP 453-e LTS and all later versions including the current 477-e LTS.

config.properties on All Nodes

# === MEMORY MANAGEMENT ===

# Maximum distributed (cluster-wide) USER memory a single query can use
# User memory = data directly attributable to query execution (hash tables, sort buffers)
# Default: 20GB — almost always needs tuning upward for production workloads
query.max-memory=500GB

# Maximum user memory a single query can use on a single worker node
# The sum of (query.max-memory-per-node + memory.heap-headroom-per-node)
# MUST be less than the JVM heap (-Xmx). SEP validates this at startup.
query.max-memory-per-node=150GB

# Memory reserved as headroom for untracked JVM allocations
# Default: JVM max memory * 0.3
# Increase if workers are experiencing unexpected OOM errors despite query memory limits looking fine
memory.heap-headroom-per-node=50GB

6.4 Understanding query.max-memory — The Cluster-Wide Cap

This is one of the most commonly misunderstood settings in Starburst/Trino deployments. query.max-memory is NOT a per-node limit. It is the total memory a single query is allowed to consume across all worker nodes combined.

Example: If you have 25 workers and set query.max-memory=500GB, a single query can use up to 500GB distributed across all 25 machines — approximately 20GB per worker on average. In practice, distribution is uneven (JOIN build sides concentrate memory on specific nodes), which is why per-node limits also matter.

How to set query.max-memory:

• Start with a value that covers your largest realistic single-query memory consumption (measure with EXPLAIN ANALYZE on your heaviest queries)

• Set it high enough that legitimate large queries succeed, but low enough that a runaway query cannot consume the entire cluster

• Use resource groups to apply different query.max-memory limits for different user groups — interactive users get lower limits than ETL jobs

6.5 Memory and Data Skew

Even with correct memory configuration, data skew can cause EXCEEDED_LOCAL_MEMORY_LIMIT errors on individual workers while the cluster-wide memory usage looks fine. This happens because the per-node limit (query.max-memory-per-node) is enforced independently on each worker, and a skewed key causes one worker to accumulate disproportionately large hash table data.

Example from production: A JOIN on user_id where user_id=0 represents 30% of all rows. Worker assigned hash(0) % 25 must build a hash table 7.5x larger than the average worker (30% vs 4%). If query.max-memory-per-node is set to the average expected usage, this worker will OOM while others have abundant headroom.

Remediation options:

• Pre-filter known skewed values before the JOIN, If user_id=0 (anonymous) is not analytically relevant, filter it out with WHERE user_id != 0

• Increase query.max-memory-per-node to handle the skewed case , but this reduces concurrent query capacity

• Restructure the query to avoid the skewed join key being the shuffle key

• For GROUP BY skew: GROUP BY with skewed keys benefits from partial pre-aggregation on workers (Trino does this automatically) before the final shuffle

Chapter 7: JVM Configuration — The Current Official Template

Trino is a Java application and JVM configuration directly affects query throughput, GC pause times, and stability. The JVM configuration has evolved significantly as SEP has moved through Java versions. What follows is the current official template from the Starburst Kubernetes configuration documentation, validated against SEP 477-e LTS.

7.1 Java Version Requirements by SEP Version

This is operationally critical. SEP has strict Java version requirements that change with each LTS release, and using the wrong Java version causes SEP to refuse to start.JDK is now bundled: Since SEP 462-e, SEP ships with the correct JDK version bundled. You do not need to separately install Java on nodes. The bundled JDK is Eclipse Temurin OpenJDK from Adoptium. If you set JAVA_HOME to override the bundled JDK, ensure it matches the minimum required version for your SEP release, using an older JDK causes startup failure.

7.2 The Official jvm.config Template (SEP 477-e LTS)

This is taken directly from the official Starburst Kubernetes configuration examples at docs.starburst.io/latest/k8s/sep-config-examples.html. For bare-metal or VM deployments, the same flags apply in etc/jvm.config, with -Xmx set explicitly instead of using RAM percentage flags:

server
-Xmx205G                           # Set to 70-85% of total node RAM                                   
-XX:InitialRAMPercentage=80        # K8s-friendly alternative to explicit Xmx & use InitialRAMPercentage/MaxRAMPercentage instead
-XX:MaxRAMPercentage=80            # K8s-friendly alternative to explicit Xmx
-XX:G1HeapRegionSize=32M           # G1GC region size — 32M recommended for large heaps
-XX:+ExplicitGCInvokesConcurrent   # Prevents stop-the-world GC when System.gc() is called
-XX:+ExitOnOutOfMemoryError        # CRITICAL: exit cleanly on OOM; let process manager restart
-XX:+HeapDumpOnOutOfMemoryError    # Write heap dump for post-mortem analysis
-XX:-OmitStackTraceInFastThrow     # Preserve stack traces even for frequently thrown exceptions
-XX:ReservedCodeCacheSize=512M     # Code cache for JIT-compiled query operators
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000
-Dfile.encoding=UTF-8
-XX:+EnableDynamicAgentLoading    # Required for JOL (Java Object Layout) agent support

7.3 Key JVM Principles Explained

Why -XX:+ExitOnOutOfMemoryError Is Critical

When a JVM hits an OutOfMemoryError, the JVM is in an inconsistent state. Objects may be partially initialised, locks may be held, and the garbage collector may be unable to recover. Attempting to continue running after an OOM typically leads to cascading failures, corrupted state, and queries that appear to hang rather than fail cleanly.

With -XX:+ExitOnOutOfMemoryError, the JVM exits immediately when OOM occurs. The process manager (systemd on VMs, Kubernetes on containers) restarts the Trino process in a clean state. Other workers that were not affected continue serving queries. The coordinator detects the worker departure and marks any in-flight queries on that worker as failed, allowing clients to retry.

Why OS Swap Must Be Disabled

Trino’s JVM uses G1GC (Garbage First Garbage Collector), a compacting garbage collector. G1GC works by moving objects between memory regions and continuously compacting the heap to eliminate fragmentation. Because objects move frequently, there are no stable ‘cold’ memory regions that the OS could safely page to swap disk.

If the OS swaps any part of the JVM heap, G1GC may need to touch that swapped page during a collection cycle, triggering a page fault and dramatically extending GC pause times. A GC pause that should take 200ms can extend to many seconds when swap is involved, manifesting as erratic query latency spikes.

The official Starburst documentation explicitly states: ‘Allocation of all memory to the JVM or using swap space is not supported, and disabling swap space on the operating system level is recommended.’

The Code Cache — An Often Overlooked Limit

Trino uses runtime code generation for query operators & it generates optimised Java bytecode at query execution time based on the specific types and operations in each query. This generated code is stored in the JVM code cache.

If the code cache fills up (-XX:ReservedCodeCacheSize), the JVM stops compiling new code and falls back to interpreted execution. This causes a progressive and difficult-to-diagnose performance degradation & queries get slower over time as the cache fills, but there may be no obvious error. The 512MB setting in the official template is appropriate for production SEP workloads.

7.4 Kubernetes-Specific JVM Considerations

When running SEP on Kubernetes, several JVM behaviours require specific attention:

• CPU limits vs. requests: Do NOT set CPU limits on SEP coordinator or worker pods. The official Starburst Kubernetes documentation documents this explicitly hard CPU limits cause JVM CPU throttling even when the node has headroom, degrading performance. Set CPU requests only.

• Container memory limits: Set memory limits equal to memory requests to prevent OOM eviction. The JVM heap (-Xmx or MaxRAMPercentage) plus non-heap overhead must fit within the container memory limit.

• JVM cgroup awareness: Modern JDK versions (17+) are cgroup-aware InitialRAMPercentage and MaxRAMPercentage calculate percentages of the container’s memory limit, not the node’s total RAM. This makes K8s deployments cleaner than specifying absolute -Xmx values.

  # Recommended K8s resource configuration (values.yaml)
  coordinator:
    resources:
      requests:
        memory: '256Gi'
        cpu: 32
      limits:
        memory: '256Gi'
        # NO cpu limit — critical for JVM performance
  worker:
    resources:
      requests:
        memory: '256Gi'
        cpu: 32
      limits:
        memory: '256Gi'
        # NO cpu limit — critical for JVM performance
  

Chapter 8: Disk Spilling Why I Recommend Against It

Spilling is Trino’s mechanism for reducing memory consumption during JOIN and aggregation operations by writing intermediate data to local disk and reading it back in batches. It is a legitimate feature that can save queries that would otherwise fail with OOM errors. My recommendation after years of production experience: design your cluster so you never need it.

8.1 How Spilling Works

When a query exceeds its memory limit during execution:

• JOIN spill: The build-side hash table (and matched probe rows) are serialised to disk in sorted partitions. Each partition is later read back from disk and the join continues partition by partition. The join completes, but requires multiple passes of disk I/O.

• Aggregation spill: Partial aggregation accumulators are written to disk in sorted order and then merged in a second pass similar to an external sort.

8.2 The Real Cost of Spilling

The mechanics sound manageable. The production reality is less friendly:

8.3 Configuration

Spilling is disabled by default in Trino/SEP. If you are on an older version where it was previously enabled or are considering enabling it, the current configuration properties are:

# Default: false — spilling disabled
# Only enable if you have a specific, accepted use case
spill-enabled=false

# If enabling spill, configure the spill path to a fast local disk
# Do not use object storage or network filesystems for spill
spiller-spill-path=/var/trino/spill
max-spill-per-node=500GB
query-max-spill-per-node=100GB

What to do instead of spilling:

1. Add memory to worker nodes , the right long-term fix.

2. Rewrite the query to use approximate functions (approx_distinct, approx_percentile) dramatically lower memory requirements.

3. Pre-aggregate or pre-sort the data to reduce JOIN memory requirements.

4. Use resource groups to prevent multiple memory-heavy queries from running concurrently.

5. Use bucketing to enable node-local joins that eliminate the distributed hash table pattern. All of these are better than relying on spill.

Chapter 9: Task Concurrency and Worker Scheduling

After memory and JVM, the next configuration layer is how Trino schedules work within and across workers. Trino uses a cooperative, fair-share task scheduler all queries share the same thread pool on each worker, with preemption to prevent long queries from starving short ones.

9.1 Core Scheduling Properties

These go in config.properties on all nodes. Verify property names against docs.starburst.io/latest/admin/properties.html for your specific SEP version, as property names occasionally change across major releases:

# === TASK AND SCHEDULING ===

# Number of concurrent processing threads per task on each worker
# Rule of thumb: set to 2x physical core count (NOT vCPU count on hyperthreaded instances)
# Example: m7i.16xlarge has 32 physical cores → task.concurrency=64 (or match vCPU count)
# Example: Graviton ARM has no hyperthreading → task.concurrency = vCPU count
task.concurrency=32

# Threads handling HTTP data exchange between workers (shuffle network traffic)
# Increase for high-concurrency clusters with many simultaneous queries
task.http-response-threads=100

# Timeout for HTTP data exchange between workers
# Increase if network latency between nodes is high or cluster spans availability zones
task.http-timeout=2m

9.2 Split Assignment Configuration

# Maximum number of splits queued per worker node
# Increase for workloads with many small splits (many small files)
node-scheduler.max-splits-per-node=100

# Maximum pending (not yet started) splits per active task on a worker
node-scheduler.max-pending-splits-per-task=10

9.3 Resource Groups — Sharing the Cluster

Resource groups are the mechanism for controlling how concurrent queries share cluster resources. They define limits on the number of running and queued queries per group, and they implement a psychology-based approach to fairness: the design goal is to maximise user happiness, not just mathematical resource utilisation.

From the original Presto Training Series (Dain Sundstrom, Starburst 2020): ‘Focus on maximizing user happiness — psychology not computer science. User experience should match expectations. Small, fast, trivial queries should run immediately. People hate waiting in line.’

SEP Resource Group Systems

SEP 477-e LTS offers two resource group implementations:

• Built-in Resource Groups: The current and preferred approach. Configured via the Starburst Enterprise Web UI under the Workload Management section, or via the Starburst REST API. Enabled by setting resource-groups.configuration-manager=starburst in config.properties.

• Legacy SEP Resource Groups: JSON-file-based configuration, compatible with the Trino open source resource group format. Still functional but superseded by built-in resource groups in current SEP.

Designing Effective Resource Groups

The principles I apply in practice:

• Separate interactive from batch: Interactive queries (dashboards, ad-hoc exploration) have latency expectations measured in seconds. Batch ETL jobs have latency expectations measured in minutes. Give each group enough capacity that interactive queries are never blocked by batch jobs.

• Allow every user at least one concurrent query: No one should hit a queue waiting to start even a single query during normal operation. Queuing should only occur when a user or team is genuinely overloading their allocation.

• Group by organisational unit: Teams regulate each other better than administrators can regulate everyone. Let team leads see what their team is running and kill queries in their group.

• Set conservative initial limits and expand: It is easier to relax a limit that is too tight than to tighten a limit that users have come to expect as available capacity.

Example Resource Group Structure

# config.properties — enable built‑in resource groups
resource-groups.configuration-manager=starburst

# Example hierarchy (configure via Web UI, REST API, or resource-groups.json)
# Semantics only, not a literal config file

Global (root group)
  softMemoryLimit: 80%      # Soft memory ceiling (queries can briefly exceed)
  hardConcurrencyLimit: 150 # Total concurrent running queries cluster‑wide
  maxQueued: 1000           # Max queued before rejecting new queries

  Interactive (subgroup)
    softMemoryLimit: 50%
    hardConcurrencyLimit: 75
    maxQueued: 300
    schedulingPolicy: fair     # Each user gets equal share

  Batch-ETL (subgroup)
    softMemoryLimit: 30%
    hardConcurrencyLimit: 20
    maxQueued: 500
    schedulingPolicy: weighted # Priority‑weighted execution

  System (subgroup for internal jobs)
    softMemoryLimit: 10%
    hardConcurrencyLimit: 10
    maxQueued: 100

9.4 Autoscaling on Kubernetes

When running SEP on Kubernetes (EKS, AKS, GKE), you can configure Kubernetes autoscaling to add worker pods under load and remove them when idle. This is particularly useful for:

• Bursty analytical workloads where peak demand is 3–5x average demand & paying for peak capacity 24/7 is wasteful

• Cost optimisation when workloads follow business-hours patterns

• Scaling for large one-off jobs without permanently increasing cluster size

SEP integrates with the Kubernetes Horizontal Pod Autoscaler (HPA) via KEDA (Kubernetes Event-Driven Autoscaling), which allows scaling based on Trino-specific metrics like queue depth or running query count. Configure KEDA integration through the SEP Helm chart values.yaml. We’ll explore this concept in more detail in Part 4 of this series.

Chapter 10: Hive Caching — What It Does and What It Does Not Do

Starburst Enterprise provides several caching layers for Hive connector operations. Caching can help with specific bottlenecks, but each layer has trade-offs that matter in production.

10.1 Metastore Metadata Caching

The Hive Metastore Service (HMS) is a relational database that Trino queries at the start of every query to resolve table schemas and list partitions. A slow HMS causes queries to sit in the planning phase with CPUs idle.

# Cache HMS table and partition metadata in Trino’s memory
hive.metastore-cache-ttl=10m
hive.metastore-refresh-interval=5m
hive.metastore-cache-maximum-size=10000

10.2 File Listing Cache

Before reading data from HDFS or object storage, Trino must list files in the relevant partition directories. With millions of small files in S3, this can take seconds to minutes.

# Cache file listing results to avoid repeated S3/HDFS list operations
hive.file-status-cache-expire-time=5m
hive.file-status-cache-size=1000000

10.3 Native File System Data Cache

SEP’s native file system cache stores actual file data blocks on Presto worker local disks, enabling repeated access to the same data without hitting object storage each time.

# CURRENT property name 
fs.cache.enabled=true
# Configure the cache storage path (fast local SSD recommended)
fs.cache.directories=/mnt/cache/trino

Reality check on the native data cache: It helps for repeated access to the same data by the same worker with a pattern that occurs in interactive dashboards where users repeatedly query the same date range. It does not significantly help for:

• First-time access to data (cold cache always misses)

• Queries that access different data ranges on each run (most ETL/batch patterns)

• Object storage (S3, GCS, ADLS) performance — the cache helps, but network round-trip latency to S3 is already low, so the benefit is less pronounced than for HDFS

In the next part, Part 3: Query & Workload Tuning, we’ll dive into EXPLAIN ANALYZE, Cost-Based Optimization (CBO), and effective data organization strategies

Further Reading

The concepts covered in this part are grounded in official Starburst documentation and Trino fundamentals. The following references are recommended for deeper exploration and version-specific details.

• Starburst Enterprise Documentation – Resource Management Properties

• Starburst Enterprise Documentation – Deployment Basics

• Starburst Enterprise Kubernetes Configuration Examples

• SEP Breaking Changes (LTS Compilation)

• Starburst Workload Management Documentation

• Trino: The Definitive Guide (2nd Edition) Fuller, Moser, Traverso — O’Reilly,
  

What sparked this series: A junior engineer on our team noticed that RAM utilization on our Starburst cluster was consistently above 80%, with all 25 worker nodes running at full capacity. Rather than raising an incident, I realised this was a teaching moment about how MPP query engines actually work and it became the foundation for this entire series of posts.

About This Series

This is a three-part practitioner series on Starburst Enterprise performance tuning. Each part is standalone but they build on one another. The series covers what I have learned deploying and tuning Starburst Enterprise in production, cross-referenced against official Starburst documentation (SEP 477-e LTS, November 2025) and the Presto/Trino Training Series.

Chapter 1: The Naming Story — Presto, PrestoSQL, and Trino

Before any performance discussion, you need to understand the naming landscape. If you have read documentation, training slides, blog posts, and academic papers about this technology, you will have encountered ‘Presto’, ‘PrestoSQL’, ‘PrestoDB’, and ‘Trino’ used in ways that seem interchangeable but are not. This causes real confusion, so let me set the record straight.

1.1 Origins at Facebook (2012–2018)

Presto was created at Facebook in 2012 by four engineers Martin Traverso, Dain Sundstrom, David Phillips, and Eric Hwang as a replacement for Apache Hive. The problem they were solving was straightforward: Hive was too slow for the interactive analytics Facebook needed over its multi-petabyte Hadoop data warehouse. Facebook open-sourced the project in November 2013. Inside Facebook the project was called PrestoDB to distinguish it from unrelated projects.

1.2 The 2018-2019 Governance Disagreement and Fork

In late 2018, Facebook made a governance decision that the original creators considered incompatible with healthy open source community management: granting Facebook engineers commit access without the standard community merit process. The four founders left Facebook and in January 2019 forked the codebase, creating the Presto Software Foundation as an independent non-profit. Their fork was initially named PrestoSQL , the same software, same team, different governance.

1.3 The December 2020 Rename to Trino

In September 2019, Facebook transferred PrestoDB to The Linux Foundation, which also applied for and received a trademark on the name ‘Presto’. After extended negotiations between the communities, the PrestoSQL team was forced to rename. On December 27, 2020, PrestoSQL was officially renamed Trino. The community, codebase, and leadership remained identical only the name changed. Since the fork, Trino has seen approximately three times the development velocity of PrestoDB.

1.4 Where Starburst Fits

Note: When you see ‘Presto’ in Starburst training materials from 2020 (including the training series that informs much of this guide), that referred to what is now called Trino the same engine, same original authors, different name. The architecture, concepts, and configuration principles documented in those 2020 sessions remain fully valid for current Trino/Starburst Enterprise.

1.5 Why Starburst Documentation Still References ‘Presto’

You will still encounter ‘Presto’ references in Starburst’s own documentation, especially in older blog posts, some connector code, and historical training materials. This is because:

• Content written before December 2020 legitimately referred to the engine as Presto (PrestoSQL)

• The academic research paper ‘Presto: SQL on Everything’ (published by Facebook engineers, 2019) predates the rename and is universally cited by its original title.

• Some third-party tools use the Presto JDBC driver or Presto SQL parser for compatibility , the SQL dialect between Trino and PrestoDB remains largely compatible.

• Some GCP marketplace scripts and older Helm chart references still show ‘starburst-enterprise-presto’ in artifact names but this is a naming legacy, not a different product.

In this series: ‘Trino’ means the open source engine. ‘SEP’ or ‘Starburst Enterprise’ means the commercial product. ‘Presto’ in quoted training materials means Trino as it was known before December 2020.

Chapter 2: MPP Architecture — How the Engine Actually Works

The most important step in understanding Starburst performance is understanding what kind of system you are actually dealing with. Starburst Enterprise is not a database, not a microservice, and not a batch processing framework. It is a Massively Parallel Processing (MPP) federated SQL query engine and that distinction matters enormously for how you interpret metrics.

2.1 What MPP Actually Means

Massively Parallel Processing means that a single SQL query is simultaneously executed across every available worker node in the cluster. This is the defining characteristic:

This is the single most important fact to internalise from this series. When a junior engineer sees 80% RAM across 25 workers and all CPUs busy, the correct interpretation is: the cluster is working exactly as designed. The questions to ask are not ‘why is utilization high?’ but rather ‘are queries completing successfully?’ and ‘are query latencies within acceptable ranges?’

2.2 Coordinator and Worker Architecture

A Trino/SEP cluster has two node types. The official Starburst documentation describes the coordinator as ‘the brain of a Trino installation’. Here is what each does in practice:

2.3 The Split — Trino’s Unit of Parallelism

The smallest unit of parallel work in Trino is called a split. For the Hive/Iceberg connector reading from object storage, a split typically corresponds to one file or a portion of a large file.

The number of splits available determines the maximum parallelism achievable for a scan:

• 1,000 splits across 25 workers × 32 threads each = all splits can be processed in one wave.

• 10 splits across 25 workers = most workers sit idle during the scan stage regardless of cluster size.

• More splits is better up to the limit of available worker threads after that you get scheduling overhead without benefit

The implication for file management: Files smaller than roughly 8MB create very small splits with high scheduling overhead. Well-sized files (128MB to 1GB) produce appropriately-sized splits that keep all worker threads busy. This is why file compaction is one of the highest-leverage optimisations in Part 3 of this series.

2.4 Query Lifecycle — SQL to Results

Understanding the full execution path helps identify where slowdowns occur. A query travels through these stages:

1. Client submits SQL via HTTP POST to the Coordinator’s REST endpoint.

2. Coordinator parses the SQL and resolves table/column names against the Hive Metastore or Iceberg catalog this is where HMS slowness manifests.

3. The Cost-Based Optimizer (CBO) reads table statistics and builds a logical query plan, choosing join order, join distribution type, and aggregation strategy.

4. The physical planner converts the logical plan into a distributed execution plan composed of stages each stage is a set of tasks that can run on multiple workers in parallel.

5. The scheduler assigns available splits to workers and begins stage execution. Multiple stages can pipeline worker doesn’t wait for stage N to fully complete before starting stage N+1

6. Workers exchange intermediate data with each other via HTTP shuffle. This is the ‘network shuffle’ step that redistributes rows for GROUP BY keys and JOIN keys across workers.

7. The final output stage streams results back to the Coordinator, which returns them to the client.

Trino does NOT write intermediate results to disk between stages. All data flows through memory. This is what makes it fast and why memory management is so consequential.

2.5 The Shared Hash Table — Why More CPU Helps Memory-Heavy Queries

During a JOIN operation, Trino builds a hash table from the build-side table (typically the smaller one) and streams the probe-side table through it looking up each row. This hash table is built once on each worker and shared across all probe threads on that worker it is not duplicated per thread.

The implication: adding more CPU cores to a JOIN-heavy query allows the probe phase to run faster, completing the query sooner and freeing the hash table memory. This is why high CPU utilization during JOIN operations is a sign the cluster is working efficiently, not a sign of resource waste.

Chapter 3: Cluster Sizing — The Right Foundation

Before any tuning is worth doing, the cluster needs to be correctly sized. Tuning a cluster that lacks sufficient headroom just moves the bottleneck around without solving it. The guidance from the 2020 Presto Training Series (Dain Sundstrom, Starburst) and production experience both point to the same starting strategy: start larger than you think you need, verify stability, then rightsize.

3.1 General Sizing Strategy

8. Build a cluster that has more capacity than your initial workload estimate. You need headroom to differentiate configuration problems from resource problems.

9. Verify stability first make sure queries complete without OOM errors and cluster behaviour is predictable.

10. Do NOT try to stabilize and performance-tune simultaneously they require different diagnosis approaches.

11. Measure the actual workload over time: peak memory per query, concurrent query count, CPU utilisation patterns.

12. Then rightsize downward reduce nodes or machine size until utilisation targets are met.

The most common failure pattern I’ve seen in Starburst deployments: teams provision a small cluster and immediately try to optimize it. Without headroom, you cannot tell whether slow queries are caused by configuration problems or by the cluster simply being too small. Always start bigger than you need.

3.2 CPU vs. Memory — Two Independent Dimensions

Trino has two independently scaling resource dimensions that behave very differently and require different responses when they become bottlenecks:

CPU — Controls Query Speed

• More CPU cores → queries run faster. The relationship is approximately linear: doubling CPU halves query execution time.

• CPU does NOT determine whether a query can run , only how fast it completes.

• Running two identical queries concurrently with the same total CPU means each takes roughly 2x longer.

• If queries are slow but completing without errors: the first thing to consider is whether you are CPU-constrained.

Memory — Controls Query Viability and Concurrency

• Memory determines whether a query CAN run, not how fast it runs.

• If a query requires 50GB of distributed hash table memory and your cluster only has 40GB available, the query fails, there is no ‘slow but succeeds’ for memory.

• Memory is consumed by: JOIN hash tables, GROUP BY aggregation accumulators, ORDER BY sort buffers, window function frame data.

• More memory increases the number of concurrent queries you can run and enables larger individual queries.

• If queries are failing with EXCEEDED_LOCAL_MEMORY_LIMIT or EXCEEDED_GLOBAL_MEMORY_LIMIT errors: this is a memory problem, not a CPU problem

3.3 Fewer Bigger Machines vs. Many Smaller Machines

This is one of the most consequential sizing decisions. The guidance from both the Presto Training Series and production experience is clear: fewer, larger machines outperform many smaller ones for Trino workloads.

3.4 Sizing for Concurrent Workloads

The hardest sizing question is not ‘how much does one query need?’ but ‘what is the peak concurrent demand?’. Some things I have learned:

• Memory concurrency is additive: if query A needs 30GB cluster-wide and query B simultaneously needs 40GB, your cluster needs at least 70GB of query-accessible memory to run both at the same time.

• CPU concurrency is time-sliced: the scheduler shares CPU time across concurrent queries. Adding queries increases latency for all queries rather than causing outright failures.

• Peak concurrency, not average, determines minimum cluster size, size for your realistic P95 concurrent query memory demand.

• Bigger clusters get a mix of query sizes that statistically even out. Small clusters are more vulnerable to a single heavy query consuming most available resources and causing other queries to queue.

3.5 Plan for Growth

Every Starburst deployment I have been involved with saw demand grow faster than expected. The platform is easy to use for SQL-familiar users, fast results encourage more queries, and federated access enables use cases teams never previously attempted. It is not unusual for query volume to double or even triple year over year once the platform is well-adopted. Build this into your sizing assumptions and your budget.

Chapter 4: Instance Type Selection — All Three Clouds

Starburst Enterprise is cloud-agnostic and fully supported on AWS, Azure, and GCP. You can deploy via native Kubernetes (EKS, AKS, GKE), directly on virtual machines using Starburst Admin, or via the respective cloud marketplaces. The right instance type for Trino/SEP workloads shares the same characteristics regardless of cloud: high memory-to-core ratio, high network bandwidth, and modern CPU platforms.

The guidance below is based on my experience selecting instance types for SEP workloads and on published instance specifications from each cloud provider. As cloud instance pricing and availability change frequently, always verify current pricing and availability in your region before committing.

4.1 General Principles Across All Clouds

• Memory-optimized families are the first choice for JOIN-heavy and aggregation-heavy analytical workloads they give the highest memory per vCPU.

• General-purpose balanced families work well when workloads are more scan-heavy or when memory per query is moderate.

• Compute-optimized families are appropriate when workloads are CPU-bound with relatively small hash table sizes.

• Network bandwidth matters: Starburst recommends a minimum 10 Gbps within the cluster; 25 Gbps is recommended for object storage access.

• ARM/Ampere-based instances (Graviton on AWS, Ampere Altra on GCP and Azure) offer meaningful cost savings for the same workload — worth evaluating for production after benchmarking

4.2 Amazon Web Services (AWS)

The Presto Training Series (Starburst, 2020) originally provided AWS-centric guidance because that was the dominant cloud for Starburst workloads at the time. The recommendations remain relevant, though newer instance generations (m6i, r6i, m7i, r7i) offer better price-performance than the m5/r5 series shown in the original 2020 training.

AWS note on vCPU vs physical cores: AWS instances use hyperthreading (2 threads per core). A 64-vCPU instance has 32 physical cores. Trino’s task.concurrency should be set to roughly 2x physical cores so 64 for a 32-core machine. For Graviton instances, each vCPU IS a full physical core, so task.concurrency can be set to the vCPU count directly.

4.3 Microsoft Azure

SEP is supported on Azure Kubernetes Service (AKS) and directly on Azure Compute Engine VMs via Starburst Admin. Azure’s instance families that match the workload profile for Trino are primarily in the E-series (memory-optimised) and D-series (general-purpose balanced).

Azure-specific note: Azure’s E-series Esv5 instances (without ‘d’ in the name) do not have local temporary storage. The Edsv5 variants include local NVMe SSD. For SEP deployments where you might enable spilling (which I recommend against — see Part 2), you need the ‘d’ variants with local storage. For standard SEP use, either works, though local storage is useful for HMS and other auxiliary services.

Azure-specific considerations for SEP deployments:

• Azure Data Lake Storage Gen2 (ADLS Gen2): SEP supports ADLS Gen2 natively as object storage via the Hive and Iceberg connectors. Use fs.native-azure.enabled=true in catalog configuration for the current native file system implementation.

• Azure Kubernetes Service (AKS): The primary deployment mechanism for SEP on Azure. Use node pools with the instance types above dedicated to SEP coordinator and workers.

• AKS Accelerated Networking: Ensure Accelerated Networking is enabled on all SEP node pool VMs this is critical for achieving the advertised network bandwidth figures

• Azure Marketplace: SEP is available directly from the Azure Marketplace with PAYG licensing

4.4 Google Cloud Platform (GCP)

SEP is supported on Google Kubernetes Engine (GKE) and Google Compute Engine VMs. The official Starburst documentation lists GKE as the primary and fully certified deployment method on GCP. GCP’s machine naming convention is different from AWS and Azure: machines are described as ‘machine series’ (e.g., n2) + ‘machine type suffix’ (e.g., standard, highmem, highcpu).

GCP-specific considerations for SEP deployments:

• Google Cloud Storage (GCS): SEP reads from GCS via the native file system implementation. Use fs.native-gcs.enabled=true in your catalog configuration. The legacy GCS support is deprecated and will be removed in a future SEP version.

• GKE Standard clusters: Recommended over Autopilot for SEP deployments that need precise control over node pool configuration and affinity rules.

• Dataproc Metastore: Can serve as the Hive Metastore Service (HMS) for SEP catalogs on GCP — eliminates the operational overhead of managing your own HMS.

• Sustained Use Discounts: GCP automatically applies sustained use discounts for N2 and N2D instances that run for most of the month — no upfront commitment required.

• Committed Use Discounts (CUDs): Available for N2 and N2D — up to 55% savings for 3-year commitments

4.5 Multi-Cloud Comparison — Equivalent Instance Families

For teams operating across multiple clouds or evaluating which cloud to use for a new SEP deployment, this table maps roughly equivalent instance families across all three:

4.6 Kubernetes vs. VM Deployments

SEP supports both Kubernetes (K8s) and VM-based deployments. From a sizing perspective:

Chapter 5: What High Utilisation Actually Means

Returning to where we started: the junior engineer’s observation of 80% RAM and fully utilised worker nodes. With the architectural understanding from this document, let me now give a proper answer.

5.1 The Correct Diagnostic Framework

5.2 Metrics That Actually Matter

Rather than watching raw CPU or RAM percentages, monitor these metrics for a true picture of cluster health:

• Query success rate: What percentage of submitted queries complete successfully? OOM errors show up here.

• Query wait time (queue time): How long does a query spend waiting to be scheduled before execution begins? High queue time means the cluster is at capacity for concurrent queries.

• Query execution time percentiles (P50, P95, P99): Are these stable or trending upward over time?

• EXCEEDED_LOCAL_MEMORY_LIMIT error count: Queries failing with this error indicate memory pressure.

• GC time ratio (via JMX): If GC time exceeds 10% of wall clock time on workers, memory pressure is affecting JVM performance.

References

• Presto Training Series, Session 4: Cluster Sizing & Performance Tuning

• Trino: The Definitive Guide, 2nd Edition — Matt Fuller, Manfred Moser, Martin Traverso. O’Reilly Media, October 2022.

• Starburst Enterprise Documentation, SEP 477-e LTS (November 2025)

• Starburst Enterprise on Azure

• Starburst Enterprise on Google Cloud

• Starburst Kubernetes Configuration Examples

• Microsoft Azure: E Family VM Size Series

• Google Cloud: General-Purpose Machine Family

• Announcing Trino (PrestoSQL rename)

• Starburst: PrestoSQL becomes Trino

Series: Starburst Enterprise Performance Tuning | Part 1 of 3 | March 2026 | Validated against SEP 477-e LTS

will Continue with Part 2: Configuration Deep Dive : Memory, JVM & Concurrency

Sometimes it comes from a teammate who has been staring at a SQL statement for an hour. Sometimes it comes from a stakeholder who just wants to know why their dashboard is taking so long to load and occasionally it comes from a senior engineer who knows something is wrong but cannot pinpoint where.

My answer is almost always the same: let us look at the plan.

Reading query plans has become the most reliable diagnostic habit I have built working on this platform. Not because it solves every problem on the first read, but because it consistently points me in the right direction faster than anything else. This post walks through how to use Starburst’s EXPLAIN tooling and how to interpret what you find. Every example uses real queries run against the TPC-DS benchmark dataset on Starburst Galaxy at scale factor 1 (tpcds.sf1). The plan outputs shown are taken directly from those runs, with key lines highlighted.

Whether you are an analyst trying to understand why a query is crawling, an engineer chasing a cluster error, or a stakeholder wanting a clearer picture of what this platform actually does when you hit run, this is written for you.

How Starburst processes a query

Before EXPLAIN makes sense, it helps to understand the two-tier model Starburst uses. There is a single coordinator node responsible for receiving your SQL, parsing it, building a query plan, and scheduling work. Then there are one or more worker nodes that do the actual work of retrieving and processing data in parallel.

When the coordinator receives your query, it builds a distributed query plan: a structured description of how work will be divided into stages and tasks across workers. The EXPLAIN command lets you see that plan before any data is touched. It is a way of asking the coordinator to show its thinking before committing to execution.

That pre-execution window is the point. You can read the plan, spot problems, and rethink the query before a single byte of data is processed.

How Starburst’s EXPLAIN compares to other tools

Query plan inspection exists in most modern query engines, and it is worth being precise about what is different here.

Snowflake has an EXPLAIN command, but it returns a logical plan only. Snowflake’s own documentation states the actual physical execution order may differ, and there are no tuning knobs to act on the output. The real distributed picture with actual row counts and timing is only available post-execution via the Query Profile UI.

Redshift’s EXPLAIN is explicitly limited. Amazon’s documentation states it does not illustrate the details of parallel query processing. You have to run the query first and query system tables after.

BigQuery has no SQL-level EXPLAIN command at all. Plan information is post-execution only, via the console or API.

Starburst gives you the full physical fragment graph before execution as a standard SQL command, with fragment types, data distribution strategy, cost estimates, and pushdown confirmation all visible. Because Starburst queries across multiple data sources simultaneously, the plan shows the full execution graph across all connectors in a single view. And the plan tells you with certainty whether optimizations like predicate pushdown actually fired. With that context, here is how to use it.

1. The command to start with

EXPLAIN (TYPE LOGICAL) also exists but is confirmed deprecated in current Starburst releases. The recommended command is EXPLAIN (TYPE DISTRIBUTED), which shows the full physical execution graph. It does not run the query.

Important: EXPLAIN (TYPE LOGICAL) is officially deprecated in current Starburst releases and will be removed in a future version. If you have existing scripts or workflows using TYPE LOGICAL, migrate them to EXPLAIN (TYPE DISTRIBUTED). All examples in this post use the distributed type, which is the recommended command going forward.

All examples below use tpcds.sf1 on Starburst Galaxy. The TPC-DS table sizes at this scale factor are: store_sales 2.88 million rows, date_dim 73,049 rows, item 18,000 rows, store 12 rows, customer 100,000 rows. These sizes drive the optimizer’s decisions about how to distribute joins.

Example 1: Simple scan

Query:

EXPLAIN (TYPE DISTRIBUTED)
SELECT * FROM tpcds.sf1.store_sales LIMIT 10;

Key plan output:

Fragment 0 [SINGLE]
Output partitioning: SINGLE []
Limit[count = 10]
└─ LocalExchange[partitioning = SINGLE]
└─ RemoteSource[sourceFragmentIds = [1]]
Fragment 1 [SOURCE]
Output partitioning: SINGLE []
LimitPartial[count = 10]
└─ TableScan[table = tpcds:store_sales:sf1.0]
Estimates: {rows: 2880404 (537.32MB), cpu: 537.32M, memory: 0B, network: 0B}

Even for LIMIT 10, the TableScan estimates reading all 2.88 million rows (537MB) from the connector. The LimitPartial in Fragment 1 stops sending rows once 10 are found, but the scan starts against the full table. This is why SELECT * with a LIMIT is not as cheap as it looks.

Read the plan from bottom to top. Fragment 1 executes first, reads from the connector, and feeds rows upward to Fragment 0 via the RemoteSource. Fragment 0 applies the final limit and returns results to the client.

Example 2: Join with a small dimension table and see BROADCAST in action

Query:

EXPLAIN (TYPE DISTRIBUTED)
SELECT
  ss.ss_store_sk, s.s_store_name, COUNT(*) AS transaction_count
FROM tpcds.sf1.store_sales ss
JOIN tpcds.sf1.store s ON ss.ss_store_sk = s.s_store_sk
GROUP BY ss.ss_store_sk, s.s_store_name;

Key plan output:

Fragment 0 [HASH]
    Output partitioning: SINGLE []
    Aggregate[type = FINAL, keys = [ss_store_sk, s_store_name]]
Fragment 1 [SOURCE]
    Output partitioning: HASH [ss_store_sk, s_store_name]
    InnerJoin[criteria = (ss_store_sk = s_store_sk), distribution = REPLICATED]
    ├─ ScanFilter[table = tpcds:store_sales:sf1.0]
    └─ RemoteSource[sourceFragmentIds = [2]]
Fragment 2 [SOURCE]
    Output partitioning: BROADCAST []
    TableScan[table = tpcds:store:sf1.0]
        Estimates: {rows: 12 (201B), cpu: 201, memory: 0B, network: 0B}

Fragment 2 uses BROADCAST partitioning because the store table has only 12 rows at 201 bytes. Every worker node receives a complete copy of it, and the join runs locally on each worker without any network shuffle. A BROADCAST on a table with millions of rows would be a serious warning sign.

Example 3: Three-table join with all fragment types together

Query:

EXPLAIN (TYPE DISTRIBUTED)
SELECT i.i_category, d.d_year, SUM(ss.ss_net_paid) AS total_revenue
FROM tpcds.sf1.store_sales ss
JOIN tpcds.sf1.item i ON ss.ss_item_sk = i.i_item_sk
JOIN tpcds.sf1.date_dim d ON ss.ss_sold_date_sk = d.d_date_sk
WHERE d.d_year = 2001
GROUP BY i.i_category, d.d_year
ORDER BY total_revenue DESC;

Key plan output (6 fragments):

Fragment 0 [SINGLE]      ← final output
Fragment 1 [ROUND_ROBIN] ← distributed sort
Fragment 2 [HASH]        ← final aggregation
Fragment 3 [SOURCE]      ← store_sales scan + both joins
    dynamicFilterAssignments = {i_item_sk -> #df_633, d_date_sk -> #df_634}
Fragment 4 [SOURCE]  Output partitioning: BROADCAST []
    ScanFilter[date_dim]  filterPredicate = (d_year = 2001)
    Estimates: {rows: 73049}/{rows: 363}   ← 73K rows read, 363 survive
Fragment 5 [SOURCE]  Output partitioning: BROADCAST []
    TableScan[item]  Estimates: {rows: 18000 (246.15kB)}

Fragments 4 and 5 are both BROADCAST SOURCE fragments. date_dim is filtered to 363 rows (from 73,049) before being broadcast. item has 18,000 rows and is broadcast in full. The optimizer also assigns dynamic filters from these dimension scans to the store_sales scan in Fragment 3, further narrowing the rows processed before the joins.

2. Layout, Estimates, and what question marks mean

Inside each fragment, every operator shows two sub-sections.

Layout describes the data shape at that point: column names and types. In the store_sales scan from Example 1, the Layout lists all 23 columns. A SELECT * carries all of them through every exchange, making downstream operations more expensive than necessary.

Estimates show the optimizer’s predicted resource consumption. When table statistics exist, they are reasonably accurate:

Estimates: {rows: 2880404 (537.32MB), cpu: 537.32M, memory: 0B, network: 0B}

When statistics are missing or the optimizer cannot derive costs from available information, you see question marks:

Estimates: {rows: 50 (1.79kB), cpu: ?, memory: ?, network: ?}

Example 4: Question marks and ScanFilterProject

Query:

EXPLAIN (TYPE DISTRIBUTED)
SELECT c.c_customer_sk, c.c_first_name, c.c_last_name,
       SUM(ss.ss_net_paid) AS lifetime_value
FROM tpcds.sf1.store_sales ss
JOIN tpcds.sf1.customer c ON ss.ss_customer_sk = c.c_customer_sk
JOIN tpcds.sf1.date_dim d ON ss.ss_sold_date_sk = d.d_date_sk
WHERE d.d_year BETWEEN 2000 AND 2002
  AND c.c_birth_country = ‘UNITED STATES’
GROUP BY c.c_customer_sk, c.c_first_name, c.c_last_name
ORDER BY lifetime_value DESC LIMIT 50;

Key plan output:

Fragment 1 [HASH]
    TopNPartial[count = 50]
    │   Estimates: {rows: 50 (1.79kB), cpu: ?, memory: ?, network: ?}
Fragment 3 [SOURCE]  Output partitioning: BROADCAST []
    ScanFilterProject[table = tpcds:customer:sf1.0,
        filterPredicate = (c_birth_country = varchar(20) ‘UNITED STATES’)]
        Estimates: {rows: 100000 (1.87MB)}/{rows: 458 (8.75kB)}
Fragment 4 [SOURCE]  Output partitioning: BROADCAST []
    ScanFilterProject[table = tpcds:date_dim:sf1.0,
        filterPredicate = (d_year BETWEEN 2000 AND 2002)]
        Estimates: {rows: 73049}/{rows: 730}

Two things to notice. First, the TopN operator in Fragment 1 shows cpu: ? and memory: ? because the optimizer cannot predict the cost of ranking without reliable row estimates from the join below it. Second, ScanFilterProject on customer reads 100,000 rows from the connector and 99.5% are discarded by the filter in Trino. If the customer connector supported predicate pushdown for this filter, only 458 rows would be read. The presence of ScanFilterProject rather than ScanFilter is how you confirm pushdown did not fire.

3. EXPLAIN ANALYZE: what actually happened

Once a query has run, EXPLAIN ANALYZE replaces estimates with real measured values: actual CPU time, scheduled time, blocked time, and row counts per fragment. It also reports standard deviation per task, which is the signal for data skew.

Example 5: Finding where time went

Query:

EXPLAIN ANALYZE
SELECT i.i_category, SUM(ss.ss_net_paid) AS revenue
FROM tpcds.sf1.store_sales ss
JOIN tpcds.sf1.item i ON ss.ss_item_sk = i.i_item_sk
GROUP BY i.i_category ORDER BY revenue DESC;

Key plan output (Fragment 4 — where the work happened):

Fragment 4 [SOURCE]
    CPU: 21.59s, Scheduled: 24.07s, Input: 2898404 rows (49.79MB)
    InnerJoin[criteria = (ss_item_sk = i_item_sk), distribution = REPLICATED]
        Left (probe) Input avg.: 66799.50 rows, Input std.dev.: 100.00%
        Right (build) Input avg.: 18000.00 rows, Input std.dev.: 0.00%
    └─ ScanFilter[table = tpcds:store_sales:sf1.0,
           dynamicFilters = {ss_item_sk = #df_452 await}]
           CPU: 20.70s (90.15% of total query CPU)
           Input avg.: 1440202.00 rows, Input std.dev.: 100.00%
           Dynamic filters:
               df_452, [18000 item key ranges], collection time=3.62s

Fragment 4 consumed over 90% of total CPU time. The std.dev.: 100.00% on the probe side of the join means one worker task processed all the rows while the other processed none, that is data skew. The dynamic filter collected 18,000 valid item key ranges from the item scan and applied them to the store_sales scan before rows reached the join, reducing unnecessary work. Without this filter, all 2.88 million store_sales rows would have been evaluated against the join condition.

4. Operators to watch in complex plans

When a query joins multiple fact tables using CTEs, the plan grows significantly. The following query compares revenue across store, catalog, and web sales channels for 2001:

Example 6: Multi-CTE cross-channel query

Query:

EXPLAIN (TYPE DISTRIBUTED)
WITH store_channel AS (
  SELECT ss.ss_store_sk, ss.ss_item_sk, ss.ss_customer_sk,
         SUM(ss.ss_net_paid) AS store_revenue
  FROM tpcds.sf1.store_sales ss
  JOIN tpcds.sf1.date_dim d ON ss.ss_sold_date_sk = d.d_date_sk
  WHERE d.d_year = 2001
  GROUP BY ss.ss_store_sk, ss.ss_item_sk, ss.ss_customer_sk
),

catalog_channel AS (
  SELECT cs.cs_item_sk, cs.cs_bill_customer_sk,
         SUM(cs.cs_net_paid) AS catalog_revenue
  FROM tpcds.sf1.catalog_sales cs
  JOIN tpcds.sf1.date_dim d ON cs.cs_sold_date_sk = d.d_date_sk
  WHERE d.d_year = 2001
  GROUP BY cs.cs_item_sk, cs.cs_bill_customer_sk
)

SELECT i.i_item_sk, i.i_item_desc, i.i_category,
       sc.store_revenue, cc.catalog_revenue
FROM tpcds.sf1.item i
LEFT JOIN store_channel sc ON i.i_item_sk = sc.ss_item_sk
LEFT JOIN catalog_channel cc ON i.i_item_sk = cc.cs_item_sk
ORDER BY (COALESCE(sc.store_revenue,0) + COALESCE(cc.catalog_revenue,0)) DESC
LIMIT 100;

Key plan output (11 fragments total):

Fragment 1 [SOURCE]
    LeftJoin[criteria = (i_item_sk = ws_item_sk), distribution = REPLICATED]
        Estimates: {rows: 77018361 (11.65GB), cpu: 11.65G, memory: 3.56MB}
Fragment 4 [SOURCE]  Output partitioning: BROADCAST []
    ScanFilterProject[table = tpcds:date_dim:sf1.0,
        filterPredicate = (d_year = integer ‘2001’)]
        Estimates: {rows: 73049}/{rows: 363}  ← 73K in, 363 survive
Fragment 7 [SOURCE]  Output partitioning: BROADCAST []   ← catalog_sales date_dim
    ScanFilterProject  filterPredicate = (d_year = 2001)
        Estimates: {rows: 73049}/{rows: 363}  ← same pattern
Fragment 10 [SOURCE] Output partitioning: BROADCAST []   ← web_sales date_dim
    ScanFilterProject  filterPredicate = (d_year = 2001)
        Estimates: {rows: 73049}/{rows: 363}  ← same pattern, third time

Three things to notice. First, the intermediate LeftJoin in Fragment 1 has an estimated size of 11.65GB before the TopN filter reduces this to 100 rows. This is an expensive intermediate result. Second, ScanFilterProject appears three times , once per sales channel CTE each scanning all 73,049 date_dim rows and discarding 99.5% of them. Third, the pattern repeating three times is a direct consequence of the CTE structure: the optimizer unrolls each CTE independently rather than scanning date_dim once and reusing the result.

The operators worth stopping on in any complex plan:

• Exchange data moving between workers. Each one carries a network cost. Unnecessary exchanges mean data cannot be co-located for a join.

• ScanFilterProject filter evaluated in Trino after reading from the connector rather than pushed to the source. The two-stage estimates ({rows in}/{rows out}) tell you exactly how many rows were wasted.

• Join distribution = REPLICATED confirms a BROADCAST join. Right on small tables, wrong on large ones.

• Dynamic filter assignments the optimizer’s plan to pre-filter a scan using runtime values from a join. Confirm it fired using EXPLAIN ANALYZE.

5. When a query fails before it reaches the workers

At some point you will likely receive this error before any data is processed:

Number of stages in the query (126) exceeds the allowed maximum (100)

This is the QUERY_HAS_TOO_MANY_STAGES error. The coordinator built the distributed plan, counted the stages, and refused to proceed. The query never reached the workers.

The multi-CTE query in Example 6 already produces 11 fragments from two sales channels and one shared date dimension. Extending it to include web_sales as a third channel (as shown in the full version with fragments 0 through 10) adds another four fragments. In production, queries built on views that reference multiple other views can reach the stage limit without anyone realizing how deep the dependency chain runs.

The limit is controlled by query.max-stage-count with a default of 100. The official Starburst documentation notes that setting it too high can cause instability, causing unrelated queries to fail with REMOTE_TASK_ERROR due to request queue pressure. The fix is simplification: intermediate CTEs or temporary tables give the optimizer smaller problems to solve. If that is not possible, the limit can be raised in config.properties or overridden per session using query_max_stage_count.

The stage count in the error is exact, not an estimate. If it says 126, that is precisely how many stages the coordinator determined the query requires.

One important limit of EXPLAIN: it shows the plan the optimizer intends based on statistics and configuration at planning time. It cannot predict runtime behaviour if data distribution is uneven. EXPLAIN ANALYZE is the ground truth. For any query that runs but performs worse than expected, EXPLAIN ANALYZE is the next step after EXPLAIN.

Final thoughts

EXPLAIN is the first command I run when something unexpected happens on the cluster. Not because it answers every question immediately, but because it answers the most important one first: what is the engine actually planning to do?

The practical workflow: run EXPLAIN (TYPE DISTRIBUTED) before submitting any uncertain query. Read for unexpected fragment types, unnecessary exchanges, and question marks in cost estimates. Check for ScanFilterProject and use the two-stage row estimates to quantify what is being wasted. Follow up with EXPLAIN ANALYZE for real CPU time, row counts, and the standard deviation figure that signals skew. If a query fails at planning with a stage-count error, simplify the SQL before raising the configuration limit.

For stakeholders: Starburst gives you a level of pre-execution transparency that is genuinely distinctive in the context of federated multi-source queries. You can see the full distributed plan before running anything, confirm whether optimisations fired, and trace exactly which connector handled which part of the execution. That visibility is available as a standard SQL command and it compounds in value the more complex your data landscape becomes.

Suggested Reads & References

To deepen your understanding of query planning, execution, and performance tuning across Trino and modern data platforms, explore the following resources:

Starburst & Trino

• EXPLAIN documentation

• EXPLAIN ANALYZE documentation

• Query management properties (query.max-stage-count)

• Concepts: stages, tasks, and splits

• Query plan analysis

• Pushdown optimizations (predicate, aggregation, Top-N)

• Architecture overview: Coordinator, Workers, and Cost-Based Optimizer (CBO)

Other Data Platforms (for Comparison)

• Snowflake – EXPLAIN documentation

• Amazon Redshift – EXPLAIN documentation

• Google BigQuery – Query plan and execution timeline

Why These Matter

These resources will help you:

• Understand how query planners work across engines

• Compare execution strategies between distributed systems

• Learn optimization techniques like pushdown and cost-based planning

• Troubleshoot performance issues more effectively

What I found underneath that requirement was a piece of mathematics that shows up in almost every distributed system I’ve worked with since: quorum. The idea that a cluster of nodes needs to reach a majority agreement before it can make a decision, and that this majority requirement makes odd-numbered clusters fundamentally more efficient than even-numbered ones. Once I understood that, a lot of architectural decisions that had previously felt like conventions started to feel like consequences, things that had to be true given how consensus works rather than rules someone had decided on.

This post traces that understanding from Tableau Server in 2012 through to how I think about Kubernetes replica counts today. The tools have changed significantly over the past decade, but the underlying reasoning has not moved at all, and I find that worth writing about.

Where it started: Tableau Server and the three-node minimum

Tableau Server’s high availability requirement of a minimum three nodes has been part of its architecture for a long time. The documentation frames it clearly: with fewer than three nodes, you cannot eliminate single points of failure. With two nodes, at least one critical component (the repository, the file store, or the service responsible for cluster coordination) has only a single instance running. A single instance is a single point of failure, regardless of how many other components are redundant.

What made this click for me was spending time with what Tableau actually needed at the coordination layer. The service responsible for maintaining cluster state, tracking which nodes were available, and making sure the cluster could reach agreement during failures required an odd number of instances to function correctly. You could run one instance, three instances, or five instances. Running two was not a supported configuration, and understanding why is what opened up the quorum concept for me.

At the time of my 2012 deployments, Tableau’s internal coordination mechanism worked on the same majority-agreement principle, even if the specific ZooKeeper-based Coordination Service component that Tableau documents today was formalized in later versions of the product. The three-node minimum and the odd-instance requirement were present and enforced from those early versions. The underlying reason, that a coordination service needs to be able to reach a majority decision and that two instances cannot break a tie, was the same then as it is now.

Tableau Server documentation: “A minimum of three Tableau Server nodes are required for high availability.” The Coordination Service, which is built on Apache ZooKeeper, is deployed in ensembles of one, three, or five instances (never two or four) to guarantee quorum. Source: Tableau Server documentation

It took me a while to fully internalize why the even numbers were excluded. The short answer is that two instances can disagree and have no way to resolve the disagreement. Three instances can disagree, and the third instance breaks the tie. That’s the entire principle, and it scales: five instances can lose two and still form a majority of three. Four instances can lose one, but then you’re down to three, which is the same fault tolerance as a three-node cluster at greater cost. The even numbers are not just suboptimal; they provide no additional safety over the odd number below them.

The mathematics of quorum: why odd numbers are structurally different

To understand why even-numbered clusters are wasteful from a resilience perspective, it helps to work through the quorum formula directly. In any distributed system that requires nodes to agree before making a decision, whether that is electing a leader, committing a write, or changing cluster membership, quorum is the minimum number of nodes that must participate in that agreement for it to be valid.

The formula is simple and worth knowing:

quorum = floor(N / 2) + 1

For a three-node cluster, quorum is 2. For a five-node cluster, quorum is 3. This means a three-node cluster can lose one node and still reach quorum with its remaining two. A five-node cluster can lose two nodes and still reach quorum with its remaining three. Now look at what happens when you apply the same formula to even-numbered clusters:

Cluster  Quorum    Can Fail  Gain vs previous
------   -------   --------  ----------------
2        2         0         —
3        2         1         +1 node, +1 failure tolerance
4        3         1         +1 node, no extra tolerance
5        3         2         +1 node, +1 failure tolerance
6        4         2         +1 node, no extra tolerance

The pattern in that table is the core of the odd-number argument. Every time you move from an odd cluster size to the even number above it, you add a node and gain nothing in fault tolerance. A four-node cluster tolerates exactly one failure, the same as a three-node cluster. A six-node cluster tolerates exactly two failures, the same as a five-node cluster. The etcd documentation makes this explicit, noting that adding a member to bring a cluster to an even number does not buy additional fault tolerance. You have simply paid for a node that makes the quorum requirement stricter without giving you any more room to absorb failures.

etcd FAQ: “An odd-size cluster tolerates the same number of failures as an even-size cluster but with fewer nodes. Adding a member to bring the size of cluster up to an even number doesn’t buy additional fault tolerance.” Source: etcd official documentation

There is a second problem with even-numbered clusters that is worth understanding: split-brain. If a four-node cluster experiences a network partition that isolates two nodes from the other two, neither half has quorum. Both sides see exactly half the cluster, neither can elect a leader or process writes, and the cluster becomes unavailable until the partition heals. With a three-node cluster experiencing the same type of partition, one side has two nodes and the other has one. The side with two nodes has quorum and continues operating normally. The minority side correctly stops accepting writes. The system degrades gracefully rather than becoming completely unavailable.

This is not a theoretical edge case. Network partitions happen in production, and the behaviour of a cluster during a partition is determined by its topology at the time the partition occurs. Odd-numbered clusters are designed to handle this scenario gracefully. Even-numbered clusters are structurally exposed to it.

ZooKeeper, which as we’ve seen underpins Tableau’s Coordination Service, has always recommended clusters of 3 or 5 nodes for exactly these reasons. The Raft consensus algorithm, which etcd uses and which is the foundation of Kubernetes’ control plane, works on the same majority-quorum principle. These are not independent design choices by different teams; they are the same mathematical conclusion, arrived at separately because the mathematics does not leave much room for alternatives.

From coordination services to Kubernetes: the same principle, a different layer

When I started working seriously with Kubernetes, the architecture felt immediately recognisable. Kubernetes uses etcd as its cluster state store, the same consensus-based coordination layer with the same Raft algorithm and the same odd-number requirement. Production Kubernetes control planes run three or five etcd nodes. Running two etcd nodes gives you zero fault tolerance, because any single failure drops you below quorum. Running four gives you the same fault tolerance as three, at higher cost. The recommendation of three or five is not a convention that the Kubernetes community adopted by habit; it is a direct consequence of how Raft works.

But etcd is the infrastructure layer, and most engineers working with Kubernetes day-to-day are not making decisions about the control plane topology. The decisions that come up more frequently are about application workloads: how many replicas should this deployment run? And here the odd-number principle applies, but in a different way that is worth explaining carefully, because the mechanism is not the same as quorum consensus.

Stateless pods do not vote. When a request arrives at a service, any available pod can handle it. There is no leader election, no write quorum, no consensus protocol. The reason three replicas is the right answer for stateless services is not that three pods can form a majority; it is that three pods give you enough redundancy to absorb the operational events that happen routinely on a live cluster without ever dropping to a state where you have no redundancy at all. The connection to the odd-number principle is structural rather than mathematical: in both cases, three is the number where the system can lose one instance and still function safely, and in both cases, two is the number where any single loss leaves you without a safety net.

With consensus systems, that safety net is quorum: the ability to continue making decisions. With stateless pod replicas, the safety net is operational continuity: the ability to continue serving traffic while the cluster does the things it routinely does, draining nodes, rolling out deployments, rescheduling pods. The minimum that delivers that continuity, under realistic operating conditions, is three.

What the numbers look like under real operating conditions

The probability view

Let’s make the availability argument concrete. Assume each pod has a 99% availability rate, which is a reasonable figure for a healthy pod in a well-run cluster that accounts for restarts, rescheduling events, and brief periods of unavailability. With two replicas, you need both pods to be healthy simultaneously to maintain full redundancy. With three replicas, you need any two of the three to be healthy. The difference in degraded exposure across a working year is significant:

The ‘degraded state’ referred above means running on a single pod with no redundancy. The service is technically up, but any restart during that period becomes a user-facing outage. What’s worth noting is that the better your individual pods are, the larger the proportional benefit of the third replica. At 99.9% per-pod availability, the risk gap between two and three replicas is around 600 times. The third replica is not giving you marginally more reliability; it is placing you in a fundamentally different risk category.

The node drain: the scenario that makes this practical

Beyond the probability framing, the operational scenario that most clearly illustrates why two replicas is not enough is the routine node drain. A kernel security patch, a Kubernetes version upgrade, or a cloud provider recycling the underlying hardware all require taking a node offline. The standard mechanism is kubectl drain, which gracefully evicts all pods from the target node before it goes offline.

With two replicas, consider what happens during a drain where one of your pods happens to be on the affected node. That pod is evicted. A replacement is scheduled on another node and begins its startup sequence. Between the moment of eviction and the moment the replacement passes its readiness probe and is marked healthy, your service is running on a single pod. That window can be seconds or it can be several minutes, depending on image pull time, initialization duration, and readiness check configuration. During that window, if the surviving pod restarts for any reason, whether that is an OOM kill, a liveness probe timeout, or a brief spike in a downstream dependency, you have an outage. And this is not an edge case scenario; node drains happen on every production cluster, regularly, as part of normal maintenance.

With three replicas, the same drain plays out with two pods continuing to serve traffic throughout the entire process. The evicted pod’s replacement starts up in the background, and by the time it is healthy, the cluster is back to full redundancy. The on-call engineer does not need to think about whether the drain timing is safe, or whether there might be a risk window. The service is continuously protected throughout the operation.

The official Kubernetes documentation notes that kubectl drain respects Pod Disruption Budgets, but PDBs only protect against voluntary disruptions. If the surviving pod is lost to a hardware failure or an OOM kill while the replacement is starting up, the PDB offers no protection at that point. The replica count itself is what determines how much margin you have.

Rolling deployments and what happens when a new pod gets stuck

Kubernetes’ default rolling update strategy sets both maxUnavailable and maxSurge to 25% of the desired replica count. For small deployments, the rounding behaviour is important to understand: 25% of 2 rounds down to 0, meaning Kubernetes will not terminate any existing pod until its replacement is healthy and ready. That sounds like a sensible safeguard, and in the normal case it is. But consider what happens when the new pod fails its readiness probe.

With two replicas, a new pod that gets stuck in a failing state leaves one old pod carrying all traffic while the deployment is blocked and cannot proceed. The rollout is frozen. To resolve it, someone needs to manually intervene, diagnose the issue, and trigger a rollback. During the time that takes, your service is running without any redundancy, and the deployment pipeline is blocked. With three replicas in the same scenario, two healthy old pods continue serving traffic throughout the investigation. The deployment failure is a problem to be solved, but it is not simultaneously degrading your service. That distinction matters considerably when the failure happens at an inconvenient time.

Translating this into configuration

The principles above translate into a small set of configuration decisions that work together. None of them are complicated in isolation, but they are most valuable when applied as a set rather than individually.

Set a minimum of three replicas

The starting point is simply setting the replica count explicitly at three, rather than leaving it at whatever default was in the deployment template:

spec:
  replicas: 3

Add a Pod Disruption Budget

A PDB tells Kubernetes how many pods your service needs to remain available during voluntary operations such as node drains, cluster upgrades, and maintenance windows. Without a PDB, Kubernetes may evict multiple pods from the same service simultaneously during a drain. With minAvailable set to 2, Kubernetes is instructed to ensure at least two pods are healthy before proceeding with any eviction. It is important to understand that PDBs only cover voluntary disruptions: hardware failures, OOM kills, and pod crashes are involuntary and bypass PDB enforcement entirely. The replica count itself is your protection against those.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: your-service-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: your-service

Configure the rolling update strategy explicitly

Rather than relying on the 25% default behaviour, which rounds unpredictably at small replica counts, it is better to state your intent directly. Setting maxUnavailable to 0 means the full desired number of pods must be available at all times during a rollout. A new pod must pass its readiness probe before any old pod is removed. This eliminates the risk of a stuck deployment silently reducing your available pod count:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

Distribute pods across availability zones

Three replicas running on the same node or the same availability zone give you pod-level redundancy but not infrastructure-level redundancy. A single node failure or a zonal outage can take all three pods down simultaneously. Topology spread constraints tell the Kubernetes scheduler to distribute pods across failure domains, ensuring that infrastructure-level events can only affect a subset of your replicas at once:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: your-service

Apply the same logic to etcd

If you are responsible for the Kubernetes control plane itself, the same odd-number reasoning applies directly to etcd. A production cluster should run three or five etcd nodes. Two etcd nodes provide zero fault tolerance, because any single failure drops below quorum. Four etcd nodes provide the same fault tolerance as three, at greater infrastructure cost. Losing etcd quorum does not degrade your workloads gradually; it makes the entire cluster unable to process any changes until quorum is restored, which makes etcd topology one of the most consequential architectural decisions in a Kubernetes deployment.

When running fewer than three replicas is the right call

Three replicas is the appropriate default for stateless services running in production environments where availability matters. But not every workload and not every environment meets that description, and it is worth being clear about where the tradeoff genuinely falls in the other direction.

Development and staging environments are the most obvious case. If a pod restart causes a brief disruption in a development environment, the cost is a developer noticing a timeout and refreshing their browser. The risk calculus is completely different from production, and running two replicas or even one in those environments is entirely reasonable. Similarly, low-traffic internal tools where the business impact of a brief outage is negligible may not justify the infrastructure cost of a third replica, particularly if the team responsible for them has explicitly weighed that tradeoff.

Async workers and batch processing jobs represent another legitimate exception. If a worker pod restarts mid-job and the job retries from the beginning or resumes from a checkpoint, the impact is a delay rather than a failure. For those workloads, replica count is more about throughput than availability, and the right number depends on the processing requirements rather than the resilience argument made in this post.

The underlying principle is that these should be deliberate decisions made with the tradeoffs in mind, not defaults that were never reconsidered. Two replicas in a staging environment is a sensible choice. Two replicas in a production service because the task to increase it never made it to the top of the backlog is a different situation entirely, and one that tends to surface at the worst possible moments.

The principle outlasts the tools

Looking back across more than a decade of infrastructure work, what strikes me most about the odd-number rule is how stable it has been. The tools I have worked with have changed substantially, from Tableau’s internal coordination mechanism in 2012, through ZooKeeper-based systems in the mid-2010s, to etcd and Kubernetes today, but the mathematical reason for preferring odd cluster sizes has not changed at all. Quorum requires a majority. A majority decision cannot be made by an even number of participants without the risk of a tie. The minimum odd number that provides both redundancy and majority decision-making is three, and that is why three appears as the minimum HA configuration in so many different systems across so many different years.

What I find useful about understanding the origin of the rule rather than just following it is that it tells you when the rule applies and when it does not. For consensus-based systems like etcd or ZooKeeper, the odd-number requirement is mathematically strict: two nodes provides no fault tolerance, and four nodes provides no more fault tolerance than three. For stateless pod replicas, the argument is structurally similar but not mathematically identical. Three replicas is not required for quorum, but it is the minimum that keeps your service in a safe state across the routine operational events that every production cluster experiences regularly.

When you configure replicas: 3 for a Kubernetes deployment, you are not following a convention that someone decided on and everyone else copied. You are making a decision that is consistent with the same reasoning that Tableau encoded into its HA architecture in the early 2010s, that ZooKeeper encoded into its ensemble configuration, and that the Raft algorithm formalizes in its design. The context is different each time, but the underlying logic is the same one, and it has held up across every distributed system I’ve encountered since I first came across it in a Tableau Server deployment guide over a decade ago.

References :

• Tableau Server Coordination Service — quorum, ensemble, three-node HA requirement

• Deploy a Coordination Service Ensemble — odd-only instances (1, 3, or 5)

• Tableau Server high availability installation — minimum three nodes

• etcd FAQ — odd cluster size, even-numbered clusters provide no extra fault tolerance

• Google SRE Book: Managing Critical State — distributed consensus and quorum

• Kubernetes Deployments — rolling update defaults, maxUnavailable and maxSurge (25%)

• Kubernetes Pod Disruptions — voluntary vs involuntary, PDB scope and limits

• Safely drain a Kubernetes node — kubectl drain, PDB enforcement

• Pod Topology Spread Constraints — zone and node distribution

Two fundamental metrics are commonly used to define recovery expectations for systems that must maintain reliability in the presence of failures. These metrics are Recovery Point Objective (RPO) and Recovery Time Objective (RTO). Together they provide a framework for determining how much data loss is acceptable and how quickly a system must recover when disruptions occur.

Understanding these concepts is essential for designing distributed systems that preserve both data durability and service availability.

Recovery Point Objective (RPO)

Recovery Point Objective represents the maximum amount of data that a system can afford to lose after a failure occurs. The objective is typically defined as a time interval that describes how far back the system may need to revert during recovery.

For example, consider a system with an RPO of ten minutes. In this situation, if a failure occurs, the organization accepts that up to ten minutes of recently generated data may be lost during recovery. This lost information could include user transactions, database updates, application events, or other forms of system state that were generated after the last successful recovery point.

RPO Timeline Illustration

Time  ───────────────────────────────────────────────>

Last Backup          Failure Occurs
     │                     │
     │<------ RPO -------> │
     │                     │
Data that may potentially be lost

RPO is closely related to the mechanisms used for data replication and backup. Systems with larger RPO values often rely on periodic backups or asynchronous replication strategies. In contrast, systems that require very low RPO values must replicate data more frequently or maintain near real-time copies across multiple storage nodes.

Zero RPO Architecture

          Client
             │
             ▼
       Primary Database
             │
    Synchronous Replication
             │
     ┌───────────────┐
     ▼               ▼
Replica Database  Replica Database

In this model, transactions are committed only after they are written to multiple replicas, ensuring that data is preserved even if the primary node fails.

Recovery Time Objective (RTO)

Recovery Time Objective defines the maximum acceptable duration that a system may remain unavailable after a failure occurs. While RPO addresses data durability, RTO focuses on service availability.

When an outage occurs, several processes must take place before normal system operation resumes. These processes include detecting the failure, activating backup resources, restarting services, and redirecting traffic toward healthy components.

RTO Recovery Timeline

Time ─────────────────────────────────────────────>

Failure      Detection      Failover      Recovery
  │              │             │             │
  │<------------------ RTO ----------------->│

If a system has an RTO of five minutes, the entire recovery process—from failure detection to service restoration—must complete within that five-minute window.

Achieving low RTO values requires automated monitoring, load balancing, and redundant infrastructure.

Relationship Between RPO and RTO

Although RPO and RTO are closely related, they address different dimensions of system reliability.

| Metric | Focus           | What It Measures                |
| ------ | --------------- | ------------------------------- |
| RPO    | Data durability | Maximum data loss after failure |
| RTO    | Availability    | Maximum downtime after failure  |

Combined RPO and RTO Illustration

Time ───────────────────────────────────────────────────>

Last Safe Data        Failure            Service Restored
      │                  │                      │
      │<---- RPO ----->  │<-------- RTO -------->│

The section before the failure represents potential data loss, while the section after the failure represents system downtime.

Failure Scenario in a Distributed System

Consider a distributed application consisting of multiple service nodes positioned behind a load balancer.

Normal System Architecture

                 Load Balancer
               /       |        \
             Node1   Node2    Node3
               │       │        │
               └───────┼────────┘
                       │
                Distributed Database

During normal operation:

• incoming requests are distributed across nodes

• each node processes requests independently

• the database replicates data across storage nodes

Node Failure Scenario

Suppose Node2 experiences a hardware failure.

                 Load Balancer
               /       X        \
             Node1   Node2    Node3
                        DOWN
               │                │
               └───────┬────────┘
                       │
                Distributed Database

Monitoring systems detect that Node2 is no longer responding.

Traffic Rerouting After Failure

The load balancer removes the failed node and redirects requests to the remaining nodes.

                 Load Balancer
                 /          \
               Node1      Node3
                 │           │
                 └───────┬───┘
                         │
                Distributed Database

The system continues operating while infrastructure automation replaces the failed node.

Node Recovery and Rejoin

           Replacement Node
                  │
                  ▼
           Data Synchronization
                  │
                  ▼
        Node Rejoins Service Cluster

Once the node synchronizes with the cluster, it resumes handling requests.

Measuring Recovery Objectives in the Scenario

A typical incident timeline might look like this:

00:00  Node failure occurs
00:03  Monitoring detects failure
00:05  Load balancer reroutes traffic
00:30  Replacement node joins cluster

From a user’s perspective:

• RTO ≈ 5 seconds (time to restore service)

• RPO = 0 if synchronous replication preserved all writes

Tiered Recovery Objectives Across Systems

Organizations often assign recovery objectives based on system importance.

| Tier   | Example Systems              | Typical RPO | Typical RTO |
| ------ | ---------------------------- | ----------- | ----------- |
| Tier 1 | Payments, healthcare systems | near-zero   | seconds     |
| Tier 2 | Core business applications   | minutes     | minutes     |
| Tier 3 | Internal services            | hours       | hours       |
| Tier 4 | Analytics and reporting      | hours–days  | hours–days  |

This classification ensures that mission-critical systems receive the highest level of resilience.

Architectural Techniques for Reducing RPO and RTO

Distributed Infrastructure

         Service Cluster
      /        |        \
   Node A    Node B    Node C

Multiple nodes prevent single points of failure.

Multi-Region Deployment

             Global Traffic Router
               /              \
           Region A          Region B
         /    |    \        /    |    \
      Node  Node  Node   Node  Node  Node

If one region fails, traffic can shift to the other region.

Replication Strategies

Synchronous Replication

Client → Primary DB → Replica1 → Replica2

Ensures no data loss but increases write latency.

Asynchronous Replication

Client → Primary DB
              │
              ▼
         Replica DB

Improves performance but introduces a small window of possible data loss.

DevOps Practices That Improve Recovery

Modern engineering teams integrate reliability into deployment processes. Organizations such as Netflix and Stripe rely heavily on automation to maintain operational resilience.

Important practices include:

• Infrastructure as Code for reproducible environments

• automated deployment pipelines for rapid recovery

• immutable infrastructure that replaces failed instances rather than repairing them

These practices help reduce recovery times and improve system stability during failures.

Business Impact of Downtime

System outages have measurable financial and operational consequences. Research from Gartner suggests that the cost of downtime can reach thousands of dollars per minute for many organizations.

Beyond financial losses, outages may lead to:

• failed customer transactions

• inconsistent data records

• loss of customer trust

• operational disruptions for internal teams

For this reason, recovery objectives are an essential part of infrastructure planning.

Practical Limits of Zero Recovery Objectives

Although organizations often aim for zero RPO and zero RTO, achieving absolute zero is extremely difficult due to network latency, failure detection delays, and coordination overhead in distributed systems.

Most production systems instead target near-zero recovery objectives, ensuring that failures cause minimal disruption and that services remain effectively available from a user’s perspective.

Conclusion

Reliable system design requires acknowledging that infrastructure failures are inevitable. Recovery Point Objective and Recovery Time Objective provide a structured framework for defining how systems should respond to these failures.

RPO determines how much data loss is acceptable, while RTO defines how quickly services must recover. By combining distributed architectures, automated failover mechanisms, robust replication strategies, and modern DevOps practices, engineering teams can design systems capable of maintaining reliability even in the presence of infrastructure failures.

I’m currently preparing for executive-level interviews, and reflecting on my leadership experiences has been invaluable. As I attend these interviews, I’ve realized how critical it is to structure your stories, show measurable impact, and communicate your leadership thinking effectively. In this blog, I share lessons from my experience, combined with insights from resources like Austen McDonald’s thebehavioral.tech, Byte-by-Byte, Leetcode, and Gayle Laakmann McDowell.

1. Resolving Cross-Team Conflict at Scale (CARL)

• Context: During a rollout of a new analytics platform, Data Engineering and Product Infrastructure teams disagreed on delivery timelines.

• Action: I facilitated an alignment workshop with team leads and product owners, mapping dependencies and creating a shared delivery plan.

• Result: Both teams aligned and delivered the project on time.

• Learning: Structured dialogue, empathy, and focus on organizational goals are essential at scale.

2. Making High-Stakes Decisions Under Uncertainty (CARL)

• Context: Two hours before a major platform release, a critical issue was discovered.

• Action: I evaluated risks, consulted key leads, and approved a targeted hotfix.

• Result: The release went live successfully without user impact.

• Learning: Calculated risks, rapid evaluation, and clear escalation paths define executive leadership.

3. Delivering Tough Feedback to Leaders (CARL)

• Context: A newly promoted engineering manager was missing key deadlines.

• Action: I shared specific examples, proposed a clear improvement plan, and offered support.

• Result: Six months later, their team’s delivery improved by 40%.

• Learning: Feedback must be clear, empathetic, and measurable.

4. Owning Big Bets or Strategic Projects – Feature Y (CARL)

• Context: At BigCo, the company struggled to close additional clients due to lack of differentiation from competition. I suggested building Feature Y after exploring technical feasibility in a proof-of-concept.

• Action: I worked with design, PM, data science, and a senior engineer to plan and build Feature Y over a quarter.

• Result: The feature brought in 50% more revenue.

• Learning: Proof-of-concepts are extremely helpful for convincing leaders of what’s possible and securing buy-in for strategic initiatives.

5. Learning from Failures (CARL)

• Context: A migration of our legacy data pipeline was delayed due to underestimated complexity.

• Action: I redesigned the timeline with phased delivery and introduced stronger risk assessment protocols.

• Result: Future projects now face 60% fewer delays.

• Learning: Ownership, learning, and process improvement define great leadership.

6. Balancing Vision and Execution (CARL)

• Context: While focusing on our AI platform roadmap, we had to deliver quarterly business-critical enhancements.

• Action: I implemented prioritization frameworks and aligned executives and engineers through clear communication.

• Result: Both long-term and short-term goals were delivered successfully.

• Learning: Balancing trade-offs and keeping teams aligned is key at the executive level.

7. Vendor Negotiation at Scale (CARL)

• Context: During a cloud migration, we needed to negotiate terms with two enterprise vendors.

• Action: I highlighted long-term partnership value, strategic importance, and key project constraints.

• Result: Secured a 20% discount, additional support hours, and guaranteed SLA improvements, while keeping the project on schedule.

• Learning: Vendor negotiation requires data, leverage, and clear communication, not just price haggling.

8. Preparing for Executive-Level Behavioral Interviews

As I prepare for executive-level interviews, I’ve invested significant time studying behavioral interview preparation resources that are especially relevant for leadership roles.

These helped me:

• Structure stories around big bets, cross-team impact, and influence

• Map experiences to commonly assessed signal areas

• Communicate not just outcomes, but leadership thinking

Frameworks like STARR (Situation, Task, Action, Result, Reflection) and CARL (Context, Action, Result, Learning) have been particularly effective in helping me articulate lessons from conflict, risk-taking, and vendor negotiations.

Key takeaway: Behavioral interview preparation is essential for executive roles. It’s about reflection and clarity, not memorization.

9. Key Considerations for Executive Behavioral Interviews

When preparing for executive-level interviews, it’s important to go beyond stories and frameworks. Here are the critical areas I focus on:

• The Importance of Behavioral Interviews: Assess how you think, lead, and influence, not just what you did.

• Evaluating Candidates in Behavioral Interviews: Interviewers look for clarity, consistency, and evidence of leadership impact.

• Understanding Interviewer Signals: Watch verbal and non-verbal cues—they reveal what qualities are valued.

• Presenting Your Experience Effectively: Show context, action, result, and learning. Link technical work to business impact.

• The Importance of Authenticity: Executive interviewers spot rehearsed answers—be genuine while professional.

• Common Candidate Mistakes and Red Flags: Overloading details, avoiding accountability, or failing to show cross-functional influence.

• Making a Strong First Impression: Confidence, clarity, and humility matter.

• Team Match and Asking Questions: Thoughtful questions about culture, strategy, and leadership expectations show alignment and curiosity.

Key takeaway: Understanding these fundamentals ensures you communicate experience effectively, demonstrate leadership qualities, and make a lasting impression.

Key Lessons for Executive Leaders

1. Conflict resolution: Align teams strategically.

2. Decision-making: Take calculated risks under uncertainty.

3. Feedback: Deliver clear, empathetic guidance.

4. Big bets: Lead with vision, execution, and influence.

5. Failure: Own it, learn, and improve processes.

6. Strategy vs execution: Balance long-term goals with short-term needs.

7. Vendor negotiation: Influence partnerships strategically.

8. Behavioral prep: Reflect on experiences, structure them effectively using CARL or STARR, and communicate impact confidently.

Behavioral interviews and leadership challenges go hand in hand. Preparing thoughtfully, reflecting on leadership experience, and framing stories with CARL or STARR ensures that when it comes to big bets, tough decisions, and high-stakes projects, you can communicate both your impact and your thought process effectively.

References:

The following resources played an important role in shaping my understanding of behavioral interviews, executive storytelling, and leadership reflection while preparing for executive-level interviews. I highly recommend them to anyone seriously investing in interview preparation and long-term leadership growth.

• thebehavioral.tech: This is the primary site for Austen McDonald, where he hosts his complete behavioral preparation system and Substack newsletter.

• Byte-by-Byte: Note that Byte-by-Byte was recently acquired by interviewing.io, and much of its content is currently being integrated there.

• LeetCode: The industry-standard platform for practicing technical coding problems and participating in competitive contests.

• Gayle Laakmann McDowell: The author’s personal site. You can also find her interview database and forum at CareerCup and book-specific resources at Cracking the Coding Interview.
  
  

Leadership, in its earliest and most enduring form, predates formal titles, organizational hierarchies, and defined reporting lines. Across time, leadership has emerged not through designation but through action, particularly in situations where uncertainty demanded judgment and responsibility extended beyond defined roles. Groups have consistently gravitated toward individuals who could interpret ambiguity, manage risk, and guide collective direction when outcomes mattered. Authority, as it is commonly understood today, evolved later as a mechanism for structure and scale, but influence has always remained the foundation upon which leadership rests.

Modern engineering organizations often assume that leadership accompanies position or mandate. Agile and platform-driven environments repeatedly expose the limitations of this assumption. As systems become more interconnected and delivery more distributed, authority fragments while accountability persists. This tension is especially visible in platform and infrastructure engineering, where teams operate at the intersection of reliability, security, compliance, and delivery. The consequences of technical decisions often extend far beyond the boundaries of any single team, yet ownership of those decisions is rarely centralized.

In such environments, platform engineers and senior individual contributors are routinely drawn into decisions during incidents, releases, and architectural transitions. They may influence infrastructure design, operational controls, and risk posture without formally owning application architecture, product priorities, or delivery roadmaps. Decisions cannot always wait for clarity of ownership. Systems demand timely judgment, and leadership emerges through those who provide direction when authority is unclear.

Influence in these contexts does not come from control but from the ability to create alignment under constraint. Product teams operate under delivery pressure. Application teams focus on maintainability and velocity. Security and compliance functions prioritize risk mitigation and regulatory adherence. Effective influence requires understanding these differing priorities and framing technical considerations in terms of shared outcomes rather than isolated constraints. When discussions begin with common objectives instead of positional mandates, collaboration becomes more natural and resistance decreases.

Clarity plays an equally critical role. Platform and infrastructure work is inherently complex and often invisible until something fails. During moments of uncertainty, stakeholders are rarely seeking exhaustive technical detail. They are seeking orientation and direction. Leaders operating without formal authority add the most value by clearly articulating the decision at hand, outlining viable options, explaining trade-offs, and making consequences visible. The ability to reduce complexity without oversimplifying reality is a defining characteristic of informal leadership in technical environments.

Credibility remains the enduring currency of influence. It is built gradually through consistency, fairness, and presence, particularly during high-pressure situations such as production incidents or critical releases. Credibility strengthens when individuals demonstrate sound judgment, acknowledge uncertainty when it exists, and share credit across teams. It erodes quickly when ego, defensiveness, or misattribution enters the equation. In environments where authority is ambiguous, trust becomes the mechanism through which influence is sustained.

Leading without owning the decision requires deliberate attention to behaviour and conduct. Psychological safety and clarity form the foundation of such leadership. When individuals feel able to question assumptions, raise concerns, and express dissent without fear of dismissal, decision quality improves and systemic risks surface earlier. This is not merely a cultural aspiration; it is an operational necessity in complex, interdependent systems.

Equally important is the ability to subordinate personal ego to collective outcomes. Informal leadership is not concerned with being seen as correct but with enabling effective decisions. Those who influence most successfully tend to ask questions that sharpen understanding rather than assert conclusions prematurely. They recognize that formal ownership of decisions matters less than shared responsibility for outcomes. When outcomes improve, leadership has already been exercised, regardless of attribution.

Visibility and recognition also play a meaningful role. Platform and infrastructure contributions often operate in the background, and success is frequently measured by the absence of failure rather than visible achievements. Leaders who consciously acknowledge contributions and ensure recognition is distributed fairly strengthen trust across organizational boundaries. Over time, this practice becomes a powerful source of informal authority.

The capacity to influence others is closely tied to personal effectiveness. Sustained leadership without authority requires emotional and cognitive clarity, and these resources are finite. Continuous fatigue, excessive context switching, and reactive working patterns reduce one’s ability to engage constructively with stakeholders. Reflection on personal effectiveness therefore becomes essential rather than optional. The quality of influence one brings to an organization often reflects the quality of attention, composure, and preparation brought to each interaction.

Time offers a revealing signal of priorities. Environments dominated by perpetual firefighting and unstructured decision-making leave little space for thoughtful influence. Protecting time for preparation, reflection, and synthesis enables leaders to frame issues more effectively and engage stakeholders with clarity. In complex systems, disciplined use of time is inseparable from effective leadership practice.

Leadership without a title ultimately begins with self-examination. Before attempting to influence stakeholders or shape decisions, it is worth considering whether one is acting from clarity or anxiety, whether trust is being built or consumed, and whether actions strengthen the system as a whole rather than advance narrow interests. Influence may be external in its expression, but leadership remains deeply personal in its foundation.

The ability to lead without authority is not an innate trait reserved for a few. It is a capability developed through experience, observation, and deliberate practice. Structured learning can help, but the most meaningful growth occurs through consistent application in real organizational contexts.

In contemporary engineering organizations, authority is often distributed, situational, or unclear, while responsibility remains constant. Platform and infrastructure leaders operate within this tension daily. Leadership without a title is not a secondary form of leadership. In complex engineering systems, it is often the primary way leadership is exercised. Authority may be defined by role, but influence is shaped by conduct, consistency, and trust built over time.

For readers interested in exploring this further, the following resources offer useful perspectives on influencing without authority:

• Leadership and Persuasion: Influencing Without Authority

• Influence Without Authority: Practical Strategies

• How to Influence Without Authority at Work

• Leadership and Influence (edX)

• Influencing Without Authority (Codecademy)

Share Data, Chai & Dialogue - The Leadership Lounge

Subscribe now

Without ADRs, teams often repeat debates, lose institutional knowledge, or struggle to understand why a system is designed a certain way. ADRs solve this by making decisions explicit, traceable, and easily accessible.

Why ADRs Matter

ADRs are not just bureaucratic overhead; they are a critical tool for sustainable architecture. They help teams align on choices, track the evolution of systems, and support continuous improvement. For new team members, ADRs serve as a practical learning resource that explains the “why” behind decisions rather than just the “what.”

Consistently maintaining ADRs ensures that architectural knowledge is not lost over time, helps prevent repeated mistakes, and enables confident decision-making in complex systems. For more details, see TechTarget ADR Best Practices.

Good Practices for Creating ADRs

Creating ADRs effectively requires discipline and consistency. Here are some best practices to follow:

1. Use a consistent format and template. Ensure all ADRs follow the same structure so that team members can quickly understand each record.

2. Keep ADRs concise. Focus on the key decision points and avoid unnecessary detail that does not aid understanding.

3. Document decisions promptly. Capture the decision as soon as possible after it is made to preserve context and reasoning.

4. Store ADRs in an accessible repository. Use tools like GitLab or Confluence so all stakeholders can view and reference them easily.

5. Encourage collaboration. Gather feedback from all relevant stakeholders to ensure decisions are well-informed.

6. Track architectural evolution. Use ADRs to maintain a clear history of decisions, showing how architecture has evolved over time.

7. Use ADRs for continuous improvement. Review past decisions to learn lessons and refine architecture practices.

8. Leverage ADRs for onboarding. New team members can quickly understand the system and its design rationale.

9. Review and update ADRs regularly. Ensure they remain relevant as the system and context evolve.

Key Considerations

• ADRs should be maintained for all products and systems, even if the team decides the storage location or process.

• ADRs must be searchable, auditable, and uniquely referencable.

• While teams may choose the tools that fit their workflow, Git repositories or Confluence are commonly used for ADR storage.

• Retention and governance must meet the minimum organizational standards to ensure compliance and long-term usefulness.

Recommended ADR Template

Each ADR should follow a consistent structure, with short, clear titles and logical sections. Titles should use present-tense imperative phrasing, similar to a Git commit message, and less than 50 characters.

Template Structure

Title: Short, action-oriented description of the decision

Example: Use GraphQL for internal APIs

1. Status
Current state of the decision: Proposed, Accepted, Deprecated, Superseded.

2. Context
Describe the problem, issue, or motivation for this decision. Explain why a decision is necessary and what triggered it.

Example: “The current REST APIs are causing repeated versioning conflicts and increased overhead in frontend-backend coordination.”

3. Considered Options
List all the options that were considered to solve the problem.

Example: “1) Continue using REST 2) Switch to GraphQL”

4. Pros and Cons
Provide advantages and disadvantages for each option.

Example:

• REST: Well-known, widely supported; difficult to manage multiple versions

• GraphQL: Flexible, single endpoint, reduces over-fetching; requires learning and standardization

5. Decision
State the chosen option clearly, describing the change or action being implemented.

Example: “Adopt GraphQL for all new internal APIs.”

6. Consequences
Explain what becomes easier or harder due to this decision. Include both intended benefits and potential drawbacks.

Example:

• Frontend developers can fetch exactly the data they need

• Backend teams require training on GraphQL patterns

• Monitoring and caching patterns must be updated

Storage Recommendations

• Store ADRs in a central, version-controlled repository for accessibility and traceability.

• Use Git repositories for code-aligned projects or Confluence for documentation-focused projects.

• ADRs should be searchable and linked to the relevant project or system.

• Ensure ADRs are reviewed and updated periodically.

ADR Tools with Features

A variety of ADR tools exist to help teams create, manage, and visualize architecture decisions. For a full list, see TechTarget ADR Tools.

From personal experience, two tools stand out for practical day-to-day use:

Backstage ADR Plugin

• Description: Integrates ADR management directly into the Backstage developer portal.

• Features: Visual ADR overview, linking decisions to components and services, easy access for developers.

• Use case: Ideal for teams already using Backstage as their internal developer portal, providing a centralized place for architectural decisions.

Visual Studio Code Extensions

• Description: In-editor extensions for creating ADRs directly in VS Code.

• Features: Predefined templates, scaffolding, and linking ADRs to related files or projects.

• Use case: Developers can write ADRs alongside code, enforcing template consistency and saving time.

For other ADR tools such as ADR Manager, dotnet-adr, Log4brains, or PyADR, see TechTarget ADR Tools

Closing Thoughts

ADRs help teams capture decisions early, track the evolution of systems, and onboard new members efficiently. When used consistently, ADRs form the backbone of maintainable and transparent architecture practices.

By combining ADRs with documentation as code, teams can create architecture documentation that is always up to date, accessible, and useful, setting the foundation for confident decision-making and scalable system design.

ADR Markdown Template (Ready to Use)

You can copy and adapt this template for any decision:

# [Short, Imperative Title]  
**Status:** Proposed / Accepted / Deprecated / Superseded  

## Context
Describe the issue, motivation, or problem that triggered this decision.

## Considered Options
1. Option A  
2. Option B  
3. Option C  

## Pros and Cons
**Option A:**  
- Pros: ...  
- Cons: ...  

**Option B:**  
- Pros: ...  
- Cons: ...  

## Decision
State the chosen option and the action being implemented.

## Consequences
Explain benefits, challenges, and trade-offs resulting from this decision.

Next blog posts in the series will cover:

• Series 3: Documentation as Code – How to integrate ADRs and diagrams in Git/Markdown workflows.

• Series 4: AI-assisted System Documentation – Using AI to generate, update, and review architecture documentation automatically.

Gherkin is more than a testing framework. It is a bridge between product thinking and engineering execution. While Behaviour Driven Development (BDD) and Test Driven Development (TDD) have been part of engineering practices for years, surprisingly few Product Owners use Gherkin as a communication tool. That’s a missed opportunity, because when used well, it eliminates ambiguity, shortens feedback loops, and aligns everyone to the same definition of “done.”

Why User Stories Aren’t Always Enough

Most of us are used to writing requirements as User Stories:

“As a user, I want to generate a daily report so I can monitor anomalies.”

Stories are great. They capture the user, the action, and the expected result. They help us focus on value, cover expected interactions, and provide context. But when the system grows complex — multiple services, data flows, or dependent APIs — stories start to fall short. You end up stacking acceptance criteria, notes, exceptions, and examples, bloating Jira tickets with what should have been clear from the start.

The gaps show up as rework, misunderstandings, and frustrated engineers. Gherkin comes to the rescue. By writing behaviour in examples, Product Owners can provide explicit instructions without dictating implementation.

What Gherkin Brings to the Table

Gherkin introduces a simple syntax:

Given — the initial state
When — the action performed
Then — the expected outcome

This seems trivial at first, but it’s transformative. Every scenario becomes a conversation starter. Every ambiguity is exposed before coding begins. For platform and data teams, this means that pipelines, APIs, and workflows can be specified with precision while still keeping the language accessible to non-technical stakeholders.

A Real Platform Engineering Example

Consider a banking data platform where transaction data flows into a fraud detection pipeline. A naive requirement might read:

“The system should process daily transaction files.”

With Gherkin, behaviour is explicit:

Scenario: Successful daily ingestion
Given the daily transaction file arrives before 2 AM
When the ingestion pipeline starts
Then schema validation runs successfully
Then valid records are enriched with customer metadata
Then fraud rules are executed
Then clean data is stored in the warehouse
Then alerts are triggered for suspicious activity

Now imagine an edge case:

Scenario: Corrupted records in transaction file
Given the file contains corrupted rows
When validation runs
Then corrupted records are quarantined
Then processing continues for valid data
Then operations receives an error summary

Every step is explicit. No assumptions. No hidden gaps.

The Cheat Sheet: Given, When, Then

Over time, teams start thinking in this flow naturally:

• Given — current state of the system

• When — the action or event

• Then — what must be true afterward

• And / But — extend previous steps without breaking clarity

This becomes the backbone of all behaviour specification. A cheat sheet that guides design, testing, and automation. Complex flows, like multi-stage pipelines or API orchestrations, can be broken into multiple scenarios while keeping them readable and verifiable.

Complexity, Heuristics, and Controlled Granularity

Not every feature needs dozens of scenarios. But as complexity grows, Gherkin exposes it early. You’ll notice when you keep adding scenarios: maybe your feature is too broad, maybe your user segmentation is unclear, or maybe you can split the feature into smaller behaviours.

The goal isn’t exhaustive detail but enough clarity to communicate intent and allow engineering to implement safely. Think of it as early risk detection before code even hits the repository.

Product Owner Practices for Using Gherkin Effectively

1. Start from the user goal — What outcome matters to them?

2. Define system states and events — Given/When/Then for each behaviour.

3. Collaborate with engineers — Review scenarios together; ensure feasibility and coverage.

4. Keep the language business-friendly — Avoid endpoints, DB columns, or implementation details.

5. Use scenarios to drive tests — Engineers can implement step definitions for automation.

6. Review and refine — Complexity grows; adjust granularity as the system evolves.

Why It Matters

When Product Owners use Gherkin effectively:

• Engineers understand requirements faster and more accurately

• Quality improves naturally, reducing rework

• Releases become predictable

• The team builds trust and shared understanding

In short, it’s not just a syntax. It’s a method for turning abstract objectives into measurable, testable outcomes — the same way OKRs turn strategy into execution for teams.

The Takeaway

A well-crafted Gherkin feature is like a map. It shows exactly where you are, what happens next, and what “success” looks like. It turns ambiguity into shared understanding and accelerates delivery without creating bureaucracy.

For Product Owners in platform and data engineering, this is a superpower: designing behaviours upfront, communicating clearly, and enabling teams to own outcomes confidently.

Over time, I’ve received consistent feedback from mentees, peers, and team members:

• Where are our architecture documents?

• What tools should we use to create diagrams?

• Do we have any templates or standards?

• How do we start learning system architecture properly?

So in today’s post, I want to focus specifically on architecture documentation — why it matters, how to approach it, and what views you should ideally cover.

This blog series is mainly focused on standardizing architecture documentation and sharing best practices to create documentation that is easy to maintain and keep up to date.

My goal is also to help junior engineers develop interest in architecture concepts early in their careers — not just coding, but understanding systems as a whole.

Core Principles of Good Architecture Documentation

Before jumping into tools and diagrams, a few fundamentals matter more than anything else.

1. Identify your primary audience

Your audience could be:

• Business stakeholders

• Developers

• Operations teams

• Security and compliance

• Leadership

Each group needs different levels of detail. One document cannot serve everyone perfectly.

2. Create only what your audience needs

A common mistake is documenting everything. More documentation does not mean better documentation.

Focus on:

• What decisions people need to make

• What information helps them do their job

3. Keep documentation up to date

Outdated architecture is dangerous. It leads to wrong assumptions, broken integrations, and poor decisions.

If you cannot maintain a diagram — don’t create it.

4. Make documentation easy to find and accessible

Store documentation in places people already use:

• Confluence / Notion

• Git repositories

• SharePoint / Wiki

Hidden documentation is as good as no documentation.

Tools for Creating Architecture Diagrams

Architecture documentation is primarily represented using diagrams. There is a wide range of tools available today from simple collaboration tools to specialized enterprise modelling software. The key is choosing the right tool based on purpose, collaboration, and accessibility.

1. Giffy – great for brainstorming and early design

In my view, Giffy works best during the brainstorming and design phase.
It allows better collaboration with development teams and makes it easy to sketch ideas quickly during discussions and workshops.

Best for: early design, workshops, team collaboration.

2. Microsoft Visio – the classic enterprise standard

Most people are already familiar with Visio as it has been a standard Microsoft diagramming tool for years.

It is especially useful when:

• Embedding diagrams into MS Word and PowerPoint

• Working in organisations with Office 365 (often part of the license)

Best for: formal documentation in enterprise environments.

3. PlantUML – diagrams as code

PlantUML is an open-source tool that allows users to create diagrams using a plain text language.

It is a great choice when:

• Storing diagrams in Git repositories

• Working with Markdown documentation

• Creating sequence diagrams and system flows as code

Best for: DevOps-friendly documentation.

https://plantuml.com/

4. Mermaid – lightweight open source alternative

Mermaid is another open-source tool similar to PlantUML.

It works very well inside:

• Markdown files

• GitHub/GitLab documentation

Best for: quick technical documentation.

https://mermaid.js.org/

5. Sparx Enterprise Architect – full enterprise modelling tool

Sparx EA is a comprehensive enterprise architecture and UML modelling platform.

It is commonly used for:

• Detailed UML analysis

• Formal architecture governance

• Large organisations

https://sparxsystems.com/

6. Visual & Collaborative Tools

• Draw.io (diagrams.net) – free, simple, widely used

• Lucidchart – cloud-based, very popular

• Miro – excellent for workshops and brainstorming

• Figma / FigJam – increasingly used for system flows

7. Enterprise Modelling Tools

• Archi (ArchiMate)

• BizzDesign

Architecture Documentation Categories

(With Diagram References & Mandatory Key Elements)

1. Business Context View

Reference: C4 Model – System Context Diagram
Key Elements

• Business goals

• Stakeholders

• User personas

• External partners

• High-level business processes

Why it matters

Aligns technology with business value.

2. Technical Context View

Reference: C4 System Context Diagram

Key Elements

• External systems

• APIs and integrations

• Data flow directions

• Communication protocols

• Ownership of systems

Why it matters

Avoids dependency surprises.

3. Logical Solution Overview

Reference: C4 Container Diagram

Key Elements

• Major services/components

• Responsibilities

• Communication paths

• Technology type (API, DB, queue, UI)

Why it matters

Foundation of the system design.

4. Interface Overview

Reference: UML Component Diagrams

Key Elements

• APIs

• Message brokers

• Protocols

• Data formats

• Versioning

Why it matters

Smooth integrations.

5. Application / Service Inventory

Reference: Application Portfolio Views

Key Elements

• Service name

• Owner

• Technology stack

• Criticality

• Lifecycle status

Why it matters

Governance and modernization.

6. Data Flow View

Reference: Data Flow Diagrams (DFD)

Key Elements

• Data sources

• Processing steps

• Storage

• Consumers

• Frequency

Why it matters

Analytics and compliance.

7. Conceptual Data Model

Reference: High-level ER Diagrams , Conceptual data model

Key Elements

• Business entities

• Relationships

• Definitions

Why it matters

Common understanding.

8. Logical Data Model

Reference: Logical data model

Key Elements

• Attributes

• Keys

• Relationships

• Constraints

Why it matters

Schema design.

9. Physical Data Model

Reference: Database Schema Diagrams

Key Elements

• Tables

• Indexes

• Partitions

• Storage design

Why it matters

Performance.

10. Security View

Reference: Threat Modeling

Key Elements

• Authentication

• Authorization

• Encryption

• Trust boundaries

• Network zones

Why it matters

Risk control.

11. Deployment View

Reference: Infrastructure Architecture Diagrams

Key Elements

• Cloud services/servers

• Environments

• Scaling

• Load balancers

• Monitoring

Why it matters

Reliability.

12. Sequence Diagrams

Reference: UML Sequence Diagrams

Key Elements

• Actors

• Components

• Message flow

• Error paths

Why it matters

Clear behaviour flow.

13. Cloud Account / Subscription Model

Reference: Cloud Landing Zone Architecture

Key Elements

• Accounts/subscriptions

• Network segmentation

• Shared services

• Access controls

Why it matters

Security and cost control.

Additional Highly Recommended Sections

Architecture Decision Records (ADR)

Reference: https://adr.github.io/

Key Elements

• Context

• Decision

• Alternatives

• Consequences

Non-Functional Requirements

Key Elements

• Performance

• Availability

• Scalability

• Disaster recovery

• Security

Technology Standards

Key Elements

• Approved tools

• Frameworks

• Cloud services

• Design patterns

Blog Series Roadmap

This post is part of a larger blog series focused on architecture documentation standardization and best practices.

Upcoming posts will cover:

📌 Series 2 – Architecture Decision Records (ADR)

What ADR is, why it matters, and how to write good decisions.

📌 Series 3 – Documentation as Code

Using Git, Markdown, PlantUML, Mermaid to keep docs always updated.

📌 Series 4 – Using AI for System Documentation

How AI can generate, update, and review architecture docs.

Final Thoughts

Architecture documentation is not about drawing pretty diagrams.

It is about:

• Clear communication

• Better decisions

• Long-term system health

• Faster onboarding

• Less chaos

• Start small.

• Document what matters.

• Keep it updated.

• Make it accessible.

Everything changed when I began using OKRs not as a reporting exercise, but as a way to deliberately redesign systems and measure meaningful outcomes. Reading Measure What Matters by John Doerr helped me understand that objectives without measurable outcomes are just intentions. Experience taught me something deeper: metrics alone don’t create improvement. Improvement happens when teams change the systems that produce those metrics. This insight now shapes how I run quarterly and half-yearly planning sessions with platform and data teams. We focus not on tasks alone, but on impact, learning, and the behaviour of the systems we operate.

Why OKRs Matter in Platform and Data Engineering

Most engineering goals sound sensible on paper: improve reliability, reduce incidents, make pipelines faster. Yet these are aspirations, not mechanisms for change. High-performing platform teams don’t succeed by working harder. They succeed by improving the underlying systems that naturally produce reliability, speed, and trust.

OKRs shift the focus from shipping more work to engineering better outcomes. A DevOps team succeeds when recovery becomes automatic. A data platform succeeds when failures are rare and visible. A BI team succeeds when trusted data is adopted by default. OKRs make these system-level improvements intentional, measurable, and aligned with broader business priorities.

Objectives, Outcomes, and the Missing Link: Causality

An objective defines direction. A key result proves that something changed. But high-impact OKRs include a third, often overlooked layer — causality.

Causality asks: what actions or system changes will directly produce the desired outcome? Without this, OKRs risk becoming little more than scoreboard watching or checkbox exercises. With it, OKRs become engineering experiments.

For example, “Reduce MTTR to 15 minutes” is measurable but vague. Reframed causally: “Reduce MTTR to 15 minutes by automating rollback, introducing SLO-based alerting, and removing manual recovery steps” is actionable. The metric tells you if it worked. The causal design explains why it should.

This logic applies across all domains — technical, operational, and even human productivity. Every key result must break down the objective into measurable outcomes that matter, clearly linking action → system change → benefit.

Bridging Strategy and Daily Engineering Work

Leadership goals often live at a high level: faster innovation, lower operational risk, better data-driven decisions. OKRs translate these into concrete engineering levers.

If the business needs faster experimentation, platform OKRs focus on deployment speed and recovery. If the business needs trustworthy analytics, data OKRs focus on freshness, validation, and visibility. Each key result becomes a controllable system input that drives measurable business impact.

Even behavioral OKRs — mentoring, documentation, process improvements — fit naturally. They ensure that achieving technical outcomes does not come at the cost of team health or long-term capability.

Common Causal Mistakes Teams Make

Even skilled teams fall into familiar traps. One is confusing activity with impact: migrating tooling, building dashboards, or writing scripts — these are tasks, not outcomes. Another is metric chasing without system change: “reduce incidents by 50%” without specifying how. The worst is mistaking busyness for progress: lots of work completed, but no improvement in reliability, speed, or quality.

Strong OKRs always connect: action → system change → measurable outcome.

Real OKRs Across Teams

Platform / DevOps — Designing Reliability Into the System

When deployments kept failing, the team stopped chasing individual incidents and reframed their direction around a single objective: make production deployments predictable, low-risk, and uneventful. Reliability was no longer something hoped for after each release; it became something engineered into the delivery system itself.

Over the quarter, Mean Time to Recovery dropped from ninety minutes to under fifteen as automated rollback mechanisms and service-level alerting replaced manual intervention. Deployment frequency increased from weekly releases to daily changes by removing approval bottlenecks and introducing automated validation gates. At the same time, pipeline success rates climbed to 99.9 percent as testing and infrastructure checks became part of every release.

What changed wasn’t just tooling. The system itself evolved. Engineers stopped firefighting and started trusting the platform. The objective wasn’t automation for its own sake — it was building a delivery pipeline that naturally produced reliability.

Data Platform — Making Trust a System Property

The data platform team faced a familiar problem: pipelines were breaking constantly, but failures were often discovered hours or days later. Rather than reacting faster, the team reframed their objective: build reliable and observable data systems.

Schema validation at ingestion, real-time freshness monitoring, and automated retries for transient failures reduced defects by more than 70% within one quarter. Data freshness SLAs consistently stayed within target windows, and failures were detected within minutes instead of hours. Trust grew organically. Analysts spent less time validating numbers and more time using insights. Reliability was no longer heroic effort — it became the system default.

BI & Analytics — Turning Usage Into a Natural Outcome

The BI team initially believed their challenge was a lack of dashboards. More reports didn’t help adoption. The real objective: enable confident self-service analytics. Standardized certified gold datasets, optimized dashboard performance, and focused enablement sessions helped users understand data meaning and limitations. Usage of certified datasets rose over 50%, while ad-hoc requests dropped. Trust plus speed changed behaviour; adoption became the natural outcome.

Cloud & Cost Management — Efficiency Without Slowing Teams

Cost optimization often conflicts with delivery speed. The team set an objective to run infrastructure efficiently without compromising velocity. Autoscaling, idle resource cleanup, and real-time cost visibility reduced spend by 25% while deployment frequency remained unchanged. Efficiency became a shared responsibility; impact was measurable, immediate, and visible.

Causal OKR Templates Teams Can Reuse

Objective: Improve a system outcome that matters.
Key Results: Each KR describes what measurable result you want and what causal action will drive it.

Reliability / Stability: Reduce downtime to <30 minutes/month via automated recovery; lower incident rate by 50% with validation checks; remove manual interventions through runbook automation.

Data Quality / Accuracy: Reduce defects by 40% using schema/freshness checks; achieve 99.9% SLA with monitoring and automated retries; detect failures within 10 minutes.

Delivery / Speed: Increase deployment frequency by removing manual approvals; reduce lead time via automated testing; maintain failure rate <2% using rollback automation.

Reclaiming Time: Golden Hours for High-Impact Work

After optimizing systems, the next frontier is time. Every minute spent on repetitive tasks is a minute lost for learning, experimentation, and platform improvement. “Golden hours” are blocks of protected, high-value time.

Objective: Reclaim engineer time for learning and high-value work.
Key Results: Reduce manual deployment verification by 2 hours/engineer/week; cut incident recovery from 90 to 30 minutes with automated alerts, rollback, and monitoring; ensure 2 hours/week of protected “golden hour” for experimentation and learning.

Measurement matters: time saved is observable via logs, telemetry, or calendar tracking. Linking productivity to system-level improvements turns minutes into measurable outcomes. Teams that embrace this see engineers planning work that actually moves the needle.

Universal / Personal OKRs

Even outside core engineering, causal OKRs apply:

Productivity / Time Management: Increase focused work hours to 30/week by blocking deep work; reduce context switching; complete 90% of priority tasks.

Learning / Skill Growth: Complete 3 courses + 2 projects in 6 months; apply learnings to optimize pipelines; share insights in team sessions monthly.

Health: Reduce body weight by 5 kg in 3 months through diet and exercise; cut sugar intake 50%; track weekly progress.

The principle is universal: measurable outcomes + causal actions = real impact.

Tracking, Evolving, Celebrating, and Reflecting

OKRs are most powerful when they are visible, tracked, and reviewed. A simple cadence works:

Track → Evolve → Celebrate & Reflect
Quarterly or half-yearly reflection sessions ask: Are we on track? Are key results driving meaningful system change? Which objectives should feed into the QBR? Did any OKR fail, and what did we learn?

Individual OKRs should be 8–10 total, divided between performance and behavioral outcomes. Setting them 1 year in advance is fine for strategy, but remain flexible — change is expected. Leadership isn’t about perfection; it’s about learning, adapting, and improving continuously.

Measurement is Part of Engineering Work

You can’t improve what you can’t observe. MTTR requires incident telemetry. Pipeline reliability requires failure metrics. Time savings require logs or calendar tracking. These aren’t overhead; they’re foundational platform engineering. Every OKR cycle should improve both systems and observability.

How OKRs Change Culture

Over time, the shift becomes visible. Engineers stop asking, “What ticket should I pick up?” They start asking, “What will actually improve reliability, speed, quality, or learning?” Ownership grows, learning accelerates, and impact becomes tangible. OKRs stop being goals and become feedback loops guiding continuous improvement.

Final Thoughts

OKRs work when they connect vision to execution, action to outcome, and effort to system improvement. Causality is what makes that connection real. When objectives set direction, key results measure change, and teams design the system improvements that produce results, OKRs stop being management theatre. They become the operating system of high-performing platform and data teams.

Not about doing more work. About building better systems that naturally produce better outcomes. When golden hours, learning time, and measurement are included, teams don’t just work — they grow, innovate, and lead.

I strongly believe that people are the driving force behind every success story.

Working in the financial and banking domain, within the business division of technology, I have experienced constant organizational change. Over the last eight years, I have moved across four different teams, each with different priorities, structures, and expectations. Change was never an exception. It was the norm.

What made this journey meaningful was not systems or processes alone, but the people and leaders I worked with. I had supportive managers, matrix managers, and leaders who were cooperative and open. These experiences shaped how I think about leadership today.

In a world increasingly shaped by automation, digital transformation, and shifting business models, organizations that succeed are the ones that do not lose sight of their people

The Foundation of High Performance

Building high-performance teams is not easy, especially in high-pressure and fast-moving environments like banking, platform engineering, and shared services.

Most of what I practice today comes from leaders who guided me early in my career. Over the last five to six years, I have applied these lessons while building and running platform, service, and application teams that support multiple stakeholders.

The objective was never speed alone. It was sustainable delivery through people. These principles consistently helped move teams from good to great.

1. Prioritize a Thriving Culture

Culture needs active maintenance, not just well-written policies.

While benefits and flexibility matter, culture is shaped more by daily signals than by documents. One practice I learned from a manager that stayed with me is the idea of a team health score or mood check.

This does not need to be complex. A quick check-in during stand-ups or weekly syncs helps leaders sense energy levels, stress, and early signs of burnout. Over time, these small signals provide far more insight than any annual engagement survey.

Another practice we adopted was protecting golden hours for learning. This meant consciously carving out focused time for upskilling, knowledge sharing, or deep work without meetings. In platform and banking teams, where operational load is high, learning time is usually the first thing to disappear unless it is explicitly protected.

When organizations regularly review how teams actually work and support practices like health checks and learning time culture stays aligned with reality, not assumptions. Without this, even well-intentioned policies slowly drift away from day-to-day experience.

Actionable Practices:

Team or Pod Charter
In my crew, we created a simple charter that defined purpose, responsibilities, and ways of working. This lightweight framework reduced ambiguity and improved focus without introducing heavy process.

The Five-Minute Rule
Starting meetings with five minutes of open discussion helped teams transition into complex technical conversations. This small habit improved context sharing and trust, especially during long or demanding sessions.

2. Embrace Open and Accessible Leadership

At every stage of my journey, I worked with leaders who were approachable and available. That had a lasting influence on how I lead today.

People want to understand how their work connects to outcomes. When that connection is missing, engagement fades. In banking and platform environments, where work often spans multiple systems and timelines, this connection becomes critical.

Approachable leadership creates psychological safety. Teams that feel safe will raise risks early, challenge assumptions, and experiment responsibly. As a leader, you do not need to have all the answers. What matters is:

• Presence: Being available when teams hit blockers

• Transparency: Connecting daily work to business and platform outcomes

• Safety: Making it easy for someone to message you with a concern or an early-stage idea

3. Foster Continuous Team Development

Teams must evolve at the same pace as the platforms they build. In platform and service teams, we consciously avoided the SME trap. Relying on a single subject matter expert creates bottlenecks, increases risk, and limits team growth.

Instead, we focused on continuous upskilling, shared ownership, and knowledge distribution. The mindset shifted from “who owns this task” to “how does this help our customers and stakeholders”.

Growth is not only about learning new tools or frameworks. It is about learning how to think together, how to adapt under pressure, and how to respond to change without fear.

From Principles to Outcomes

One hard lesson I learned along the way is this: If a minor infrastructure change requires detailed explanation to a junior engineer, the issue is rarely the person. It is usually the architecture, the practices, or both.

When this happens, it is not an individual failure. It is a team and system failure. Complexity has leaked into places where it should not exist. Clear paths are missing. Documentation, ownership, or automation is not doing its job.

As platform and data teams, our responsibility is to design systems that are safe to operate, easy to reason about, and forgiving by default. Teams should be able to make small changes confidently without fear or excessive hand-holding.

This shift in thinking led to tangible outcomes:

• We started saying “yes” to user and stakeholder requests with confidence — “I can get that done tomorrow” — because of a wonderful team that took ownership and consistently delivered.

• Development environments were optimized for learning and speed: yesterday’s data with today’s code, across multiple environments. This was made possible through our ephemeral environments, an idea from the team and executed by a junior engineer, which gives everyone the freedom to test, learn, and iterate safely.

• Production remained stable and predictable: today’s data with yesterday’s code, running in a single, controlled environment

These outcomes were not achieved by pushing people harder. They came from fixing the platform, clarifying ownership, improving documentation, and simplifying paths.

From Good Teams to Great Teams

Great teams do not happen by chance. They are built on:

• A culture that evolves with the organization

• Trust created through approachable leadership

• A shared commitment to continuous growth

Every crew, pod, or platform team must recognize that its true strength lies in its people. When flexibility, collaboration, and development are prioritized, teams consistently deliver strong and sustainable results.

If you are a new leader, a team lead, or someone scaling teams in a changing environment, I hope these lessons help. I learned them from leaders who generously shared their frameworks with me, and I try to pass them forward with humility.

If this resonates and you want to exchange ideas on leadership, platform teams, or engineering culture, feel free to connect. We grow better by learning together.

During my recent job search, I noticed a common pattern. Different companies use different titles for similar engineering roles. Sometimes the same title means very different responsibilities depending on the product or team. Around this time, a few of my juniors asked me direct questions:

• How do I choose the right position?

• How do I compare what I’m doing today with what the market is offering?

• How do I understand where I actually fit?

To give them a proper answer, I started building this blog. I also looked at how various top companies structure their engineering functions and how they define growth. Every organization has its own style, but the fundamentals are largely the same. Whatever your discipline software, quality, data, ML, security, SRE, TPM, or engineering management the core principles of scope, responsibility, autonomy, and impact do not change. You simply become a subject matter expert in your chosen area.

This blog brings everything together as a simple and practical Career Engineering Framework. It explains how engineering roles are structured, how levels grow, and how engineers and managers can use these principles to choose roles confidently and plan long-term development with clarity.

1. What a Career Framework Is and Is Not

What it is?

• A transparent view of expectations at each level

• A clear way to understand growth

• A guide for impact, behaviour, and technical depth

• A tool to drive better feedback and performance conversations

What it is not?

• Not a checklist for promotion

• Not a fixed list of tasks for every engineer

• Not a replacement for mentorship or good judgement

2. The Structure of the Framework

Every engineering role and level can be understood through three common anchors:

A. Level Expectations

This defines how an engineer grows in:

• Scope – size and complexity of problems

• Autonomy – level of guidance required

• Collaboration Reach – who and what your work impacts

B. Core Responsibility Pillars

These apply to all engineering roles:

1. Results

2. Direction

3. Talent

4. Culture

C. Craft Responsibilities

This refers to role-specific technical depth such as:

• Coding

• Testing

• Data modelling

• ML experimentation

• SRE reliability patterns

• Security architecture

• Program leadership

• People leadership
  

  This structure keeps things consistent across disciplines.

3. Understanding Levels: Scope, Autonomy, and Collaboration

Engineering levels follow a predictable growth arc. To keep things simple and relatable across companies, this framework uses both naming styles:

• L1–L6 (commonly used in many global organizations)

• IC2–IC6 (commonly used in tech companies)

They map naturally, so you can understand where you stand regardless of the title used in any job description.

L1 – Entry Level (IC1/IC2 Equivalent)

Focus: Learning and executing with support.
Characteristics:

• Takes clearly defined tasks

• Learns tools, codebase, and processes

• Requires regular guidance

• Building consistency and fundamentals

L2 – Junior Engineer (IC2/IC3 Equivalent)

Focus: Executing independently on low-complexity tasks.
Characteristics:

• Works on small features end-to-end

• Understands team workflows

• Asks for help when blocked

• Starting to recognize risks, but relies on senior engineers

L3 – Mid-Level Engineer (IC3/IC4 Equivalent)

Focus: Independent execution and steady ownership.
Characteristics:

• Designs and ships features independently

• Works with limited direction

• Handles moderate ambiguity

• Collaborates with cross-functional partners

• Begins contributing to design discussions

L4 – Senior Engineer (IC4/IC5 Equivalent)

Focus: Leading complex systems and improving team execution.
Characteristics:

• Leads medium-to-large initiatives

• Designs components or subsystems

• Makes strong architectural decisions

• Mentors junior engineers

• Identifies long-term risks early

• Drives cross-team conversations when needed

L5 – Staff Engineer (IC5/IC6 Equivalent)

Focus: System-wide ownership and high-impact leadership.
Characteristics:

• Owns major product areas or platforms

• Leads multi-team efforts

• Sets technical direction for a domain

• Deep domain expertise

• Strong influence across teams

• Solves problems that have no predefined path

L6 – Principal / Senior Staff (IC6+ Equivalent)

Focus: Organization-wide technical strategy and long-term impact.
Characteristics:

• Shapes technical strategy for large org areas

• Designs long-lived systems and frameworks

• Mentors senior engineers and managers

• Handles high ambiguity and high-risk decisions

• Influence extends across products or business units

• Trusted technical advisor for leadership

How Levels Grow?

Across L1 → L6, four arcs evolve:

1. Scope
   Small tasks → features → systems → multi-team platforms → organization-wide strategy

2. Autonomy
   Guided → Independent → Leading → Defining direction

3. Collaboration Reach
   Immediate team → Multiple teams → Product area → Whole organization

4. Impact
   Execution → Ownership → Influence → Strategy

As you grow, you handle:

• bigger problems

• deeper complexity

• longer time horizons

• broader influence

4. What “Impact” Really Means?

Impact is the heart of career growth. It is not the number of tasks you close or the amount of code you write. Impact usually comes through five levers:

1. Deep domain expertise

Becoming the go-to person in an important product or system area.

2. Innovation

Finding faster, simpler, or more effective solutions.

3. Product thinking

Balancing user needs, engineering quality, and delivery speed.

4. Project leadership

Handling large, ambiguous, cross-team work.

5. Technical leadership and mentorship

Uplifting people around you through guidance and clarity.

5. Core Responsibilities (Across All Engineering Roles)

A. Results

• Deliver measurable outcomes

• Solve real problems end-to-end

• Reduce risks and improve stability

• Prioritize effectively

B. Direction

• Bring clarity in ambiguous situations

• Make sound technical and execution decisions

• Introduce healthy change and improvements

C. Talent

• Grow your own skills intentionally

• Mentor others through examples and guidance

• Help in hiring and onboarding

D. Culture

• Communicate clearly

• Build trust with teammates

• Collaborate without ego

• Support an inclusive and healthy work environment
  

6. Craft Responsibilities by Discipline

6.1 Software Engineering

• Writing clean, maintainable code

• Designing components and systems

• Balancing short-term delivery with long-term stability

• Driving technical decisions

• Owning operational health

6.2 Quality Engineering

• Strong test strategy

• Automation mindset

• Risk identification and communication

• Quality metrics and measurement

• Influencing teams towards quality-first thinking

6.3 Site Reliability Engineering

• Deep systems understanding

• Observability and incident response

• Reliability architecture

• Failure domain design

• Long-term resilience improvements

6.4 Data Engineering

• Data modelling

• Pipelines and ETL architecture

• Data governance and quality

• Scalability and performance

• Observability and lineage

6.5 Machine Learning Engineering

• Algorithm selection

• Experimentation and evaluation

• ML system design

• Productionising models

• Monitoring model drift and behaviour

6.6 Security Engineering

• Threat modelling

• Secure architecture

• Control implementation and validation

• Security reviews and risk communication

• Embedding security into design

7. Technical Program Manager (TPM)

TPMs handle the execution backbone of a multi-team engineering organisation.

Scope

• Cross-team programs

• Complex migrations

• Infra and platform initiatives

• Compliance and regulatory work

• Multi-quarter product launches

Responsibilities

• Breaking down large problems

• Driving alignment across engineering, product, design, data, security

• Managing risks, dependencies, and timelines

• Creating visibility through dashboards and metrics

• Keeping teams unblocked and focused

• Ensuring predictable and sustainable execution

TPM Impact Levers

• Clear execution

• Strong stakeholder trust

• Cross-team clarity

• Reduction of delivery risk

• Making the organisation move faster and safer

8. Engineering Manager (EM)

Scope

One team, or a focused product area.

Key Responsibilities

• Build a healthy, high-performing team

• Develop engineers through coaching and feedback

• Guide technical direction with senior ICs

• Manage execution and delivery

• Partner with PMs, TPMs, and senior leaders

• Maintain technical quality, reliability, and process health

Impact Levers

• Team culture

• Decision clarity

• Talent development

• Predictable delivery

9. Senior Engineering Manager (Sr EM)

Scope

Multiple teams or a product area with broad responsibilities.

Responsibilities

• Setting long-term area strategy

• Managing managers and senior ICs

• Driving multi-team execution governance

• Improving organizational structures

• Balancing product, tech, infra, and operational work

• Hiring and developing leadership talent

Impact Levers

• Quality of leadership pipeline

• Strategic decision-making

• Area-level delivery and stability

• Cross-functional influence

10. How Engineers Should Use This Framework

A simple, practical approach:

1. Understand your current level honestly

2. Identify your strongest impact levers

3. Pick one or two growth areas

4. Plan quarterly with your manager

5. Focus on outcomes, not activities

6. Document your contributions

7. Share your work openly

11. How Managers Should Use This Framework

Managers can use this framework to:

• Set transparent expectations

• Give objective, behavioural feedback

• Build structured growth plans

• Ensure fairness in promotions

• Hire better and faster

• Build stronger, healthier teams

12. Closing Summary

Across all roles ,ICs, managers, TPMs, and domain specialists fundamentals remain the same:

• Grow your scope

• Increase your autonomy

• Expand your collaboration reach

• Strengthen your craft

• Deliver meaningful impact

• Support people around you

Titles, tools, and domains change. But growth always follows the same arc:

13. Further Reading & Resources

📋 Data Team Handbook
https://github.com/sdg-1/data-team-handbook
A practical companion for data engineers and leaders, covering IC-to-manager growth, team culture, hiring, onboarding, and data infrastructure.

📋 Consulting Handbook
https://github.com/sdg-1/consulting-handbook
Templates and strategies for senior engineers and TPMs exploring consulting or independent paths.

Bonus: Compare with real-world ladders from GitLab , Glitch’s Engineering Ladder and the Awesome Engineering Ladders repository.

This blog was sparked by a thoughtful 1:1 discussion I had with Michael Bharwani (My Mentor). We were talking about automation, how organizations are rapidly adapting to AI-led workflows, and what it truly means when machines start replacing human work. Somewhere in that conversation, one simple question came up:

“If bots don’t pay taxes, what happens to the government’s income?”

That question stayed with me. Because as AI grows in influence, it is not just jobs and skills that will change. Taxation systems, national budgets, and even social contracts will have to adapt.

Automation is already part of everyday operations. Across industries, bots and AI platforms are performing human tasks. But there is one big difference: they don’t pay taxes. No income tax, no social contribution, no pension.

This simple shift can have deep economic effects. So let’s unpack what it means for organizations, governments, and society.

What’s Happening Around the World

A few countries have already started experimenting with ways to handle this challenge.

• South Korea introduced an automation levy on companies using robots, channeling the funds into reskilling programs.

• Italy debated a “robot tax” to support welfare systems for displaced workers.

• The European Union is exploring digital tax frameworks, recognizing that automation is now both an economic and social issue.

These examples show one thing clearly: the world is already searching for fair ways to adapt tax systems to automation.

The South Korea AI Framework Act, enacted in early 2025, focuses on AI system regulation, transparency, and responsible innovation but stops short of specifying an automation or “robot” tax. However, their broader 2025 tax reform aligns with global minimum tax principles targeting multinational enterprises’ profits, reflecting a move to capture economic value from digital activities including AI. The government is also investing heavily in AI infrastructure and retraining programs, effectively channeling automation gains back into the workforce.

In Italy, the “robot tax” has been debated mainly in the political arena as a potential levy on companies that replace human labor with automation to fund social welfare systems for displaced workers. While a formal nationwide tax has not been enacted yet, ongoing discussions show a strong interest in linking automation to social equity. Several proposals suggest a productivity or profit-based levy instead of a simple headcount tax, reflecting the global trend of moving towards value-linked taxation.

Across all jurisdictions, similar policy challenges persist. Defining what counts as taxable automation or AI is complex because systems vary widely from assistive tools to fully autonomous agents. Measuring AI-driven value objectively is difficult without standardized metrics. Poorly designed taxes also risk slowing innovation or driving investment to regions with fewer regulations.

Data privacy and ethics add another layer of complexity. Any tax model that relies on measuring automation intensity such as query volumes or compute usage must balance transparency with privacy rights. Excessive monitoring or intrusive data collection could easily trigger resistance from both businesses and citizens.

The Big Question: What Counts as a Robot?

Defining “robot” used to be simple. In factories, robots replaced physical labor. But today, automation happens in code, in the cloud, through APIs, and in AI workflows.

So where do we draw the line?

• Is a chatbot a robot?

• Is AI-generated content part of taxable work?

• Or is automation inside data platforms like Starburst or Trino, processing millions of queries a day, also “labor” in some sense?

Once automation becomes digital, even defining work becomes complex. Governments will have to rethink what counts as “value creation.”

This is a debatable area and still evolving. There are no clear global standards yet, but these questions are now part of serious policy discussions in many countries.

Can Governments Really Do Without Taxes?

No government can run without taxes. It funds everything from education and healthcare to infrastructure and welfare.

For now, automation seems beneficial, giving higher productivity and better margins that can increase corporate tax contributions. But over time, as human jobs reduce, income tax revenues will fall, and that is where the real gap begins.

Governments will then need to shift from taxing people to taxing productivity and automation. That change will not be easy, but it is necessary.

Possible Tax Models for the Future

Here are some practical models that are being explored or discussed:

• Automation Offset Funds: Companies contribute a share of what they save through automation into skill development or retraining programs.

• Value-Based Taxes: Linked to productivity gains or automation-generated profits rather than headcount.

• Energy or Compute-Based Taxes: Levies tied to energy or compute resources used by large-scale AI systems.

• Data Monetization Taxes: Taxes on revenue derived from AI-driven data usage or analytics.

Each approach moves away from labor and focuses on the value created by automation.

Some of these ideas are still theoretical. Whether they become real policy tools depends on how governments interpret automation data and design fair frameworks.

Balancing Innovation and Fairness

This balance is delicate. Tax automation too heavily, and innovation could slow. Leave it untaxed, and inequality widens. The key lies in designing policies that encourage innovation but also ensure fairness and redistribution.

AI is global, but taxation is local. That makes policy design harder. Collaboration between economists, technologists, and governments will be crucial to keeping systems both competitive and fair.

Does It Still Make Sense to Automate If AI Is Taxed Like Humans?

(A thought that shaped this blog from my conversation with Michael Bharwani)

In that 1:1 discussion, Michael asked a powerful question: If governments start taxing AI like humans, will automation still make business sense? It made me think deeply. My answer: yes, automation will still be valuable. But the reason will change.

Consistency and Scalability
AI works round the clock, at scale, and with precision that humans can’t match.

Lower Overheads
Even with taxation, automation saves long-term costs, with no benefits, offices, or HR dependencies.

Smarter Decisions
AI learns continuously. Data feedback loops improve decision quality across business functions.

Global Consistency
Automation systems can operate uniformly across countries, unlike human processes that need localization.

Human-AI Collaboration
AI doesn’t replace human creativity; it amplifies it. Humans focus on strategy, judgment, and empathy.

So even if AI is taxed, it will still deliver strong operational and analytical value, not through cost-cutting but through smarter, scalable capabilities.

How Data Platforms Fit In

Platforms like Starburst and Trino are clear examples of invisible automation. They process massive data volumes, automate reporting, and support large-scale decision-making every second.

If governments start measuring automation intensity such as number of queries, data processed, or compute usage, these could become new indicators for taxation.

That is how we may see platform-based tax models, where tax links to the amount of automation rather than the number of human workers.

Interestingly, we are already seeing hints of this approach. In Poland, for instance, a creative tax model was introduced that indirectly captures value from intellectual and digital work. It is not automation-specific yet, but it is a step in that direction.

Every country will eventually design its version of this model. It will also create new roles: economists, policy specialists, and domain experts who translate automation data into policy frameworks.

But this will not be simple. Balancing innovation with fairness will take time. It is a wait and watch phase right now, as the dialogue between automation and taxation continues to evolve.

This section remains debatable, as such models are not yet standardized or globally adopted. But the direction is clearly forming.

Defining Possible Automation Tax Metrics

For technically proficient audiences, such as those using platforms like Starburst or Trino, automation tax metrics could be defined through measurable system activity. For example:

Using such metrics as inputs for value-based taxation models can help quantify the economic activity driven by AI systems rather than relying only on human labor metrics. Again, this idea is open for debate, as we are still figuring out how to measure such automation impact effectively.

Productivity vs. Equity

Automation brings productivity. But unchecked, it can also widen economic gaps. That is why taxation plays a critical balancing role, ensuring some of the gains go back into human skill development and social support.

The real success of automation will not be how many tasks it replaces, but how many people it uplifts.

Conclusion: Preparing for a Tax System That Understands AI

AI is redefining work, value, and contribution. Yet most taxation systems are still built around jobs, salaries, and payroll. The future will need a value-based taxation system, one that measures productivity, automation, and data-driven value creation.

It is a complex transformation, but a necessary one. Automation should not mean fewer opportunities for humans. It should mean smarter, fairer, and shared progress for all.

A collaborative effort between technologists, economists, and policymakers will be the key to shaping a fair, future-ready taxation framework that keeps innovation alive while ensuring everyone benefits from the value AI creates.

"How are you doing? Really doing?" I looked up from my laptop, surprised. It was our weekly one-on-one, and I'd prepared my usual status update: tasks completed, blockers identified, next week's priorities. But my manager wasn't opening his notebook or checking his calendar. He was just... waiting.

That conversation marked the beginning of my transformation from someone who managed work to someone who could lead people. Over 17 years in various roles, but especially in the last 8, I've learned that manager and leader aren't just different job titles—they're completely different ways of showing up.

A Note from the Author: I really enjoyed crafting this story from start to finish! After I completed the writing, I used ChatGPT to help polish the grammar and make it even easier to read, all while keeping my original thoughts intact.

The Difference That Matters

Here's what I've observed after working under both managers and leaders:

Managers focus on:

• Tasks and deliverables

• Status updates and timelines

• Compliance and process

• What needs to be done

• Control and oversight

Leaders focus on:

• People and their growth

• Purpose and meaning

• Development and potential

• Why it matters

• Trust and empowerment

Three Leaders Who Shaped My Journey

My transformation didn't happen overnight. It came through working with exceptional people who modeled what leadership actually looks like:

Paul Banoub laid the foundation through his "House View of Leadership" sessions and genuine investment in people. He was a great mentor and an even better listener. Every one-on-one felt like he had all the time in the world for whatever was on my mind. His belief in leadership training showed me that developing others wasn't optional—it was essential.

Michael Bharwani has been my guiding presence for three years. He's been consistent with feedback, generous with encouragement, and sharp in identifying improvement areas. What sets him apart is his accessibility—he's still someone I turn to today, long after our direct reporting relationship ended.

Kevin Ruscoe helped me discover strengths I didn't know I had in product thinking, process design, and stakeholder management. He was one of the first to believe I could do more than just deliver—that I could lead. He didn't just assign me work; he assigned me purpose.

Each of these leaders shared something crucial: they didn't just manage my output; they invested in my potential.

My Leadership Framework: CARE

Through observing these leaders and eventually leading teams myself, I developed what I call the CARE Framework:

Connect — Build real relationships beyond tasks and tickets. Know what motivates your people, what challenges them outside of work, what they aspire to become.

Align — Link people's individual goals to the larger purpose. Help them see how their work creates impact beyond just completing assignments.

Recognize — Celebrate effort, outcomes, and progress openly. Not just the big wins, but the daily improvements and growth moments that often go unnoticed.

Empower — Give ownership and create space for people to figure things out. Trust your team's judgment and provide support when they need it.

This isn't a checklist to complete—it's how I try to show up every day.

The Practical Shift

If you're a manager wanting to become a leader, here are the shifts that made the difference for me:

Listen more than you talk. Your team is constantly telling you what matters to them, what frustrates them, and what energizes them. Most of us just aren't tuned in.

Guide instead of just assigning. When you give someone a task, help them understand why it matters and how it connects to something bigger.

Make career growth a regular conversation. Don't wait for annual reviews. Ask about aspirations, create development opportunities, and advocate for your people.

Create visibility for your team. Let them present to leadership, lead important calls, and get recognition for their contributions.

Celebrate progress, not just outcomes. Recognition builds confidence, and confidence drives performance.

When Leadership Really Clicks

The shift became real for me when I started managing my own teams. Instead of focusing solely on deliverables, I began asking different questions: How is this person growing? What do they need from me? How can I help them succeed beyond this project?

The results spoke for themselves. Team engagement improved, people stayed longer, and surprisingly, our actual delivery got better too. When people feel invested in, they invest back.

Resources That Helped My Journey

These books gave me language for what I was experiencing and frameworks for what I was trying to become:

• "Leaders Eat Last" by Simon Sinek — Showed me how empathy builds safety and trust in teams

• "The 5 Levels of Leadership" by John Maxwell — Helped me view leadership as a journey of influence, not just authority

• "The Making of a Manager" by Julie Zhuo — A refreshingly honest take on modern leadership challenges

The Truth About Leadership

Here's what I've learned: Leadership doesn't begin with a promotion or a title. It begins with intention. It starts when you decide that your success is measured not just by what you accomplish, but by how you help others accomplish things they didn't think were possible.

If you're a manager today, pause and ask yourself: Am I managing work, or am I leading people?

The answer lies in how you show up. Do you see your role as getting tasks completed, or developing the humans who complete those tasks?

Leadership isn't about perfection. It's about getting better. It's not about having all the answers. It's about asking better questions.

Let's not just manage projects and processes. Let's lead people. And in the process, let's lead ourselves toward the kind of leaders we wish we'd had when we were starting out.

What's one shift you could make today to move from managing tasks to leading people? Sometimes the most powerful leadership development happens one conversation at a time.

I was sitting in my year-end review (2017), three months into joining a new company and team. My manager pulled up his notes and said something I'll never forget:

I'd been hearing this phrase since I joined—it was woven into how our team operated. But hearing it in my formal review, connected to my performance and future potential, made it click differently. This wasn't just team vocabulary. This was the standard I was being measured against.

The feedback was clear: my strength lay in autonomous problem-solving and incident management. When issues arose, I didn't freeze—I acted. "If I want something addressing quickly and thoroughly," my manager wrote, "Thara is a great person to have on the team."

That review became my north star. The mantra wasn't just about delivery, it was about building the kind of reliability that makes managers' lives easier and teams more effective.

A Note from the Author: I really enjoyed crafting this story from start to finish! After I completed the writing, I used ChatGPT to help polish the grammar and make it even easier to read, all while keeping my original thoughts intact.

Living the Charter Daily

From 2017 onward, "Deliver. Deliver. Deliver." became more than words it became operational DNA. When licensing issues erupted, I delivered solutions. When workspace problems threatened client experience, I delivered fixes. When emergency package rollouts were needed, I delivered results.

But the real lesson came through watching my manager and the Head of Analytics as a Service. They didn't just preach delivery but they modeled it. Every project started with ruthless clarity about what we were solving and why it mattered. Then came consistent execution, adjustment based on feedback, and more execution.

I realized the charter had an invisible first step: Define. Before you can deliver effectively, you must know exactly what you're delivering and why.

The Framework in Action: Cloud Migration Reality Check

By 2021, I was applying this evolved framework “Define. Deliver. Deliver. Deliver.” to increasingly complex challenges. Our Tableau cloud adoption project exemplified the approach.

The challenge: migrate Tableau Server to the azure cloud using containers, when container support was still experimental. My initial approach was chaotic—scripting solutions, forcing DevOps practices I'd only read about, trying to solve everything simultaneously.

The breakthrough came when I applied the charter's hidden wisdom: Define first. Instead of diving into technical implementation, I stepped back:

• What problem were we actually solving? (Cloud adoption, not just containerization)

• What capabilities did we need? (Dev+Data+Ops mindset, not just tools)

• What would success look like? (Sustainable platform engineering, not just working containers)

With clarity established, delivery became systematic. We shipped small wins: basic container deployment, then orchestration, then infrastructure as code. Each delivery taught us something and built stakeholder confidence.

The project became my crash course in platform engineering—understanding how people, process, product, and technology intersect. That foundation opened doors to leadership roles and bigger challenges.

Passing the Charter Forward

When I started managing teams, the charter came with me. During our Power BI automation initiative with Microsoft vendors, I found myself echoing my old manager: "Let's define what success looks like, then deliver incrementally."

The approach eliminated friction and kept everyone aligned. Instead of getting lost in vendor capabilities, we focused on our organization's specific requirements and built solutions step by step.

Later, leading our feature store project, we delivered in record time with a small team by religiously following the framework. Currently, I manage an infrastructure engineering pod where this approach is baked into our team charter. Result: 2x efficiency improvement over two years.

Why This Becomes Career DNA

The genius of "Deliver. Deliver. Deliver." isn't in the repetition, it's in the compound effect. The first delivery proves capability. The second shows consistency. By the third, you've built a reputation.

My 2017 review highlighted something crucial: when problems arose, I didn't freeze, I acted. That behavior, reinforced by the team charter, became my professional brand. Managers knew they could count on me for "quick and thorough" solutions.

This reliability opened doors. Not because I was the smartest person in the room, but because I was the one who consistently turned problems into progress.

The Evolution Continues

Seven years later, that year-end feedback still guides everything I do. The charter has evolved I've added "Define" as the crucial first step but the core remains: consistent delivery builds careers.

When I share this framework in training sessions or with struggling teams, the response is always the same: clarity. Whether you're a new manager building credibility, an individual contributor seeking visibility, or anyone trying to make impact, the pattern works.

The Lasting Gift

My manager's greatest gift wasn't the mantra itself it was showing me how simple frameworks, consistently applied, compound into extraordinary results. That 2017 review didn't just evaluate my performance; it revealed the path to sustainable success.

The best professional advice often comes disguised as everyday expectations. Sometimes the most powerful career accelerator is simply becoming the person others can count on to deliver.

What standards or mantras guide your work? Sometimes the most transformative lessons come wrapped in the simplest expectations.

DCD suggested reads:

VizMasters

The Data Looks Healthy. But the Culture is Cracking.

The charts say we’re fine…

Read more

a year ago · 7 likes · Tharashasank Davuluru

When working with Starburst Enterprise and Delta Lake on Azure Data Lake Storage (ADLS), performance tuning often involves tweaking cache properties for Hive Metastore and Delta connectors. However, caching can sometimes do more harm than good. Recently, we experienced an outage triggered by queries not reflecting the latest file changes in ADLS. This blog walks through what happened, how we fixed it, and recommendations on cache management moving forward.

The Problem

Our catalog configuration used the following defaults:

hive.metastore-cache-ttl=20m
hive.metastore-refresh-interval=10m

This configuration meant Starburst cached metadata and refreshed it only every 10–20 minutes. When new files were added to ADLS through Delta Lake transactions, queries continued reading stale metadata. This caused users to see outdated or missing data, resulting in a production outage.

Root Cause

The combination of long-lived Hive Metastore cache and Delta metadata cache caused Starburst to serve old snapshots of Delta tables. Inserts, schema updates, or compactions applied in ADLS were invisible until the cache expired. For workloads where data correctness and freshness are critical, this lag is unacceptable.

The Fix

Following Starburst recommendations, we updated catalog properties to disable or minimize caching:

# Disable Hive metastore metadata cache
hive.metastore-cache-ttl=0m
hive.metastore-stats-cache-ttl=0m

# Disable caching of partition listings and missing entries
hive.metastore-cache.cache-partitions=false
hive.metastore-cache.cache-missing=false

# Disable Delta metadata caching
delta.metadata.cache-ttl=0m
delta.metadata.live-files.cache-ttl=0m

# Disable transaction log caching
delta.fs.cache.disable-transaction-log-caching=true

What These Settings Do

• hive.metastore-cache-ttl=0m: Queries fetch fresh table and partition metadata every time.

• hive.metastore-stats-cache-ttl=0m: Avoids using stale statistics during query planning.

• hive.metastore-cache.cache-partitions=false: Ensures partition listings are never cached and always fetched fresh.

• hive.metastore-cache.cache-missing=false: Prevents caching of missing table or partition lookups.

• delta.metadata.cache-ttl=0m: Forces reading of the latest Delta transaction log on every query.

• delta.metadata.live-files.cache-ttl=0m: Makes sure queries reflect current sets of Delta data files.

• delta.fs.cache.disable-transaction-log-caching=true: Disables caching of transaction log listings, keeping data fresh.

Metadata Caching Across Different Table Formats

Different table formats have distinct metadata caching behavior in Starburst/Trino. Understanding these differences helps in tuning cache settings effectively:

• Hive

  • Properties like hive.metastore-cache-ttl, hive.metastore-refresh-interval, cache-partitions, and cache-missing control caching.

  • Fully documented at Metastores — Trino 476 Documentation.

  • Starburst uses the Hive Metastore primarily for file location and table metadata.

• Iceberg

  • Similar caching concepts as Hive, but Iceberg tables store metadata differently (metadata files on the data lake).

  • Iceberg connector docs at Iceberg connector — Trino 476 Documentation list additional parameters like snapshot and manifest caching.

• Delta Lake

  • Delta tables have their own cache TTL properties, like delta.metadata.cache-ttl and delta.metadata.live-files.cache-ttl.

  • Documentation at Delta Lake connector — Trino 476 Documentation highlights these along with procedures to flush caches manually.

  • Unlike Hive/Iceberg, Delta’s metadata is tied directly to transaction logs, making it critical to tune cache settings for correctness when tables are updated frequently.

Key takeaway: Even though the concept of metadata caching exists across all connectors, the specific properties, defaults, and behavior differ depending on whether you are using Hive, Iceberg, or Delta.

Flushing Caches Manually

Sometimes, despite tuned TTLs, you need an immediate cache refresh. Starburst provides system procedures:

-- Flush all Hive metadata caches
CALL system.flush_metadata_cache();

-- Flush cache for a specific table
CALL system.flush_metadata_cache(schema_name => 'my_schema', table_name => 'my_table');

-- Flush cache for a specific partition
CALL system.flush_metadata_cache(
    schema_name => 'my_schema',
    table_name => 'my_table',
    partition_columns => ARRAY['dt'],
    partition_values => ARRAY['2025-08-25']
);

-- Flush filesystem cache for a table
CALL system.flush_filesystem_cache(schema_name => 'my_schema', table_name => 'my_table');

These are essential during data ingestion, compaction, or troubleshooting when new data is not visible.

Trade-offs

• Correctness & Freshness: Queries no longer read stale metadata or files.

• Performance Impact: Every query now hits the Hive Metastore and Delta log directly, increasing latency and load.

Practical Recommendation

• For production systems where correctness is critical, disable caching as shown above.

• For balanced performance, consider short TTLs (30 seconds to 1 minute) instead of completely turning off caches. This reduces load while keeping data reasonably fresh.

Sample Production-Ready Catalog Configuration

connector.name=delta_lake
hive.metastore.uri=thrift://hive:9083

# Metadata cache tuning
hive.metastore-cache-ttl=1m
hive.metastore-stats-cache-ttl=1m
hive.metastore-refresh-interval=1m

# Disable caching of partitions and missing entries
hive.metastore-cache.cache-partitions=false
hive.metastore-cache.cache-missing=false

# Delta-specific cache controls
delta.metadata.cache-ttl=0m
delta.metadata.live-files.cache-ttl=0m
delta.fs.cache.disable-transaction-log-caching=true

# Security and table registration
delta.security=starburst
delta.register-table-procedure.enabled=true

# Azure settings
fs.native-azure.enabled=true
azure.auth-type=OAUTH
azure.oauth.tenant-id=xxxx
azure.oauth.client-id=xxxx
azure.oauth.secret=xxxx

Final Thoughts

Caching is a double-edged sword. While it improves performance, it may silently introduce data staleness when working with Delta Lake on ADLS. Our choice to disable caching completely ensured query correctness at the expense of latency. Going forward, tuning cache TTLs to small values may strike the right balance.

If you use Starburst with Delta Lake on ADLS, review your cache settings today—your default settings might be hiding stale data issues.

References

1. Starburst Delta Lake Connector Documentation

2. Starburst Blog: Delta Lake Concurrent Writes and Consistency

3. Delta Lake Official Website

4. Starburst Blog: Moving On-Premise Data to Azure Cloud ADLS

5. Starburst Starburst Enterprise on Azure Overview

This guide is for Data Engineers, DevOps, and Site Reliability Engineers who want to maximize uptime and optimize Starburst clusters using native APIs combined with monitoring tools and automation.

Starburst (Trino) Architecture: Coordinator and Worker Roles

• Coordinator: Acts as the control plane — receiving queries, planning execution, scheduling tasks, and aggregating results.

• Workers: Perform the actual data processing — scanning, filtering, joining, and aggregating in parallel.

Monitoring focuses primarily on the coordinator’s APIs, which expose cluster-wide state and workload metrics.

Coordinator State Endpoint: /v1/info/state

Request:

curl http://coordinator_host:8080/v1/info/state

Response Example:

"ACTIVE"

Possible States and Operational Meaning:

ACTIVE

• Node fully initialized and serving queries

• Safe to route production queries

STARTING

• Node initialization in progress

• Hold traffic until ready

SHUTTING_DOWN

• Node gracefully stopping

• No new queries; prepare for maintenance

FAILED / INACTIVE

• Node encountered errors or is down

• Requires operator action or automated restart

Best Practice: Use /v1/info/state as the readiness probe in Kubernetes. Unexpected transitions should trigger alerts.

Cluster Metrics Endpoint: /v1/ui/api/stats

Request:

curl http://coordinator_host:8080/ui/api/stats

Sample Response:

{
  "runningQueries": 3,
  "blockedQueries": 1,
  "queuedQueries": 2,
  "activeWorkers": 5,
  "runningDrivers": 15,
  "totalAvailableProcessors": 20,
  "reservedMemory": 5120000000,
  "totalInputRows": 1500000,
  "totalInputBytes": 6000000000,
  "totalCpuTimeSecs": 4500
}

Metric Deep Dive:

• runningQueries / blockedQueries / queuedQueries → Query execution pipeline.

• activeWorkers → Number of connected, healthy workers.

• runningDrivers → Execution threads currently active.

• totalAvailableProcessors → Total CPU capacity across cluster.

• reservedMemory → Memory allocated to queries (bytes).

• totalInputRows / totalInputBytes → Cluster throughput since startup.

• totalCpuTimeSecs → Total CPU time consumed by queries.

Operational Tips:

• Alert when queuedQueries or blockedQueries spike.

• Track activeWorkers to detect node failures.

• Use reservedMemory to monitor memory pressure.

• Combine runningDrivers with CPU metrics to spot bottlenecks.

Readiness and Liveness Probes: /v1/info & /v1/status

Sample /v1/info Response:

{
  "nodeVersion": { "version": "365" },
  "environment": "prod",
  "coordinator": true,
  "starting": false,
  "uptime": "3h45m"
}

Sample /v1/status Response:

{
  "status": "OK",
  "uptime": "3h45m"
}

Usage Recommendations:

• /v1/info → Best for readiness checks; ensure coordinator=true and not starting.

• /v1/status → Ideal for liveness checks; restart pods if unresponsive.

Security Considerations

• By default, Trino/Starburst APIs may be open inside the network.

Best practices:

• Restrict API access to trusted IPs or VPNs.

• Enable TLS at ingress/load balancer.

• Monitor API logs for unusual access.

Integrations and Tooling

• Prometheus Exporters: Scrape /v1/ui/api/stats for long-term monitoring.

• Grafana Dashboards: Visualize queries, worker counts, CPU load, and memory usage.

• Kubernetes Probes: Automate restart and readiness checks with /v1/info/state and /v1/status.

• Automation Scripts: Scale clusters or trigger alerts based on metrics.

• Incident Response: Use APIs in playbooks to assess cluster quickly during outages.

Example: Kubernetes Probes

livenessProbe:
  httpGet:
    path: /v1/status
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 15
readinessProbe:
  httpGet:
    path: /v1/info/state
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

This ensures pods only serve traffic when ready, and restart automatically when unhealthy.

Monitoring Best Practices

• Poll /v1/ui/api/stats continuously for real-time metrics.

• Watch activeWorkers as a simple cluster health indicator.

• Track query distribution (running vs queued vs blocked).

• Use uptime and version to track rolling upgrades.

Conclusion

Starburst (Trino) coordinator REST APIs provide operators with a lightweight but powerful way to monitor clusters. By integrating endpoints like /v1/info/state, /v1/ui/api/stats, /v1/info, and /v1/status into dashboards, probes, and automation pipelines, teams can:

• Detect anomalies early,

• Optimize resources,

• Automate scaling and failover,

• Maintain service reliability.

In short, these APIs transform Starburst clusters from black boxes into transparent, observable systems — a foundation for running modern, production-grade analytics at scale.

But ETL and pipeline building? That typically lived elsewhere Spark clusters, Pandas scripts, or licensed ETL tools.

Recently, while going through a set of blogs from Starburst and Lester Martin, I realised something powerful: with the new Python DataFrame APIs, even ETL and pipeline workflows can be done natively in Starburst Enterprise.

What I Learned About Python DataFrames in SEP?

SEP now supports two major Python DataFrame APIs on top of Trino:

• PyStarburst: Starburst’s native DataFrame API designed to feel familiar to PySpark users. It leverages Starburst-native extensions for performance.

• Ibis: A backend-agnostic DataFrame API that works across multiple engines, including SEP, providing portability and flexibility in hybrid data landscapes.

ETL Without Moving Data?

Instead of extracting data into external tools:

• You can read tables directly from ADLS, S3, or your warehouse into a Python DataFrame inside SEP.

• Apply transformations using intuitive Python methods like filters, joins, and group-bys.

• Save the transformed results back to object stores, registered as Hive or Glue tables.

All of this happens inside Starburst Enterprise natively , no extra data movement or cluster hops required.

Pipelines That Are Enterprise-Ready

• Lazy execution: Transformations don’t run immediately; like in Spark, they only execute when you call .collect() or save results.

• SEP’s orchestration and resource management capabilities allow scaling to production workloads.

• Integration with popular schedulers like Airflow, dbt, and CI/CD pipelines allows treating SEP as a fully-fledged pipeline engine.

This means development teams can write, test, version, and deploy ETL pipelines as regular Python projects unlocking developer discipline alongside enterprise governance.

PyStarburst vs Ibis in Enterprise Context

• PyStarburst is ideal if you are fully invested in SEP and want the best performance with Starburst-native features.

• Ibis shines if your enterprise environment involves multiple SQL engines and you want a portable DataFrame API across platforms.

Why This Stands Out

In my experience with enterprise data platforms, there’s often a fragmented ecosystem:

• SEP for federation and analytics.

• Spark clusters for ETL.

• Low-code tools (like Alteryx) for smaller jobs.

This results in complexity, silos, and governance headaches.

SEP’s Python DataFrame APIs reveal a future where:

• One engine queries, transforms, and serves pipelines.

• Governance happens in a single place, backed by Hive metastores.

• The operational burden and complexity drastically reduce.

Closing Thought

With Python DataFrames, Starburst Enterprise is evolving beyond analytics into a unified pipeline platform. ETL and transformations no longer need separate clusters or tools they can run directly on the same governed, federated platform.

For me, SEP now looks less like a query engine and more like a unified data platform , one that simplifies pipelines, governance, and analytics in a single place

References :

• Ibis & Trino (DataFrame API Part Deux)

• Ibis and Trino

• How to Build Data Transformations with Python, Ibis, and Starburst Galaxy

• PyStarburst: the DataFrame API

• Introducing Python DataFrames in Starburst Galaxy

• Building Trino Data Pipelines with SQL or Python

• Building a sql based data pipeline with trino starburst

This handbook merges platform controls, observability, and query optimization with a complete, real-world COE operating model. So you can run Starburst as a reliable product, not a best-effort cluster.

Part 1 — Platform Playbook: Healthy Starburst by Design

1.1 Core Platform Responsibilities

• Cluster configuration & lifecycle: coordinators, workers, catalogs/connectors, versioning.

• Resource management: memory guardrails, workload groups, fair scheduling.

• Reliability & elasticity: HA patterns, rolling upgrades, scaling policies.

• Observability: metrics, logs, dashboards, SLOs.

Best practice: Conduct a quarterly config review against workload patterns, compliance, and cost goals.

1.2 Stability & Fair-Use Controls (non-negotiables)

Some baseline configurations every enterprise should enforce:

1.3 Alerts That Matter

Don’t drown in noise. Focus alerts on:

• High query failure rates

• Spill-to-disk thresholds exceeded

• Cluster CPU/memory saturation

• Coordinator unavailability

• Queue times rising abnormally

Integrate with Prometheus + Grafana or cloud-native monitoring to ensure fast response.

1.4 Starburst Enterprise Built-In Insights & Logs

• Query History: SQL text, duration, resource usage, errors

• Usage Metrics: CPU/memory + workload patterns + query counts

• Cluster History: performance and execution baselines,executions, error codes, runtime stats, plans, events.

• Audit Logs (BIAC): compliance-grade access events,catalog/schema/table access, DDL.

These are your rear-view mirror and dashboard for optimization and governance.

Documentation for deeper dives:

• Backend service — Starburst Enterprise

• Insights usage metrics — Starburst Enterprise

• Insights cluster history — Starburst Enterprise

• Built-in access control audit log — Starburst Enterprise

DC&D note: Logs aren’t “admin noise.” Read them right and they’ll tell you why Finance’s dashboard fails at month-end.

Part 2 — Query Optimization in Starburst

2.1 Common Bottlenecks

• Compute: insufficient CPU/memory; overloaded workers

• I/O & network: slow external storage; high latency

• Query design: excessive scans; suboptimal join order/type

• Data layout: skewed partitions; too many small files

2.2 Diagnose with EXPLAIN / EXPLAIN ANALYZE

• EXPLAIN → planned operators and join strategies

• EXPLAIN ANALYZE → actual CPU, memory, I/O by operator; real hotspots

2.3 Proven Techniques

• Resource tuning: query.max-memory, task.concurrency, workload groups

• I/O: Parquet/ORC, compression, predicate pushdown, partition pruning

• Joins: broadcast vs distributed; reorder by selectivity; dynamic filtering

• Layout: sensible partitioning + bucketing; merge small files

2.4 Long-Term Practices

• Weekly slow-query reviews; monthly partition/layout hygiene

• Keep versions current to leverage optimizer improvements

• Prefer star schemas or selective denormalization where appropriate

• Share patterns via COE enablement (playbooks, labs)

Part 3 — The Starburst COE: Operating Model, Roles, and Runbooks (Deep Dive)

The Center of Excellence makes Starburst successful organizationally: it standardizes how teams onboard, design queries, request capacity, meet SLAs, and stay compliant. Below is a complete, practical operating model.

3.0 COE Charter (What/Why)

Mission: Accelerate safe, performant, cost-aware Starburst adoption across the enterprise.
Scope:

• Governance: policies, standards, quotas, access models

• Enablement: training, labs, office hours, champions network

• Optimization: performance clinics, workload patterns, cost reviews

• Adoption: intake, onboarding, reference architectures, use-case templates

Non-Goals:

• Building every workload for teams

• Owning line-of-business data modeling

3.1 Operating Model: Hub-and-Spoke

• Hub (COE Core): sets standards, owns platform policies, runs enablement, performs complex reviews.

• Spokes (Domain Teams): build solutions using COE guardrails; nominate Champions; own SLAs for their products.

• Shared Services: SRE, Security, FinOps, Data Governance—embedded or dotted-line to COE.

DC&D note: Centralize policy, decentralize execution. That’s how you scale without chaos.

3.2 Roles, Responsibilities & Dependencies

Key dependencies

• COE relies on Platform/SRE for telemetry and change safety.

• Domains rely on COE for patterns, templates, and escalations.

• Governance relies on COE for audit evidence (BIAC), access certs, and control attestation.

• FinOps relies on COE + Platform for accurate tagging and consumption metrics.

3.3 COE RACI for Critical Activities

A = Accountable, R = Responsible, C = Consulted, I = Informed

3.4 Intake & Onboarding (How Work Arrives)

Intake form (must-have fields)

• Use case & business owner

• Data domains & sensitivity classification

• Catalogs/connectors needed

• Expected concurrency & SLA tier (e.g., Gold/Silver/Bronze)

• Cost center tag + env (dev/test/prod)

• Required timelines; dependencies

Onboarding checklist

1. Governance classification & access patterns approved

2. Workload group & queue assignment set (SLO tier)

3. Tags/labels for cost & ownership applied

4. Observability: dashboards/alerts created

5. Enablement: users enrolled in Foundations course

6. Dry run + performance baseline captured

7. Go-live with rollback and comms plan

3.5 Policy & Governance (What “Good” Looks Like)

• Access & Identity: least-privilege by role; time-bound elevations; catalog-scoped service accounts.

• Resource Governance: standard memory caps; clear queue tiers (Gold = mission-critical; Silver = BI; Bronze = ad-hoc).

• Change Control: CAB for connector upgrades and platform versions; maintenance windows; canary strategy.

• Data Classification: P0/P1/P2 categories; mask/deny patterns by class; lineage expectations.

• Audits (BIAC): monthly review of access anomalies; quarterly attestation to compliance.

3.6 Enablement & Learning (Make Good Behavior the Default)

Curriculum paths

• Foundations (2 hrs): Starburst basics, query history, limits & queues, self-service troubleshooting.

• Performance 101 (3 hrs): EXPLAIN/ANALYZE, join strategies, partition pruning, pushdown patterns.

• Data Modeling for Starburst (2 hrs): star schemas, denorm tradeoffs, layout hygiene.

• Operator Track (2 hrs): guardrails, SLOs, incident runbooks, dashboards.

• Champion Track (ongoing): monthly clinics, deep dives, early-feature pilots.

Programs

• Office Hours (weekly), Performance Clinics (bi-weekly), Champions Network (per domain), Quarterly Bootcamp (new hires).

Artifacts

• Runbooks: “When your query is killed,” “Spill-heavy workload mitigation,” “Broadcast vs distributed joins.”

• Templates: intake form, design review checklist, exception request, performance baseline report.

3.7 Rhythm of Business (Cadence)

• Weekly Ops Review: SRE + Platform + COE → incidents, SLOs, hot spots.

• Bi-weekly Performance Clinic: COE + Domains → top slow queries; remediations.

• Monthly Architecture Board: COE + Security/Gov + Platform → connector/version changes.

• Quarterly Business Review: COE → adoption, cost, wins, roadmap.

• Quarterly Control Review: Governance → BIAC findings, access attestations.

3.8 Metrics & OKRs (Measure What Matters)

KPIs

• Reliability: % queries meeting SLA tier; coordinator uptime

• Performance: P95 runtime by workload tier; spill rate trend

• Cost: queries/$; idle capacity %; right-sizing actions closed

• Adoption: active users/teams; # certified users; champions coverage

• Governance: BIAC anomalies resolved; access reviews on time

Sample Quarterly OKRs

• O1: Improve performance without overspend.

  • KR1: Reduce P95 runtime for Silver tier by 30%

  • KR2: Cut spill events per 1k queries by 40%

  • KR3: Increase queries/$ by 20%

• O2: Raise org maturity.

  • KR1: Certify 100 analysts in Foundations

  • KR2: Establish champions in 8 domains

  • KR3: Close 100% of BIAC anomalies monthly

3.9 Exceptions & Escalations

• Exception requests (e.g., higher memory limits, Gold tier access) follow a ticketed, time-boxed approval (COE + Security + FinOps as needed).

• Escalation ladder: Domain → COE → Platform/SRE on-call → COE Head.

• Sunset: exceptions expire unless renewed with evidence (utilization + value delivered).

3.10 30-60-90 Day COE Launch Plan

Days 0–30 (Crawl)

• Form COE core; publish charter & intake form

• Baseline guardrails + workload tiers

• Stand up dashboards & core alerts

• Run “Foundations” training; open office hours

Days 31–60 (Walk)

• First Performance Clinic; remediate top 10 queries

• Pilot showback with two domains

• CAB process live; document connector upgrade path

• Champions network established in 3–5 domains

Days 61–90 (Run)

• SLA definitions per tier; publish SLO dashboards

• Formal exceptions workflow; monthly BIAC review

• Quarterly roadmap & OKRs approved; QBR cadence live

3.11 How Teams Interlock Day-to-Day (Dependency Map)

• COE ↔ Platform: COE defines guardrails; Platform implements and operates them.

• COE ↔ SRE: COE sets SLOs and KPIs; SRE measures and drives incident/runbook improvements.

• COE ↔ Domains: COE enables (templates, clinics); Domains own delivery and first-line triage.

• COE ↔ Security/Gov: COE supplies BIAC evidence; Security signs off policies and exceptions.

• COE ↔ FinOps: COE enforces tagging/showback; FinOps tracks spend and flags anomalies.

DC&D note: Clear ownership beats heroics. If two teams “sort of” own it—nobody owns it.

Part 4 — Activity Resource Tracking (ART) in Starburst: A Unified Dashboard for Performance, Governance, and Adoption

Starburst Enterprise is powerful for federated analytics, but running it at scale demands visibility: Who is using what, how much, and how efficiently?

That’s where an Activity Resource Tracking (ART) Dashboard comes in. Think of ART as your “single pane of glass” — it shows query activity, resource consumption, and governance signals in one place. Platform teams, SREs, COEs, and business stakeholders all get what they need without context-switching across half a dozen tools.

4.0 Why ART Dashboard Matters

For Platform/SRE Teams: catch bottlenecks, track node health, and respond before outages.

For COE/Governance: monitor adoption, workload fairness, and compliance (who accessed what).

For Leadership/Product: quantify value — queries per dollar, adoption growth, SLA trends.

For Analysts/Engineers: self-service view of query performance history and tuning insights.

ART isn’t just an ops tool. It’s a governance enabler and adoption amplifier.

4.1 What to Track in ART

Here’s a structured view of the most critical dimensions:

4.1.1. User & Group Activity

• Queries per user/group (counts, runtimes, frequency).

• Top N consumers (by memory, CPU, or query count).

• Active vs inactive users (adoption trendlines).

• Peak usage hours (heatmaps).

4.1.2. Query Resource Consumption

• Memory per query/per group.

• CPU usage — query-level and aggregate.

• Spill-to-disk volume per query.

• Queue wait times.

• Long-running vs short-running queries.

4.1.3. Workload Mix & Patterns

• Interactive queries vs batch pipelines.

• BI dashboards vs data engineering jobs.

• Success vs failure ratio.

• Fault-tolerant execution retries.

4.1.4. Cluster-Wide Health Metrics

• Worker utilization (CPU/memory).

• Coordinator performance.

• Autoscaling events (node add/remove).

• Queue saturation trends.

4.1.5. Governance & Compliance

• Queries against sensitive catalogs/tables.

• Access anomalies (unexpected patterns).

• BIAC Audit Logs overlay → “who changed what permissions and when.”

4.2 ART in Starburst: Built-In Capabilities

Starburst already provides much of the foundation:

4.2.1. Starburst Insights & Metrics Collection

Usage Metrics: query volume, CPU/memory usage, workload mix.

Cluster History: query executions, errors, stats, execution plans, cluster events.

4.2.2. Administrative Views

Detailed system and Insights views to track:

• Query activity per user/group.

• Background tasks (metadata refreshes, scheduled pipelines).

• Queue/workload distribution.

4.2.3. Data Collection

Telemetry includes:

• Resource usage (CPU, memory, spill-to-disk).

• Session and query runtimes.

• Background workloads (reports, ETL jobs, data products).

4.2.4. Audit & Activity Logs

The BIAC Audit Log captures:

• User logins and authentication.

• Role or permission changes.

• Catalog/schema/table access.

• DDL operations.

These logs can integrate into enterprise SIEM or compliance tooling. In short: ART in Starburst combines Insights, system metrics, and BIAC into a holistic view of consumption, performance, and governance.

4.3 How to Build the ART Dashboard

4.3.1 Data Sources

• Starburst Insights → Usage metrics + Cluster history.

• BIAC Audit Logs → Compliance-grade event capture.

• Prometheus/Grafana → Low-level node stats.

• Custom Starburst SQL Reports → System tables & Insights queries.

4.3.2 Visualization Options

• Grafana dashboards → best for platform/SRE monitoring.

• Power BI/Tableau dashboards → best for COE and business visibility.

• Custom Starburst SQL reports → lightweight, embedded, shareable.

4.4 Stakeholder Views (Dashboard Lenses)

Different roles = different lenses:

• Platform/SRE Teams → Node health, autoscaling, hot queries, failure hotspots.

• COE/Governance → Adoption growth, compliance audits, workload fairness.

• Leadership/Product → Cost-per-query, SLA adherence, business value by use case.

• Data Engineers/Analysts → Self-service query history, runtime insights, tuning guides.

4.5 Example ART Metrics to Visualize

• Top 10 users by memory consumption (last 7 days)

• % of queries spilling to disk per workload queue

• Cluster idle vs active hours (utilization efficiency)

• Gold-tier queries breaching SLA this month

• Quarterly adoption growth by team/domain

4.6 How the COE Uses ART

The Center of Excellence (COE) turns the ART dashboard into action:

• Office Hours: Walkthroughs with analysts on failed queries or long runtimes.

• Performance Clinics: Identify top 10 spill-heavy queries, remediate together.

• Quarterly Adoption Reviews: Which teams are growing? Who’s plateauing?

• Governance Evidence: Pull BIAC audit logs + usage metrics for compliance audits.

• Training Loops: If a pattern keeps reoccurring (e.g., bad join order), convert it into a training module.

Part 5— Best-Practice Summary

• Align resource controls with both technical and business needs.

• Review queueing rules & fault-tolerant execution regularly.

• Use spill strategically — not as a crutch for poor queries.

• Automate alerts on leading indicators (queue waits, spill growth).

• Keep tight communication loops among Platform, SRE, COE, Enablement, Governance.

• Train stakeholders to self-diagnose common issues and read Query History/Insights.

Conclusion

The ART Dashboard transforms Starburst observability into a strategic asset.

• For operators → it’s a health console.

• For the COE → it’s an enablement and governance tool.

• For leadership → it’s proof of adoption and ROI.

• For users → it’s feedback for faster, cleaner queries.

In DC&D spirit: Don’t just monitor Starburst. Make it accountable, governed, and optimized — with ART as your single source of truth.

Run Starburst Enterprise like a product: clear guardrails, strong observability, continuous optimization, and a COE that operationalizes good behavior at scale. With the operating model above, your platform becomes reliable for operators, fast for users, and governed for leadership.

This post was originally published on Medium blog