In this issue, I want to point you toward three sessions worth your time if you work near platforms, networks, Linux, automation, or incident response. They are connected by one theme: AI in operations is only useful when the boundaries are engineered first.

That has been the thread running through recent CloudPro issues, even when the topics looked unrelated. In the last few issues, we’ve kept coming back to the same engineering instinct: slow down, separate the moving parts, and understand what should happen before you let anything change. Whether we were talking about broken containers, MCP design, or admin rights, the point was never just the tool. It was the discipline around the tool.

AI does not remove that habit. It raises the cost of skipping it. The teams that get value from agents will not be the ones that connect the most tools. They will be the ones that know what the agent can see, what it can change, when it must stop, and how a human verifies the next step.

Taken together, these sessions give you a practical way to think about AI where it actually shows up in your work: inside the platform, inside network operations, and at the Linux command line.

FINAL CALL - Building AI-Native Platform Engineering Systems
Last 24 hours!

Start at the platform layer.

If you run an internal platform, you’ve probably felt the pressure: AI in the developer portal, agents that explain failures, golden paths that do more than point to templates.

The challenge is that platform agents need context, telemetry, policy, safe actions, approvals, and clear trust boundaries. AI can’t just be bolted on.

That’s why Building AI-Native Platform Engineering Systems stands out. It covers Backstage, OpenChoreo, CI/CD, golden paths, control planes, policy-as-code, observability, guardrails, and platform intelligence - with AI treated as part of the architecture, not an afterthought.

A solid read for platform engineers, DevOps engineers, SREs, architects, and engineering leaders asking:

“Can we make our platform AI-native?”

The answer may be yes, but only with the right control surfaces.

Use code FINAL50 to get 50% off

BOOK YOUR SEAT NOW

The session is on June 11, i.e., 24 hours from now, and you can use the discount code FINAL50 when booking.

Engineering Agentic Network Operations

Once platform boundaries are in place, the next challenge is network operations, where many AI demos fall apart.

It’s easy to build an agent that summarizes data or suggests a config. It’s much harder to build one that works within real constraints, recovers from failures, and knows when to hand control back to a human.

That’s the focus of Engineering Agentic Network Operations.

The workshop covers MCP, OpenClaw, NetClaw, FastMCP, Containerlab, Arista cEOS, and more. But the real value is learning how to design agentic workflows with clear scope, guardrails, recovery paths, and human oversight.

For network automation, SRE, platform, and infrastructure teams, those questions matter long before an agent is trusted with production operations.

Use code SPECIAL50 to get 50% off

BOOK YOUR SEAT NOW

Agentic Linux – From Commands to AI Agents

Then there’s the base layer.

No matter how advanced the platform, a lot of real work still ends up in Linux - reading logs, checking services, fixing permissions, troubleshooting errors, and deciding whether a generated command is actually safe to run.

That’s why Agentic Linux – From Commands to AI Agents is a useful third piece.

The workshop covers ChatGPT CLI, Shell Genie, Aider, Goose CLI, Bash scripting, troubleshooting, automation, and system administration tasks. What I like is that it treats AI as an assistant, not a replacement for judgment.

Generating a command is easy. Knowing whether it’s safe, spotting destructive actions before they run, and keeping humans in the approval loop is the hard part.

For Linux administrators, DevOps engineers, SREs, and IT professionals, it’s a practical look at using AI at the command line without turning the shell into a roulette wheel.

Use code EARLY40 to get 40% off

BOOK YOUR SEAT NOW

A few books to keep beside these sessions

One more thing worth mentioning: these sessions pair well with a few books from our list if you want to go deeper after the labs.

For the first workshop, I’d start with Platform Engineering for Architects, which is especially relevant because attendees of Building AI-Native Platform Engineering Systems get the book for free. Mastering Enterprise Platform Engineering is the natural follow-up if you’re thinking about platform engineering, delivery workflows, and generative AI at enterprise scale. For the network operations side, AI Networking Cookbook fits neatly with the agentic network operations session. And if the Linux workshop is closer to your day-to-day work,The Ultimate AI Guide for Linux Engineers is a good companions for thinking about what AI-generated commands are actually touching underneath.

The Ultimate AI Guide for Linux Engineers

BUY NOW ON AMAZON

Mastering Enterprise Platform Engineering

BUY NOW ON AMAZON

AI Networking Cookbook

BUY NOW ON AMAZON

Platform Engineering for Architects

BUY NOW ON AMAZON

The useful question is not whether AI can be added to these workflows. It can, and most teams are already experimenting with it somewhere. The harder question is whether the workflow still makes sense once AI is inside it. Can the team understand what happened? Can they review the output? Can they stop a bad change before it lands? Can they trust the system more after adding AI, not less?

That is the thread running through all three sessions. They are not about chasing the newest tool. They are about taking the work many of us already do across platforms, networks, and Linux environments, and asking what has to change when AI becomes part of that work.

That is what makes these sessions useful. They are not abstract AI talks. They sit close to the places many of us actually work: the platform layer, network operations, and the Linux command line.

I’ll be in attendance as well, because these are exactly the conversations CloudPro needs to keep having: what is useful, what is unsafe, what is ready, and what still needs a human in the loop.

This is time-sensitive. The first session is on June 11, and the other two follow later this month. If one of these areas is on your roadmap, I would not skip it.

Cheers,

Apramit

Editor-in-Chief

Local intune admin rights are one of those problems every Windows shop runs into eventually, and EPM is usually the first answer that comes up. But how you roll it out, and who you actually license for it, makes a real difference to both your security posture and your budget. Today’s CloudPro issue is adapted from Mastering Endpoint Management using Microsoft Intune Suite by Saurabh Sarkar and Rahul Singh, and it lays out a clean way to think through the decision.

Cheers,

Shreyans Singh

Editor-in-Chief

Subscribe now

Share

CloudPro #124: Most “I Need Intune Admin Rights” Requests Aren’t About Admin Rights

Every security team eventually gets there: local admin rights have to go. Fair enough- they’re a known weakness, and leaving users as admins on their own machines isn’t defensible anymore. So you start looking at EPM, and the easiest thing to do is buy licenses for everyone, turn on self-elevation, and call it solved. Pause before you do that. You’ll spend more than you need to and you’ll be using the tool for something it isn’t really designed for.

EPM gets pitched, and often understood, as “the safe way to give users admin rights.” That framing is the source of most of the trouble. EPM isn’t an admin-rights replacement and it isn’t an app delivery mechanism. It’s elevation control. It lets a specific application run with admin privileges in a specific moment, under a rule you’ve defined. That’s a much narrower job than “make this user an admin,” and once you see the difference, the licensing decision gets a lot easier.

Start with app delivery, because most “I need admin” requests aren’t actually about admin. They’re about getting an app installed. If you push your common business apps via Intune as Required, they install in the system context and the user doesn’t need elevation at all. For the long tail of apps where you’re not sure who needs what, make them Available through Company Portal. The user installs them on demand, still in system context, still without elevation. Get this layer right and a huge chunk of the supposed “need for admin rights” disappears.

What’s left after that is the actual EPM territory. Someone needs to install something niche that isn’t in your catalog and never will be. A support engineer needs to run ProcMon elevated to debug a real issue. A developer needs an elevated PowerShell window. A user needs to restart a stuck service. These are the cases EPM was built for: controlled, rule-based elevation for specific apps and specific moments. Worth assigning a license for. Worth setting up properly.

One distinction worth being clean about: EPM is not application control. If your goal is to stop certain apps from running on your devices, EPM doesn’t do that. It decides what gets elevated, not what gets to run in the first place. App Control for Business is the right tool for that, and conflating the two leads to policies that don’t do what you think they do.

Which brings the licensing piece into focus. EPM licensing is per-user, and the population that genuinely runs elevated workloads is almost always a fraction of your fleet: engineers, IT support, certain power users. The rest of your users get their apps through Required and Available and never hit an elevation prompt. Licensing everyone “just in case” is the most common way teams overspend on the Intune Suite, and it usually happens because the team skipped the app-delivery question and went straight to the elevation question.

Get the delivery layer right first. EPM is for what’s left over.

Subscribe now

Share

This matters because once agents start relying on your MCP server, the contract gets sticky. You can change the implementation, but renaming tools or restructuring scopes breaks every instruction file and agentic workflow already pointing at them. The first version tends to become the permanent version.

Subscribe now

Rule one: don’t mix read and write.

It’s tempting to expose everything from a single server: get_logs and list_problems alongside the handy create_workflow and send_notification stuff. Don’t. Put read-only data access in one server and anything that modifies state or triggers actions in a second, separately installed one.

Two reasons. The obvious one is accidental writes. The less obvious but bigger one is prompt injection. If an agent is connected to both, a poisoned log line or a malicious doc in your RAG pipeline can talk the agent into calling an admin action it had no business calling. Splitting servers shrinks that attack surface. And yes, the MCP spec lets users disable individual tools, but realistically most people leave everything on.

Rule two: don’t ship only a generic execute_query tool

Every observability backend has a query language: NRQL, DQL, PromQL, whatever yours is. You could expose just execute_query and let the agent figure out the syntax. It’ll work, but badly. The agent guesses, gets a syntax error, retries, refines, retries again. Every round-trip costs API calls, tokens, and latency.

Build purpose-specific tools for your top use cases alongside the generic one. A get_logs tool taking a timespan and workload identifier will run a clean, optimized query on the first try. No guessing.

Don’t overcorrect though. get_logs_from_k8s, get_logs_from_hosts, get_logs_from_apps: now you’re maintaining ten tools and the agent picks the wrong one half the time anyway. Aim for the middle.

Rule three: validate locally before hitting the backend

When the agent sends a query, check it inside the MCP server first. Is the syntax valid? Does the caller actually have permission to access that data? These are cheap checks and they kill the trial-and-error loop at the source. Without them, every bad query becomes a billable backend call.

Instrument your MCP server and you’ll see the pattern fast: most failures are the same two or three mistakes. Catch them locally.

These aren’t the only mistakes you can make, but they’re the ones that compound quietly until you’re rebuilding the server at version two. Worth getting right the first time.

Subscribe now

This article was adapted from Observability in the AI-Native Era.

Cheers,

Shreyans Singh

Editor-in-Chief

Share

But they’ve been officially deprecated since OpenShift 4.14, and it’s worth understanding why. Not just because Red Hat says so, but because the underlying technology has genuinely moved on.

Some context. When OpenShift 3 shipped back in 2015, Kubernetes didn’t have great deployment management. DeploymentConfig filled that gap: lifecycle hooks, image change triggers, custom rollout strategies. It was ahead of its time, honestly.

But Kubernetes caught up. Deployments now do automated rollbacks, HPA-based autoscaling, pause and resume during rollouts. And they’re built on ReplicaSets, not ReplicationControllers. That bit matters because ReplicationController development has stopped upstream entirely. The foundation DeploymentConfig sits on isn’t getting any love anymore.

There’s a design difference worth knowing about too. DeploymentConfig leans toward consistency: if the node running your deployer pod dies, it just waits. Waits for the node to recover, or for someone to manually step in. Deployments lean toward availability: the controller manager runs across multiple masters, so another one picks up the work. In production, you usually want that.

The other thing is portability. DeploymentConfigs only exist in OpenShift. Deployments are standard Kubernetes. If you ever need to move workloads between clusters or providers, that distinction starts to matter a lot.

And by sticking with DeploymentConfig, you’re also cutting yourself off from tooling. No Argo Rollouts for canary or blue-green. No native HPA. No automated rollback. Manual scaling, manual rollback. It’s fine until you’re doing it at 2am during an incident and wishing you weren’t.

Nobody’s flipping a switch on you tomorrow. DeploymentConfigs still run. But “it still works” isn’t really a strategy. If you haven’t started thinking about this, now’s a good time.

This article was adapted from Learn OpenShift.

I’ve seen engineers waste hours on this because they defaulted to one or the other without thinking it through. So here’s how I think about it.

It comes down to one question: is the problem in how the container is running, or in what was built?

Subscribe now

If it’s a runtime issue: the app got stuck, a config change hasn’t taken effect, something just needs a kick, restart. That’s the fastest path back to normal and there’s nothing wrong with it.

If the image itself might be the problem, restart won’t help. You’ll just redeploy the same broken thing. You need to rebuild.

Sounds simple, but the tricky part is knowing which situation you’re in. Here’s how I usually work through it.

Config, env vars, or secrets changed? Restart the service. New values get picked up at runtime. No need for a new image.

Base image tag or digest changed? Rebuild. Even if your code is the same, the stuff underneath it isn’t. Your dependencies have moved.

You built with latest or your Dockerfile pulls in dynamic dependencies? Rebuild. You don’t really know what’s inside that image anymore. Something upstream could have shifted without anyone noticing.

Application code changed? Obviously rebuild. New code needs a new artifact.

Container is misbehaving but the image checks out- tag is right, digest matches, nothing has moved upstream? Restart. The image is fine, the container just needs a fresh start.

Not sure what version is even running? Don’t do anything yet. Inspect first. Check your image labels, check the digest. Then decide. Guessing under pressure is how you make things worse.

One small thing that helps a lot: bake a build timestamp into every image. A label, an env var, whatever. It takes seconds to add during the build and it means you can always verify what’s actually running where. Saves a ton of time when you’re half-awake and troubleshooting.

Here’s the whole thing as a quick reference:

Subscribe now

This article is adapted from Mastering Docker on Windows by Michael D. Smith.

In the article below, Stéphane tackles one of the most challenging aspects of Azure architecture: building truly resilient multi-region systems, with concrete examples using Azure SQL, Cosmos DB, and Azure Storage, complete with code samples and Terraform scripts you can adapt for your own DR testing.

Happy reading!

- Shreyans Singh

Editor-in-Chief

Subscribe now

I like to say that Azure is simple… until you go multi-region. The transition from a well-designed single-region architecture to a truly resilient multi-region setup is where simplicity gives way to nuance. Concepts that seemed abstract (high availability versus disaster recovery, failover semantics, DNS behavior, data replication guarantees) suddenly become very real, very concrete, and sometimes painfully operational.

This article is written for architects and senior platform engineers, who already understand the fundamentals but are required to build solutions that must remain available despite regional outages, service failures, or infrastructure-level incidents. The scope is intentionally narrowed to Recovery Time Objective (RTO). Data corruption, ransomware, and backup-based recovery are explicitly out of scope. Instead, the focus is on how applications and data services behave during live failover scenarios, and how architectural decisions, sometimes subtle ones, can make the difference between a seamless transition and a prolonged outage.

Through concrete examples using Azure SQL, Cosmos DB, and Azure Storage, this article explores how replication models, DNS design, private endpoints, and SDK behavior interact at runtime, and what architects must do to ensure their applications remain functional when regions fail.

Rather than focusing on theoretical patterns, the goal here is pragmatic—minimizing downtime and operational friction when things do go wrong. You’ll see diagrams, Terraform and deployment scripts, plus .NET code samples you can adapt for your own DR tests and game days.

Before getting into the details, let’s briefly revisit the difference between high availability (HA) and disaster recovery (DR).

Subscribe now

HA and DR exist on a spectrum, with increasing levels of resilience depending on the type of failure you want to withstand:

• Application-level failures: In some cases, you may simply want to tolerate application bugs—for example, a memory leak introduced by developers. Running multiple instances of the application on separate virtual machines, even on the same physical host, can already prevent a full outage when one instance exhausts its allocated memory. That is for instance, what you would get if you spin up 2 instances of an Azure App Service within the same zone (no zone redundancy).

• Hardware failures: To handle hardware failures, workloads should be distributed across multiple racks. That is what you would get if you’d host virtual machines on availability sets.

• Data centre–level outages: To withstand more severe incidents, workloads should be spread across multiple data centers, such as by deploying them across multiple availability zones. You can achieve this by turning on zone-redundancy on Azure App Service or use zone-redundant node pools in AKS. With such a setup, you should survive a local disaster such as fire, flooding, etc.

• Regional outages: Finally, to survive major outages, such as a major earthquake, a country-level power supply issue, etc., workloads must be deployed across geographically distant data centers. You can achieve this by deploying workloads across multiple Azure regions in active/active or active/passive mode.

Looking at Azure SQL

Let’s first analyse the different data replication possibilities with Azure SQL. Table 1 summarizes the different capabilities.

Table 1 – Replication capabilities

We’ll set aside named replicas and geo-restore, as the former does not contribute to disaster recovery and the latter is likely to introduce significant downtime and potential data loss. This leaves geo-replication as the remaining option. As you might have understood by now, using Azure SQL’s built-in capabilities, you cannot achieve a full ACTIVE/ACTIVE setup since it doesn’t support multi-region writes. This means that you can only have one read-write region and the secondary region(s) are read only.

Table 2 outlines the two available geo-replication techniques.

Table 2 – Geo replication options

Active geo-replication may require updates to connection strings or DNS records to point to the new primary after a failover. That said, the actual impact depends on where (*) the client application is located as well as how you deploy to both regions. Let’s look at this in more detail. Figure 1 illustrates an active geo-replication setup between Belgium Central and France Central.

Figure 1 – SQL geo replication with active geo replication

In such a setup, under normal circumstances:

• Workloads in the primary region (Belgium Central) can connect to the primary server in read/write mode

• Workloads in the primary region can perform read-only activities against the secondary replica, providing they tolerate the extra latency incurred by the roundtrip to the remote region (France Central).

• Workloads in the secondary region (if any), can perform read-only operations against the read replica with no extra latency.

Subscribe now

The configuration shown in Figure 1 supports a database-only failover. Both regions expose private endpoints to both SQL servers and rely on region-scoped DNS zones.

Although Private DNS zones are global by design, keeping them regional allows each region to resolve both the primary and secondary servers. This requires four DNS records in total—primary and secondary endpoints registered in each regional zone.

With a single shared DNS zone, this would not be possible: while all four private endpoints could be deployed, only two DNS records would be registered, since the endpoints map to just two FQDNs (primary and secondary). While this approach works, it keeps the regions siloed and prevents any cross-region traffic. From a resilience standpoint, it is preferable to provide as many fallback paths as possible.

Moreover, as we will see later, with other resources such as Storage Accounts, a single DNS zone would force us to update the DNS records upon failover, causing a minimal downtime. Bottom line: using multiple DNS zones prevents issues during failover.

Back to active geo replication! In case of failover, SQL servers switch roles: the primary becomes secondary and vice versa. This concretely means that the connection string primary.database.windows.net targets the read/write region in a normal situation but a read-only or unavailable one after failover. Workloads using this connection string would either stop working (if the regional outage persist), either talk to a read-only database instead of a read-write one, once the failover completed. Similarly, the connection string secondary.database.windows.net usually targeting the read-only region under normal circumstances now targets the read-write one after failover.

Knowing this, a few options exist:

• You may choose to fail over everything (database+compute). In that scenario, workloads running in the secondary region can use their default secondary connection string, which will automatically target the new primary after failover. This approach requires the deployment pipeline to be region-aware, detect the target region, and apply the appropriate connection string. When deployed in the primary region, the application should use primary.database.windows.net, while in the secondary region it should already be configured with secondary.database.windows.net. This design eliminates the need for any connection string changes after failover. If your webapps, K8s pods, etc. are already up and running, the only thing you still have to do is route traffic to them. Any other SQL client not running in the secondary region (eg: on-premises), would have to update its connection string to target the new primary.

• You may choose to redeploy the compute infrastructure (web apps, etc.) to the secondary region only in case of regional outage. This approach is cheaper but risky as you’re not guaranteed to have the available capacity and it is causing a significant downtime. However, such an approach allows you to adjust your pipelines, specify the right connection string and simply redeploy your infrastructure and/or application package.
  

• If you want to deploy the application with the exact same settings in both regions, you’ll need to update the connection string used by workloads in the secondary region, since primary.database.windows.net will now resolve to an unavailable server after failover. If the original primary later comes back online, it will return as a secondary (read-only) replica, which would not support write operations. You can as well make your application failover aware (**).
  

You can’t simply update DNS, meaning making secondary target primary and vice versa, because the FQDN (primary-or-secondary.database.windows.net) is validated by the target server, and the names must match—so redirecting it to a different server would simply fail.

In conclusion, when using active geo-replication as the replication technique, you should make your applications failover-aware (**) and pre-provision both connection strings and implement the failover/retry logic in the application code itself. You may wrap your Entity Framework context into a factory to abstract away the retry logic. Given we typically use a scoped lifetime, you may expect some HTTP requests to fail (in case of an API) but new instances targeting the right server would ultimately succeed without having to restart the application. You may as well use a geo-redundant Azure App Configuration and failover it along SQL, then switch the primary server connection string after failover. The SDK allows you to monitor a sentinel key and to reload the configuration without having to restart the application:

configBuilder.AddAzureAppConfiguration(options =>

{

options.Connect(new Uri(”<your-app-config-endpoint>”), new DefaultAzureCredential())

// Load all keys that start with `TestApp:` and have no label

.Select(keyFilter: “TestApp:*”, labelFilter: LabelFilter.Null)

.ConfigureRefresh(refreshOptions =>

{

// Trigger full configuration refresh only if the `SentinelKey` changes.

refreshOptions.Register(”SentinelKey”, refreshAll: true);

});

});

More on App Configuration and the sentinel key https://learn.microsoft.com/en-us/azure/azure-app-configuration/howto-best-practices?tabs=dotnet

Upon failover, you would update the sentinel key, modify connection strings in App Configuration or tell the apps that they failed over, letting them pick the new configuration without the need to restart them.

As an alternative to active geo-replication, you can use failover groups, which allow one or more databases to fail over together to another server. A key benefit of failover group compared to active geo-replication, is the presence of two stable listeners: one for read-only traffic and one for read/write operations. These listeners remain constant and always point to the current primary and secondary servers. During a failover, applications continue connecting through the listeners rather than connecting directly to a specific SQL server, eliminating the need for connection string changes.

Figure 2 illustrates the situation when the primary region is working fine.

Figure 2 – Failover group before failover

We see that clients connect to listener endpoints. The failover group comes with two endpoints:

• <failovergroupname>.database.windows.net

• <failovergroupname>.secondary.database.windows.net

These public DNS records are managed by Microsoft and they work with aliases pointing to the replicated servers. The read-write endpoint will always target the server that has the primary role. In Figure 2, it currently points to primary.database.windows.net, our server in Belgium Central.

Figure 3 shows the situation after failover

Figure 3 – Failover group after failover

We see that this time, the read/write endpoint points to secondary.database.windows.net, our server in France Central, which has become the new primary. Failover groups greatly simplify client connectivity: when private endpoints and DNS zones are properly configured, the failover is completely transparent to connected clients.

To help you test this, I have created a GitHub repo https://github.com/stephaneey/azure-and-k8s-architecture/tree/main/availability-samples/sql where you can find both a sample console application and some Terraform code. The solution is a simplified setup but let’s you grasp how failover groups work. You can find the explanations on how to deploy and test this in the repo itself.

Next, let’s look at how to maximize availability using Cosmos DB.

Looking at Cosmos DB

As of January 2026, Cosmos DB is the only Azure service that supports true active/active read-write deployments through its multi-region writes capability. For scenarios that do not rely on relational models or strict referential integrity—and where eventual consistency (most commonly session consistency) is acceptable—Cosmos DB is the preferred choice.

Figure 4 represents an API platform spanning two continents (Europe and US) and leveraging the multi-region writes feature.

Figure 4 – Leveraging Cosmos DB multi-region writes for a global API platform

Note 1: the raw diagram, extracted from my book, is available here https://github.com/PacktPublishing/The-Azure-Cloud-Native-Architecture-Mapbook-Second-Edition/blob/main/Chapter03/diagrams/diagrams.vsdx

Note 2: I delivered a 25-minute live demo of the above setup during a Microsoft API Management event (

starting at minute 53 of the recording). Microsoft slightly fast-forwarded the video, so the pace is a bit quicker, but it remains possible to follow if you stay focused 😊.

In a nutshell, US callers are sent to the US backend and EU callers to the EU backend. What is interesting in this architecture is that every layer is failure-resistant.

The entry point, Front Door uses geo-proximity by default but is able to send US callers to EU and vice versa in case the regional backend is unhealthy. API Management’s regional gateways forward to their regional backends.

Each of the backends hosted on app services is leveraging the Cosmos DB SDK, which is smart enough to detect whether the corresponding Cosmos DB region is available or not. The API will seamlessly switch to the available region in case of a regional failure impacting only Cosmos DB. That’s what I’m going to focus on in this article.

Figure 5 focuses on the backend part.

Figure 5 – focusing on backend

My US and EU app services talk respectively to their regional Cosmos DB endpoint under normal circumstances. Note that each region has three private endpoints (one for each region plus the non-regional endpoint), which makes it possible from EU to go to the US and vice versa through the private endpoint plumbing. If, for some reason, the EU backend cannot talk to the EU Cosmos endpoint, it will take the US path. Of course, there is an impact on latency but the solution keeps working and the application doesn’t even need to be restarted. In case of a full regional outage (say entire EU is down), Front Door would direct all API calls to the US backend.

This architecture can resist to both local outages and full regional ones.

To help you test this built-in resilience using Cosmos, I have developed a sample API along with the Terraform bits that deploy a simplified setup shown in Figure 6. The repo is available here

https://github.com/stephaneey/azure-and-k8s-architecture/tree/main/availability-samples/cosmos

Figure 6 – Simplified setup for testing

Here, I’m not using private endpoints but this is enough to understand the SDK mechanisms. After you have deployed the solution, you should end up with the following resources:

Figure 7 – Resources deployed by the sample application

I made sure the webapp’s managed identities are granted access to Cosmos and I pre-deployed a database and a container.

Additionally, each web app indicates its preferred region using environment variables to the Cosmos SDK:

Figure 8 – Cosmos DB settings defined as environment variables

You can directly test the APIs using Postman or Fiddler, by first performing a POST request (Figure 9), to create a document, followed by a GET request (Figure 10):

Figure 9 – Sample POST request to create a document

Figure 10 – Sample GET request to get the latest weather forecast

I developed the API in such a way that it indicates which Cosmos region was used by the backend service to perform the operation.

Now, you can start playing with Cosmos to see how both APIs react. Here are the following things you can test using the Azure Portal or Azure CLI as you wish:

• Marking one of the Cosmos regions read only. In such a case, both APIs (Belgium and France) should automatically go to the remaining writable region, but reads (GET queries) would still pick their regional instance.

• Deleting one of the Cosmos regions. In such a case, both APIs should automatically go to the remaining region for both reads and writes

• Take a Cosmos region offline. In such a case, an automatic failover should take place to the remaining region. Note that you must create a support ticket to get it back online. Just try this one after having tried the other changes.

Make sure to run both POST and GET request between each steps and check which Cosmos backend instance is used by the API. You’ll notice that whatever you do at Cosmos level, the backend is smart enough to keep working.

All the required instructions are provided in the repo.

Last but not least, let’s turn to Azure Storage.

Looking at Azure Storage

Let’s look at Azure Storage, and more specifically at Blob Storage, Table Storage, and Queue Storage, which are widely used across all types of applications. A very common scenario involves Azure Durable Functions as well as stateful Logic Apps, whose default state store is Azure Storage. Because the orchestration state is persisted in Azure Storage, it is essential to understand what happens to this state in the event of a failover.

For sake of brevity, I will only show how to handle both the blob and table sub resources but the principles are exactly the same with the queue service.

Here are the replication capabilities of Azure Storage.

All the above options are only available in paired regions. Additionally, blobs can also be replicated using the object replication feature, but this is by no means comparable to the above options.

Azure Storage replication is rather easy when dealing with internet facing Storage Accounts and public DNS but it gets a bit more complex when using private link.

A common misunderstanding is the impact of using a single DNS zone for private link, which would automatically incur additional downtime in case of regional outage and storage failover.

Figure 11 illustrates the problem:

Figure 11 – Shared DNS zones

Our shared DNS zones are attached to both virtual networks. From the primary region, I can have two private endpoints, one targeting the primary in West Europe and the other one targeting the secondary in North Europe. Both private endpoints are located in West Europe.

Because the FQDN of the Storage Account is mystorage.<service>.core.windows.net, I can only have one record at a time in the DNS zone. I could pre-create the private endpoints in the secondary region but not register them in the zone. In case of Storage Account failover, I would need to update the DNS records and put the IPs of the private endpoints in North Europe. Even doing this wouldn’t work:

Figure 12 – Suboptimal setup using shared DNS zone

With a setup as shown in Figure 12, if I have one blob “test.txt” in the “test” container, the resulting URIs will be:

• https://mystorage.blob.core.windows.net/test/test.txt (primary)

• https://mystorage-secondary.blob.core.windows.net/test/test.txt (secondary)

The primary URL will be reachable from the VM located in the West Europe VNET but not from the one located in North Europe since mystorage.blob.core.windows.net resolves to 10.0.0.4, meaning the private endpoint of West Europe, which is out of reach from North Europe (in this setup). Of course, you could peer those VNETs or route traffic through hubs but this is typically not what you’d want.

Moreover, after failover, you’d be in a situation as shown in Figure 13.

Figure 13 – Storage account after failover

The primary (West Europe) is gone. Perhaps the entire region is gone, meaning that 10.0.0.4 is lost, so even though you would have routed traffic initially from North Europe to West Europe, it’d stop working.

You’d be left with 10.100.0.4 still targeting the secondary but after failover, there is no more secondary...The failover process converts the Storage Account into a Locally Redundant Storage (LRS), so the notions of primary and secondary do not exist anymore. The North Europe has just become the new primary, making 10.100.0.4 now pointing to an unreachable target.

The only way to restore access to https://mystorage.blob.core.windows.net/test/test.txt from North Europe is to create a new private endpoint and update the DNS zone accordingly. You could pre-create the private endpoint prior to a failover but in any case, updating the DNS zone would be required. This would cause an additional short downtime and requires a scripted plan.

So, with Storage Accounts, you must use separate DNS zones if you don’t want to perform any intervention.

That is why, the best setup is the one shown in Figure 14:

Figure 14 – Best setup with Azure Storage using separate DNS zones

Each region has its own private DNS zone and each holds a set of private endpoints targeting both primary and secondary. Now, both mystorage and mystorage-secondary are resolved and routable from both sides. From North Europe, a call to mystorage would go through 10.100.0.4 and hit the West Europe location.

After full regional outage, West Europe is lost but 10.100.0.4 is still targeting the primary, which means, the promoted North Europe location, while the secondary endpoint is lost again (this is unavoidable). This setup doesn’t require any intervention other than performing the failover itself. Needless to mention that your application will encounter exceptions while Azure Storage is failing over, so you need to make sure to correctly handle exceptions.

Coming back to a more concrete example with Durable Functions, you could have an active/passive setup like this one:

Figure 15 – Adding the orchestrator to the mix

Where your orchestrator, hosted on an App Service levering VNET integration is getting access to the Storage Account through 10.0.0.4 to persist state. The North Europe version of the orchestrator could already be deployed but stopped to avoid possible interferences.

In case of regional outage or DR test, you failover the Storage Account and once the failover completed, you start the orchestrator in North Europe that now points to the new primary. Durable functions make mostly use of Queues and Tables to work and persist state. Once the failover completed, you should be able to resume ongoing orchestrations. However, since the data replication process is asynchronous, you may lose some of them.

To help you test this scenario, I have crafted a very little Durable Function available here https://github.com/stephaneey/azure-and-k8s-architecture/tree/main/availability-samples/storage that is just waiting for an external event:

public static async Task RunOrchestrator(

[OrchestrationTrigger] TaskOrchestrationContext context)

{

ILogger logger = context.CreateReplaySafeLogger(nameof(Function1));

logger.LogInformation(”Waiting for external event”);

var eventData=await context.WaitForExternalEvent<DateTime>(”event”);

logger.LogInformation(”external event received {0}”, eventData.ToString());

}

The orchestration is started by this HTTP triggered function:

public static async Task<HttpResponseData> HttpStart(

[HttpTrigger(AuthorizationLevel.Anonymous, “get”, “post”)] HttpRequestData req,

[DurableClient] DurableTaskClient client,

FunctionContext executionContext)

{

ILogger logger = executionContext.GetLogger(”Function1_HttpStart”);

// Function input comes from the request content.

string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(

nameof(Function1));

logger.LogInformation(”Started orchestration with ID = ‘{instanceId}’.”, instanceId);

return await client.CreateCheckStatusResponseAsync(req, instanceId);

}

The idea is to start a new orchestration by calling the HTTP triggered function:

Figure 16 – Starting a new orchestration

And get the instance ID of the orchestration for later use.

After this, you can initiate the failover. Once completed, you should raise the external event and see if the orchestrator is able to resume this orchestration:

Figure 17 – Raising the external event

If everything works fine. The orchestration status should show as completed:

Figure 18 – Checking the orchestration status

You can test this easily locally by making sure to:

• Use a GRS/GZRS/RA-GRS/RA-GZRS Storage Account

• Bind your local solution to the Storage Account by assigning the connection string of your Storage Account to the AzureWebJobsStorage setting in local.settings.json

Of course, in this case, we use public DNS and an Internet facing Function App, which is way simpler than the enterprise-grade design I depicted earlier. This is however enough to experiment with a concrete use case.

Let’s wrap it up!

Designing for resilience in Azure is less about individual services and more about underlying foundations such as networking and DNS. Working with single or multiple DNS zones already has an impact on downtime.

As we have seen, Azure services offer powerful built-in capabilities, but they differ from one service to another and we’ve only seen three of them. Many other services have yet, a different behaviour.

A recurring theme across all scenarios is that infrastructure-level redundancy is not enough. Applications must be designed to tolerate transient failures, retry intelligently, and adapt to changing backend roles, especially in ACTIVE/ACTIVE configurations.

Ultimately, resilience is not achieved by flipping a single switch. It is the result of deliberate architectural choices, tested failure paths, and an honest acceptance of trade-offs—latency versus availability, consistency versus survivability, cost versus operational simplicity.

If there is one takeaway, it is this: multi-region architectures must be designed for failover from day one. When they are, regional outages become operational events rather than existential crises—and “always-on” stops being an aspiration and starts becoming an outcome.

And in today’s article, he explores a radical idea: AI that runs entirely offline. No APIs, no data leaving your network. Just private, local intelligence built for sensitive environments. Sounds interesting? Read on for the full article.

If you want to learn directly from him, Saurabh is hosting a live AWS Solutions Architect Associate (SAA-C03) Workshop on January 17. Its a hands-on, fast-paced session that strips the exam down to what really matters. CloudPro readers get an exclusive 40% early-bird discount with the code CLOUDPRO. Reserve your seat.

Cheers,

Shreyans Singh

Editor-in-Chief

Early Bird Offer: Get 40% Off

Use code CLOUDPRO

AI That Runs Entirely Offline: How to Build an Offline Enterprise Assistant

By Saurabh Shrivastava

Working in defense, finance, law, or a heavily regulated industry means you can’t just plug into ChatGPT and call it a day. Cloud-based AI tools aren’t built for environments where data leakage isn’t just bad.

It’s catastrophic.

You can’t send classified intel or proprietary financial models to someone else’s servers. And if you’re operating in an air-gapped network? Forget about it.

That’s the problem this Offline Enterprise Assistant solves.

It’s a local AI setup that runs entirely on your own hardware. No cloud dependencies. No API keys. No data leaving your perimeter. You choose a model: LlamaCpp, Ollama, whatever fits your needs, and run it directly on your machine. Every prompt, every response, every log file stays inside your infrastructure.

This matters when you’re reviewing sensitive legal contracts, running R&D analyses, or automating workflows that involve confidential information. You get the productivity boost of modern AI without opening the door to external risk. It’s built for teams that need full control over their tools and can’t afford to trust a third party with their data.

Why This Architecture Stands Out

1. Runs Without Internet: Operates 100% offline, making it ideal for air-gapped networks or classified infrastructure.

2. Keeps Data on Your Device: Nothing is sent out, nothing is tracked. You stay in control always.

3. Fast and Responsive: Local inference means no lag, no rate limits, just amazing performance.

4. Built for Sensitive Workflows: Legal reviews, research, compliance, internal tooling are all handled securely.

Most teams are realizing that AI doesn’t always belong in the cloud. When you’re dealing with internal systems, sensitive data, or strict compliance rules, you need something that stays inside your walls. That’s where a local-first approach makes sense: it gives you the benefits of AI without the exposure.

This Offline Enterprise Assistant is built around that idea. It’s your own assistant, running entirely on your hardware, tuned to your environment, and never sending a single request outside your network. You control how it works, how it’s updated, and what data it touches.

Let’s break down how the architecture fits together.

Architecture Explanation

The offline MCP Client architecture is designed to deliver end‑to‑end private and local AI capability, without any reliance on cloud APIs or outbound network traffic. Here’s how it works:

1. Developer: Prepares prompts or workflows using a local development environment (such as a secure IDE or terminal). All interactions originate and remain on the local device.

2. MCP Client: Acts as the interface between the developer’s inputs and the AI model. It routes prompts to the embedded LLM, orchestrates the workflow, and handles results.

3. Offline LLMs (LlamaCpp / Ollama): Powerful large language models are loaded and executed directly on the local hardware. No external API calls; all model inference and response generation happen on the device, fully offline.

4. Local SQLite Database: Stores chat logs, prompts, and results securely and privately. Provides an audit trail and the ability to revisit past interactions, entirely within the local infrastructure.

5. Secure UI/API: Presents results to the developer via a local web interface or terminal UI. Enables further integration with internal systems while ensuring data never leaves the trusted environment.

Think about it. You don’t want your data, your prompts, or your workflows slipping out into the cloud. With this architecture, nothing leaves your machine.

Zero external exposure. No tokens. No API keys. No hidden traffic.

If you’re in a regulated industry, whether it’s defense, legal, healthcare, or any air-gapped environment, this setup checks every box. It keeps you compliant, private, and secure while still giving you the power of modern AI. And here’s the best part: it’s extensible by design.

Want to add another LLM? Done.

Need to customize workflows? Easy.

Ready to experiment with agentic AI? Go ahead. You can build without ever breaking the privacy barrier.

Most importantly, this isn’t a short-term solution. It’s future-proof. As on-device AI models become larger and smarter, this architecture will scale with you, handling more automation, more intelligence, and more complexity.

Now it’s time to get our hands dirty and implement it.

Implementation

Using LM Studio, Streamlit, and Python, you’ll set up and run local open-source models directly on your machine. Unlike online AI assistants like ChatGPT or Google Bard, which constantly need internet connectivity and send data back to external servers, this approach runs completely offline.

Along the way, you’ll gain hands-on experience with the full cycle: you’ll understand how local LLMs really work, set up all the required software and dependencies, download and run an open-source model in LM Studio, and then build a simple yet powerful chat interface using Streamlit. From there, you’ll integrate your local LLM into the Streamlit app and learn how to store and review chat history using a local database securely. By the end, you’ll have a

Before you dive into building your offline Enterprise Assistant, it’s important to get familiar with a few key concepts.

At the heart of this setup is the Offline Assistant itself: an AI system that runs entirely on your computer, performing all language model inference locally without ever needing an internet connection.

Powering this is an LLM (Large Language Model), a type of AI trained on massive datasets to generate human-like text responses.

To make it simple to use, you’ll rely on LM Studio, a desktop app that lets you download, run, and serve open-source LLMs on your machine, exposing them through a local API.

For the interface, you’ll use Streamlit, a Python framework that makes it easy to build interactive web apps and quickly prototype AI-driven tools.

And finally, for securely managing chat history, you’ll work with SQLite, a lightweight local database that keeps all your interactions private and fully stored on your device.

By the end of this hands-on exercise, you’ll have your own local Enterprise Assistant running directly in your browser—powered by an open-source LLM that operates fully offline through LM Studio. You’ll interact with it using a simple but effective interface built with Streamlit, making your assistant practical and easy to use.

Most importantly, every conversation will be securely stored as local chat logs in your system, never sent to the cloud, never exposed. By the time you’re done, you’ll walk away with a private, offline AI assistant that runs fast and stays entirely under your control.

Demo Video and Repo

Demo Video

Lab guide

Conclusion

Congratulations! You’ve just built your very own offline Enterprise Assistant, powered entirely by open-source tools and running fully on your machine. Along the way, you learned how to set up LM Studio to run an LLM locally, how to create a lightweight but effective interface with Streamlit, and how to store all your conversations securely using SQLite. Most importantly, you now understand how to put privacy first, keeping every prompt, response, and workflow under your complete control, with no reliance on external servers or cloud APIs.

This hands-on exercise gave you more than just a working prototype. You gained insight into how local LLMs work, how to integrate them into real-world applications, and how to design AI tools that balance functionality with security. You’ve also seen the bigger picture: how on-device AI can reshape the way enterprises approach sensitive tasks, from R&D to legal reviews to compliance-heavy workflows.

But this is only the beginning. You can now extend your Enterprise Assistant with advanced features:

1. Add a smarter UI with more interactive elements.

2. Try out different open-source models to experiment with speed, accuracy, and capabilities.

3. Layer in analytics and insights to track and optimize your usage.

4. Even push towards agentic AI, giving your assistant the ability to automate tasks and workflows while still running securely offline.

With what you’ve built, you’ve proven that you can harness the power of Generative AI without compromise: no data leaks, no internet dependency, no loss of control.

Your private AI journey starts here.

- Saurabh

This is where most cloud teams still operate. And it’s exhausting.

I realized this after our AI-Powered Platform Engineering workshop last week.

If you were there, you know it was packed. We had George Hantzaras (Director of Engineering, MongoDB) and Ajay Chankramath (Founder, Platformetrics) up there walking through how AI, golden paths, and internal developer platforms are completely reshaping the way modern teams ship software. The energy in the session was incredible. And the conversations didn’t stop when the session ended.

But here’s the thing that stuck with me the most:

MCP (Model Context Protocol) turns your platform into something an AI can actually do things with, not just talk about.

This is the difference between TicketOps and brittle runbooks versus workflows like provisioning, rolling back, and scaling that become assistant-triggered actions with guardrails and full visibility into what’s happening. Your AI goes from being a really smart search engine to being an actual teammate.

That’s the jump I think a lot of us have been waiting for.

So in this issue, I’m going to dig into what that actually looks like, and give you a practical starter pack you can adapt to your own stack. Because if you’ve been wondering how to move from AI as “a chat window” to AI as “a teammate who actually does the work,” this is a good place to start.

Cheers,

Shreyans Singh

Editor-in-Chief

The Problem We’re All Facing

Let me paint a picture of how most cloud teams still operate today. You write a ticket. You wait for approvals. You hunt through runbooks to find the right commands to copy-paste. You watch CI/CD spin up. You cross your fingers that the safety nets catch anything that could go wrong.

Documentation and internal portals help you find things, sure. But they rarely do things. That gap, between knowing what to do and actually doing it, creates this constant context switching, slow feedback loops, and inconsistent safety guardrails.

The result is messy. Really messy. Brittle handoffs. Workflows duplicated across teams. And what should be “golden paths” that empower teams end up becoming “golden cages” that block velocity instead of enabling it.

Here’s what I think is the real core issue: we’ve separated knowledge from action. Your runbooks live in wikis. Your SLOs live in monitoring dashboards. Your approval workflows live in ticketing systems. And your developers, the people who actually need to ship things, they’re stuck stitching all of this together manually, one step at a time.

This friction doesn’t just slow shipping down. It creates inconsistency. One team deploys with proper approvals; another skips them. One team checks SLO impact before scaling; another scales blind. You end up with fragmented practices across your org, and nobody likes that.

We need a different model. Instead of “read the docs, then run the steps,” we should be asking our systems: “propose a plan, then execute it with guardrails.”

Here’s What Changed:

The solution rests on two foundational ideas working together, and honestly, the way they complement each other is elegant.

First: golden paths. These reduce decision fatigue without sacrificing flexibility. Think of a golden path as an end-to-end workflow with sensible defaults, something like: stateless service → preview environment → automated checks → production. These paths capture your team’s collective knowledge about how to do things safely. But they’re not straightjackets. You build in escape hatches for the 20% of cases that don’t fit the mold. Golden paths let you standardize without fossilizing your processes, and they’re the antidote to “TicketOps.”

Second: the Model Context Protocol (MCP). This is the glue that makes it all work. MCP is a standard that exposes your golden paths as tools that an AI assistant can actually call. Think of it as an adapter layer between your assistant and your platform. The assistant can observe your systems: what services exist, what their SLOs are, what incidents are open right now. It can propose a plan (“I’ll deploy this to preview, run tests, then promote to prod”). And it can execute approved actions while logging everything for audit and observability. MCP turns your platform into a set of composable, auditable operations.

Here’s what this looks like in the real world. An engineer types: “Spin up a preview environment and load our staging data.”

The assistant gathers context. What’s the service’s dependency tree? What are the current SLO baselines? Who owns this service? What recent incidents or postmortems should I know about? It drafts a plan that respects your runbooks and standards. It shows you the diffs and policy checks that apply. Then it either executes immediately (low-risk actions) or routes to the right approver (medium/high-risk actions). Everything gets logged.

This isn’t theoretical anymore. It works.

How You Actually Build This

The architecture starts simple: pick one golden path. One high-value workflow, like service deployment, that you want to standardize.

Next, you build MCP servers that act as adapters to your existing systems. Your runtime layer exposes Kubernetes operations (deploy, scale, health checks) and feature flags as callable tools. Your delivery layer connects to Git, GitHub Actions, or GitLab CI to handle builds, promotions, and canary deployments. Your reliability layer taps into SLO systems, alerting platforms, and observability tools so the assistant can query metrics, create SLOs, and mute alerts with proper context. Your change control layer enforces approvals and integrates with compliance workflows.

Underneath all of this runs retrieval-augmented generation (RAG). Before the assistant proposes any action, it indexes and searches your runbooks, compliance standards, postmortems, and architectural guidelines. The plan that comes back doesn’t just say “deploy the service”, it references your procedures. “Deploy using our standard blue-green pattern [link to runbook]. SLO impact: +2% latency at p99 (within budget). Dependencies: notify the data team [link to postmortem]. Policy check: OK, proceed.”

Safety gets enforced as policy-as-code, not friction. You define risk tiers: low-risk actions (like querying logs) auto-execute. Medium-risk actions (like scaling a service) require human approval. High-risk actions (like deleting a database) are blocked unless routed through a formal approval path. Standards violations get auto-fixed when confidence is high, or escalated when not. Everything is logged.

The user experience ties it all together. An engineer uses chat or a slash command to describe what they need. The assistant gathers context from your systems and knowledge base, drafts a plan with full diffs and impact estimates, shows what policy checks apply, and executes: staying in the loop for decisions that matter (approvals, tradeoffs) but handling the rote work. Everything that happened gets logged: the initial request, the plan, who approved it, what actually ran, what the outcome was.

The Real Payoff

Over time, this architecture unifies your platform. Your internal developer platform (IDP) becomes more than a portal where people click and read documentation. It becomes an actor that moves with your engineers, understanding context, respecting guardrails, and turning workflows into outcomes.

You measure it differently too. Not by pageviews to your portal, but by lead time to production. By mean time to recovery when things break. By consistency of safety across teams. You tie actions to business impact: AI-assisted scaling and predictive capacity planning can deliver up to 25% cost optimization while keeping SLOs intact.

The shift from ticket-based workflows to action-based workflows with AI assistance and policy-driven safety doesn’t happen overnight. Start with one golden path. Build the MCP adapters for one system at a time. Add guardrails incrementally. But the direction is clear.

Push intelligence and safety down into your platform so that doing the right thing becomes the easy thing. And the hard work, decision-making, tradeoff analysis, exception handling, stays where it belongs: with your engineers.

That’s where we’re headed. And honestly? I think it’s going to change how we all work.

Today’s CloudPro is about the five batch-scoring knobs most engineers overlook. If you’ve ever watched a job stretch from minutes to hours and wondered why, this is where you start.

Subscribe now

This article is adapted from Chapter 5 of Hands-On MLOps on Azure. In that chapter, author Banibrata De dives into the gritty details of model deployment: batch scoring, real-time services, and the YAML settings that make the difference between smooth pipelines and midnight firefights.

(The book goes much further, covering CI/CD pipelines, monitoring, governance, and even LLMOps across Azure, AWS, and GCP. CloudPro readers can grab it at the end of this piece with an exclusive discount.)

Cheers,

Shreyans Singh

Editor in Chief

Get The Book

Tuning Batch Jobs on Azure ML: 5 Knobs Every Engineer Should Know

It’s late. The batch run you trusted starts crawling. Dashboards spike, Slack pings light up, and you’re debating whether to kill the job or ride it out. You don’t need a re-platform. You need to tune the controls Azure ML already gives you.

Below are the five knobs that tame throughput, flakiness, and costs. They live in your batch deployment YAML, and they work.

1) mini_batch_size: The throttle for your workload

Batch jobs in Azure ML process data in chunks. mini_batch_size controls how big each chunk is. Push it too high, and you’ll hit memory or I/O bottlenecks; keep it too low, and you’ll waste time on overhead. Think of it like loading a truck: too few boxes and you’re underutilizing space, too many and you risk breaking the axle. Getting this balance right often cuts hours off long-running jobs.

2) max_concurrency_per_instance: How many cooks in the kitchen

Each compute node can process tasks in parallel, but how many at once depends on its resources. max_concurrency_per_instance is that dial. If you pack too much onto a single node, CPU and memory will thrash, and everything slows down. Start low, then gradually raise it while watching system metrics. The goal is steady throughput, not chaos.

3) instance_count: Scale out, don’t just scale up

Even with tuned concurrency, sometimes one node just isn’t enough. That’s where instance_count comes in. It decides how many nodes you’ll spread the workload across. It’s the knob you turn when you need predictable completion times. For example, making sure the nightly run finishes before business hours. More nodes mean more cost, but also fewer late-night surprises.

4) retry_settings: Resilience for the real world

In batch jobs, things fail: a network hiccup, a corrupted file, a transient storage timeout. Without retries, the whole job can collapse because of one small blip. retry_settings lets you say, “Try again a few times before giving up.” Set sensible timeouts and retries per mini-batch so small failures don’t derail the entire pipeline.

5) error_threshold: Fail smart, not early

What happens if some data records are bad? By default, too many errors can abort the run. With error_threshold, you control how many you’ll tolerate. Setting it to -1 tells Azure ML to ignore errors completely. For messy real-world datasets, this is a lifesaver: you can still ship 99% of results and deal with the outliers later, instead of losing the entire batch.

Extra sanity checks

• Respect the contract: Batch jobs are built for files/blobs in, files/blobs out. Don’t try to wrap them around per-record HTTP calls.

• Keep scripts separate: Use batch_score.py for batch and online_score.py for real-time. Different handlers, different expectations.

• Watch metrics that matter: Throughput, per-batch latency, error rate, and CPU/GPU/memory use. Wire alerts so you’re not caught off-guard at 2 a.m.

Takeaway

Batch scoring doesn’t have to be a black box. Azure ML gives you the levers. You just have to use them. Tune these five settings, keep batch and online flows separate, and you’ll get faster, more reliable runs without babysitting every night.

This walkthrough is pulled straight from Chapter 5 of Hands-On MLOps on Azure. The full book expands on everything here: deployments, monitoring, alerting, governance, pipelines, and operationalizing large language models responsibly.

For the next 48 hours, CloudPro readers get 35% off the ebook and 20% off print. If Azure ML is part of your stack, or about to be, this is the reference worth keeping open on your desk.

Get The Book

Today’s CloudPro is written by Daren Fulwell, Field CTO at IP Fabric. Daren has over 25 years of experience in networking, and he is well-known for his work mentoring engineers, and helping organizations bridge the gap between design fundamentals and modern automation.

In today’s CloudPro, he explores why AI on its own won’t deliver true network autonomy. Instead, Daren shows how the real path forward lies in combining automation, AI agents, and a network digital twin, so teams can move beyond hype and build networks that are transparent, predictable, and genuinely autonomous.

And if you’re looking to put these ideas into practice, we’ve just released the Network Automation Cookbook, Second Edition. Packed with over 100 hands-on recipes, it shows you how to automate network devices and cloud platforms using Ansible, AWX, Nautobot, and Terraform. It’s 30% off exclusively for CloudPro readers, for the next 72 hrs. Grab your copy and start building real-world automation workflows today.

Cheers,

Shreyans Singh

Editor in Chief

Subscribe now

Share

AI alone won’t deliver the autonomous network

By Daren Fulwell

In the network engineering community right now - as in most areas of IT - you can't escape the AI hype. We've been working to understand how network automation will change the way we operate our infrastructure, and agentic AI is being proposed as the missing piece of the puzzle. Folks in the know have been experimenting and making their results available in blog posts and Youtube videos for the rest of the world to salivate over. Finally, it looks like we have taken the right turn towards the self-driving network.

Or have we? Are a handful of small-scale experiments with limited scope and even more limited capability proving anything? At best, there is a lot more investigation required, at worst the experiments that we don't see are proving that AI is not to be trusted with our critical infrastructure yet.

Networks aren't just collections of individual devices that we configure and then they do what we tell them: they are interconnected, propagating their world view to their immediate neighbors and beyond, to create a "hive mind" behavior for the whole system. And in most cases, our networks are actually networks of networks - interconnected and sharing state information to extend that collective view from user to workload.

In traditional network operations, this meant having multiple teams - with their own documentation and subject matter experts in the technologies and platforms - who all needed to interface to provide end-to-end service. Maintenance of the infrastructure required deep collaboration between teams and across silos. A thorough understanding of the networking technologies needed to be applied to tooling and documentation to ensure change impacts were tracked and understood.

In the agentic AI world, this is taken to the next level. The work is divided up for agents to be given small, carefully-defined scopes to work within, making specific types of change or reporting on specific behaviors. But due to the distributed, interconnected nature of networking, none of those agents can work independently of the others: the effects caused by one will potentially be felt by them all. Without true collaboration between the agents, it becomes impossible for us to trust that they will give us the desired outcomes without humans (with an understanding of the infrastructure) manually checking everything they do.

In short, AI agents cannot operate the network autonomously without some collective understanding of the end-to-end network.

The Sources of Truth that we have been building for our network automation processes seem to fulfil at least elements of this need. But they alone are not enough as they really represent the desired state of the network, not its current operating state.

Consider these four key requirements for that source of knowledge:

1. We need a view of the end-to-end network in the form of structured data, with a well-documented schema, and able to be accessed over clearly defined APIs or protocols to provide a consistent end-to-end view to all agents

2. It must be a complete and up to date view of the network as it is operating. There is little point in having a clear view of part of the infrastructure then little or no understanding of other parts when one can so heavily impact the operation of the other.

3. Relationships and collective behavior are key to understanding how the network behaves: maintaining a list of devices (that may or may not be complete) and some data points about those devices may be useful but does not give the full picture

4. Network behavior is on the whole deterministic: a set of devices with specific state and connectivity should always behave the same way. So the data model must be based on facts collected from the network devices themselves, analysed and modelled as behavior - rather than being formed from conjecture, opinion or correlation of related events (the best you can expect from that is a general indication of direction)

Share

A true Network Digital Twin has all of these characteristics:

A Network Source of Truth system has some of these, but misses the key aspect of understanding network behavior end-to- end. For example, consider:

1. A change is required to enable Internet users gain access to an application hosted in a private DC. The external firewall is updated with NAT rules and policy changes to provide access; DNS changes are made; and routing is checked from the DC to ensure that traffic can be forwarded from user to frontend and back. But it still fails when the changes are pushed, because the security policy applied in the DC fabric only allows testing from internal hosts. The coordinated effort across multiple domains (read AI agents) has failed due to an incomplete view of the service dependencies;

2. A DR exercise is under way, causing applications to be switched from one location to another. Load balancing rules are being changed to facilitate that and the virtual IP successfully moves traffic flows to the new location. Two of the four servers in the load balancing pool are working fine, so the pool is up and being serviced, but not at full capacity. While the remaining servers are up and the correct services are running, routing from the load balancer to those servers is not correct: using the Digital Twin this end-to-end behavior can be diagnosed in advance and remediation carried out to fix this before live traffic is diverted through this path.

AI is going to change the way we operate networks. But in order to deliver its true potential, it needs not only to be able to deliver automated process, but to be fed real understanding of the networks it will operate in order to validate that it is doing what it needs to.

Share

If today’s article got you thinking about how to move from talking about automation to actually building it, you’ll want to check out our brand new release: Network Automation Cookbook, Second Edition.

This updated edition is packed with over 100 hands-on recipes showing how to use Ansible, AWX, Nautobot, and Terraform to automate both on-prem and cloud networks. It’s written for engineers who want practical workflows, not just theory, and every recipe comes with reproducible labs so you can practice safely.

As a CloudPro reader, you can grab it at 30% off for the next 72 hours.

Get The Book

Thanks for reading CloudPro!

Subscribe now

In today's CloudPro, he looks at how developers can use LXD containers on Ubuntu to keep their setups clean, spin up secure environments quickly, and avoid the headaches of traditional VMs.

Cheers,

Shreyans Singh

Editor-in-Chief

Share

Subscribe now

Have you ever broken your Ubuntu setup by installing conflicting packages, or wasted an entire afternoon waiting for a virtual machine to boot just to test one library? These little frustrations are all too common for developers. Containers on Ubuntu offer a cleaner way forward, and with LXD, you can build fast, secure, and reproducible environments that don’t weigh down your system.

Streamlining and Securing Development with Containers on Ubuntu

Developers often add tools and services directly onto their Linux desktops because it feels quick and convenient. The problem is that every new service increases your system’s attack surface, especially if something opens a network port in the background. A safer, more efficient approach is to run your development environments in containers. You spin them up only when you need them, and they stay isolated from your daily workflow.

In this article, we’ll discuss LXD, the Linux Container Daemon, due to its outstanding integration with Ubuntu. Conceptually, this would apply to other container technologies as well, but the usage would vary.

LXD stands out as a powerful solution for developers using Ubuntu, providing a lightweight and flexible approach to containerization, offering a compelling alternative to traditional virtual machines and other container technologies.

Why LXD instead of Docker or VMs?

If you’ve worked with Docker or virtual machines, you may wonder why LXD matters. Here’s a quick comparison to put it into perspective:

That last row is the key: LXD gives you a lightweight “virtual machine–like” environment, without the VM bloat.

Why Choose LXD for Development?

LXD containers are specifically designed to meet the needs of developers, offering a unique set of benefits:

1. Lightweight and Efficient: Unlike resource-intensive VMs that require full OS installations, LXD containers share the host kernel. This minimizes overhead, leading to significantly faster boot times, a reduced memory footprint, and improved overall performance. This efficiency is crucial for rapid iteration and testing across various environments without performance penalties.

2. Image-Based Management: LXD's reliance on images for container creation transforms environment management. It enables effortless sharing, versioning, and reproducibility of development environments, ensuring consistency across different machines and simplifying collaboration. This approach streamlines workflows, allowing developers to spin up new environments with specific configurations and dependencies quickly.

3. Security Fortified: LXD provides robust isolation through kernel namespaces and advanced security features, safeguarding the host system and other containers from potential vulnerabilities. This secure environment allows developers to focus on their code with peace of mind.

4. Scalability and Flexibility: LXD excels in scalability, allowing developers to easily create multiple isolated environments for different projects, branches, or feature implementations. This fosters a highly organized and efficient development process, enabling rapid switching between environments without impacting other projects.

5. Seamless Ubuntu Integration: LXD's tight integration with Ubuntu leverages the operating system's robust package management system and offers access to a vast repository of pre-built images and tools. This streamlines development and ensures compatibility with a wide range of software and libraries.

Quick use case:

Let’s say you want to try out PostgreSQL 17 without risking your workstation setup. Launch an LXD container, install PostgreSQL inside it, test it safely, and if things break, just roll back to a snapshot. Your main system stays untouched.

Getting Started with LXD on Ubuntu

Installing and configuring LXD on Ubuntu is straightforward, as it's available directly from the Snap Store.

ken@monster:~$ sudo snap install lxd
ken@monster:~$ sudo usermod -aG lxd "$USER"
ken@monster:~$ newgrp lxd
ken@monster:~$ lxd init --auto

The lxd init --auto command initializes LXD with recommended settings. For more control, omit --auto to go through an interactive configuration process, allowing you to choose storage backends (like ZFS for advanced features or LVM for flexibility), configure network settings (bridge interfaces or NAT), and set up image remotes to access pre-built images.

Essential LXD Container Management Commands

LXD provides a comprehensive command-line interface (CLI) for managing containers:

1. lxc launch <image> <name>: Creates and starts a new container from an image.

2. lxc list: Displays a list of all running containers.

3. lxc start/stop/restart <name>: Manages container lifecycle.

4. lxc exec <name> -- <command>: Executes commands within a running container.

5. lxc file push/pull <local_path> <remote_path>: Transfers files between the host and a container.

For development, it's often more convenient and secure to run as an ordinary user with your home directory mapped into the container:

ken@monster:~$ lxc launch ubuntu:25.04 plucky-devel -c raw.idmap="both $UID 1000"

ken@monster:~$ lxc config device add plucky-devel homedir disk source=$HOME path=/home/ubuntu

ken@monster:~$ lxc exec plucky-devel -- su -l ubuntu

This configuration launches a container, maps your user ID, mounts your home directory, and provides a login shell as the ubuntu user inside the container, allowing you to use your favorite editor on your host system while executing code within the isolated container.

Unlocking Advanced Features

LXD offers features that further enhance development workflows:

1. Remote Access: Manage containers remotely via the secure REST API.

2. Networking Mastery: Configure virtual networks to isolate containers and simulate complex network topologies for testing.

3. Storage Management: Optimize storage performance with different backends like ZFS or LVM.

4. Profiles for Reusability: Define reusable profiles to simplify container creation with consistent configurations.

5. Snapshots and Rollbacks: Capture container states to revert to previous working configurations, ideal for experimentation quickly.

6. Moving and Migrating Containers: Easily move or migrate running containers between LXD hosts or even to different cloud providers.

Pro tip:

If you often create containers with similar settings, use profiles. They’ll save you from repeating the same config steps over and over. The Ultimate Ubuntu Handbook shows how to build reusable profiles for real projects.

The Future of LXD

LXD continues to evolve, with ongoing efforts to integrate with Kubernetes for seamless orchestration and improved virtualization support for demanding workloads. Enhanced security features and a developing web-based user interface (sudo snap set lxd ui.enable=true && sudo snap restart lxd) are also on the horizon, making LXD even more accessible and powerful for developers.

The bottom line? LXD is a game-changer for developers on Ubuntu, offering a compelling blend of efficiency, security, and flexibility. By embracing LXD, developers can create efficient, reproducible, and secure environments that streamline workflows, enhance collaboration, and accelerate innovation.

Share

Ken’s walkthrough is just a glimpse into what’s possible when you bring containers into everyday development on Ubuntu. If you’d like to go deeper, his book The Ultimate Ubuntu Handbook is full of practical examples, from building secure containers to streamlining workflows and preparing for production deployments. It’s a guide designed to stay useful long after the first read.

Get The Book

Cheers,

Shreyans Singh

Editor-in-Chief

Share

Subscribe now

AI Powered Platform Engineering Workshop

Most platform teams hit the same wall: tools pile up, self-service breaks down, and platforms get rebuilt every 18 months.

On September 16, join CNCF Ambassador Max Körbächer for a 3-hour live workshop on how to design internal platforms with AI baked in.

LAUNCH OFFER: 40% OFF for 48 Hours

Use Code: LIMITED40

The Hidden Platform Lesson Behind Airbnb’s Data Quality Framework

A Blueprint for Smarter Platform Engineering

A few years back, Airbnb hit a painful truth: a single data bug could quietly poison dashboards, mislead teams, and steer decisions the wrong way.

To deal with it, Airbnb launched a Data Quality Initiative. The company rolled out Midas, a certification process for critical datasets, and made checks for accuracy, completeness, and anomalies mandatory.

It sounded good on paper. But in practice, it quickly turned into a mess.

Every team wrote checks in their own way: Hive, Presto, PySpark, Scala. There was no central view of what was covered, and updating rules meant editing code in a dozen different places. Teams duplicated effort, each building their own half-complete frameworks to run checks. And pipelines grew heavy: every check was an Airflow task, DAG files ballooned, and false alarms could block production jobs.

Airbnb needed a better path.

So, they built Wall, a single framework for checks. Instead of custom code, engineers wrote checks in YAML configs. An Airflow helper ran them, keeping logic separate from pipelines. Wall added support for blocking vs. non-blocking checks, so minor issues didn’t stop critical flows. And instead of burying results in logs, it sent them into Kafka for other systems to consume.

The results were dramatic. Some pipelines shed more than 70% of their Airflow code. Teams stopped reinventing the wheel. Data-quality checks went from fragile and inconsistent to a paved path everyone could rely on.

What Platform Engineers Can Learn from Airbnb

1. Standardize Pipelines and Checks

In Mastering Enterprise Platform Engineering, authors Mark and Gautham makes a simple point: reliable AI doesn’t come from choosing the perfect model. It comes from the rails underneath: pipelines, integration layers, automation, and guardrails. The numbers are striking: clean pipelines alone can improve model performance by 30%, audits can cut errors by 90%, flexible integration can halve deployment time, and predictive automation can reduce downtime by 50%.

Reliable AI starts with consistent pipelines and repeatable checks. When pipelines are clean and quality checks are routine, the entire system gets more predictable. That’s why standardized checks and audits can boost accuracy so dramatically.

Airbnb learned this the hard way. Each team had its own approach to checks, spread across different engines. The duplication and inconsistency created constant drag. Wall fixed it by moving to YAML-based checks in a single framework. Suddenly, teams were speaking the same language. Some pipelines saw their DAGs shrink by more than 70%.

2. Decouple Checks from Workflow Code

One of the biggest risks in complex systems is tangling logic together. When validation lives inside workflow code, every change increases fragility. By pulling checks out into their own layer, you gain flexibility, reuse, and resilience.

Wall embodied this. Instead of clogging DAGs with checks, it made them independent services. Results flowed into Kafka, where other systems could consume them. Checks weren’t bound to pipelines anymore; they became a decoupled, reusable rail.

3. Close the Loop with Automation

Validation without action is just noise. The real value comes when checks automatically trigger responses: scaling a service, blocking a bad job, or notifying the right team. This kind of predict→act loop is where platform engineering proves its worth.

Wall pushed Airbnb in this direction. By publishing results as Kafka events, checks could plug into downstream tools that acted immediately. Instead of waiting for humans to parse dashboards, the system closed the loop itself.

4. Build Guardrails, Not Just Tests

Not every failed check should bring the system down. The right approach is to design guardrails: rules that let you decide what’s a blocker and what’s not. This keeps the platform safe without making it brittle.

Wall introduced blocking vs. non-blocking checks to solve this. Critical issues stopped the flow; minor ones didn’t. That simple design choice turned fragile pipelines into resilient ones. Guardrails, not tests, are what kept the system trustworthy.

5. Standardize Tools to Reduce Friction

Every extra framework, every redundant tool, adds friction. Engineers spend more time maintaining and less time building. The fix is to standardize on a common set of rails, even if it means trade-offs.

Airbnb saw this firsthand. With every team writing their own frameworks, they were duplicating effort and missing features. Wall gave them a single standard, cutting out wasted work. Once engineers stopped arguing about how to check data, they could focus on using it.

This walkthrough was adapted from Mastering Enterprise Platform Engineering and connects to Packt’s AI-Powered Platform Engineering Workshop on September 16.

It’s a live, 3-hour session led by CNCF Ambassador Max Körbächer, focused on how to build internal platforms with AI baked in: platforms that stay usable, scalable, and sustainable.

CloudPro readers get 40% off tickets for the next 48 hours.

LAUNCH OFFER: 40% OFF for 48 Hours

Use Code: LIMITED40

Subscribe now

Share

30 second summary for you:

Running your app in Kubernetes doesn’t mean it’s highly available. This walkthrough shows how ReplicaSets restore failed pods, handle node loss, and work with probes, all backed by real command-line examples. Adapted from Packt’s The Kubernetes Bible (Chapter 10).

• 8-minute read

• Hands-on commands included

The Problem: One Dead Pod, and Your App Stalls

Let’s say you’ve got a stateless NGINX app deployed in a multi-node Kubernetes cluster using a ReplicaSet. You think you’re covered because there are 4 replicas. But then you:

• delete a pod manually

• drain one of the nodes

• simulate a container failure

In all three cases, you’re expecting automatic recovery. But it’s not magic. It's ReplicaSet (and sometimes liveness probes) doing the heavy lifting.

Let’s walk through all three failure modes and see what Kubernetes does.

Pod Deletion? No Problem.

This scenario demonstrates how a ReplicaSet restores deleted pods to maintain the desired number of replicas.

Here's a step-by-step walkthrough:

1. Define the ReplicaSet manifest: Save the following YAML as nginx-replicaset-example.yaml:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-replicaset-example
  namespace: rs-ns
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
      environment: test
  template:
    metadata:
      labels:
        app: nginx
        environment: test
    spec:
      containers:
        - name: nginx
          image: nginx:1.17
          ports:
            - containerPort: 80

2. Create the namespace: This ensures all your resources are scoped properly.

kubectl create -f ns-rs.yaml

3. Deploy the ReplicaSet: The manifest defines a ReplicaSet with 4 NGINX pods.

kubectl apply -f nginx-replicaset-example.yaml

4. Delete a pod manually: Simulate a pod failure by deleting one of the running pods.

kubectl delete pod <pod-name> -n rs-ns

5. Verify that the ReplicaSet restores the pod: The controller detects the change and automatically spins up a new pod to maintain the desired count.

kubectl get pods -n rs-ns
kubectl describe rs/nginx-replicaset-example -n rs-ns

Within seconds, the ReplicaSet controller notices the missing pod and recreates it to meet the declared replica count.

Takeaway: ReplicaSets automatically maintain the number of desired pods, making recovery from manual deletions fast and hands-free.

2. Node Failure? Here's What Actually Happens

This scenario demonstrates how ReplicaSets maintain high availability when a node goes down by rescheduling pods onto available nodes:

Here's a step-by-step walkthrough:

1. Expose your app with a Service:

kubectl apply -f nginx-service.yaml

This creates a service to access your app across pods.

2. Forward traffic from your local machine to the Kubernetes Service:

kubectl port-forward svc/nginx-service 8080:80 -n rs-ns
curl localhost:8080

This confirms your service is working and traffic is flowing to the pods.

3. Check where the pods are currently running:

kubectl get pods -n rs-ns -o wide

This shows which node each pod is scheduled on.

4. Simulate node failure by cordoning and draining the node:

kubectl cordon kind-worker

Prevents new pods from being scheduled on this node.

kubectl drain kind-worker --ignore-daemonsets

Evicts all running pods from the node while ignoring daemonsets.

kubectl delete node kind-worker

Removes the node from the cluster to simulate a full node failure.

Within moments, the ReplicaSet detects the missing pods and spins up new ones on the remaining healthy nodes. Your Service automatically reroutes traffic to these new pods.

5. Verify that everything is still working:

kubectl get pods -n rs-ns -o wide
curl localhost:8080

You’ll see that traffic still flows, and the app remains accessible without downtime.

Takeaway: The ReplicaSet ensures that the desired number of pod replicas is always maintained — even when a node goes offline. It handles pod rescheduling automatically, as long as there's sufficient capacity in your cluster.

3. Unhealthy Container? Probes Save the Day

Let’s see how Kubernetes handles an unhealthy container using liveness probes.

Here's a step-by-step walkthrough:

1. Add the following liveness probe to your ReplicaSet pod spec. It instructs the kubelet to check container health after 2 seconds and repeat every 2 seconds:

livenessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 2
  periodSeconds: 2

2. Apply your updated ReplicaSet manifest and wait for the pod to be up and running.

3. Simulate a container failure by deleting the default NGINX index file:

kubectl exec -it <pod-name> -- rm /usr/share/nginx/html/index.html

4. Check what happens by describing the pod:

kubectl describe pod <pod-name>

You’ll see Liveness probe failed events, followed by automatic container restarts.

Takeaway: The kubelet, not the ReplicaSet, manages container health. But when used with ReplicaSets, probes help create a resilient system that self-heals when a container goes bad.

Cleanup

You can delete the ReplicaSet and its pods:

kubectl delete rs/nginx-replicaset-livenessprobe-example

Or just delete the controller, leaving pods untouched:

kubectl delete rs/nginx-replicaset-livenessprobe-example --cascade=orphan

Key Takeaways

• ReplicaSets guarantee pod replication and replacement—not health checking

• Liveness probes enable kubelet to restart broken containers

• Node failure recovery works if your cluster has enough capacity and replicas are spread

• HA = ReplicaSets + Probes + Services, working in tandem

Based on Chapter 10 of The Kubernetes Bible, Second Edition

In this week’s issue, there’s a quick fix for bloated Terraform states, a clean Docker Compose alternative with Quadlet, and new AWS features like remote Lambda debugging and native blue/green ECS deploys.

You’ll also find a GitOps primer with Argo CD, a free Kubernetes IDE, and real benchmark data on which Gateway API controllers hold up at scale.

If you want updates like these daily, not just weekly, follow Packt SysOps. One practical post every weekday at 9AM ET, with lessons from real cloud teams.

Cheers,

Shreyans Singh

Editor-in-Chief

Subscribe now

📦 Kubernetes & Cloud Native

Kubernetes v1.33 now supports hybrid post-quantum TLS key exchange by default, thanks to its upgrade to Go 1.24. This enables X25519+ML-KEM (Kyber) for TLS without explicit configuration. However, mismatched Go versions across clients and servers can cause silent downgrades to classical encryption. PQ signatures aren’t yet production-ready due to large key sizes and limited tooling support.

EKS Auto Mode now supports pod-specific subnets using podSubnetSelectorTerms in EKS Node Classes. This allows developers to assign pods IPs from custom subnet ranges, improving network isolation. Combined with Karpenter Node Pools and Terraform automation, teams can now declaratively manage these configurations at scale.

Intro to GitOps with Argo CD

This beginner-friendly guide explains GitOps and shows how to deploy Argo CD to automate Kubernetes app delivery. It walks through installing Argo CD, exposing it via Ingress, and logging in securely, eliminating complex CD pipelines and simplifying multi-team access.

Free IDE for Kubernetes

Freelens is a free, open-source desktop app for managing Kubernetes clusters, available for macOS, Linux, and Windows via multiple package managers. Built as a fork of OpenLens, it simplifies cluster operations with a clean UI, bundled CLI tools, and extension support.

A new open-source benchmark suite tests seven major Gateway API implementations, like Istio, Envoy Gateway, and Traefik, across route setup, scaling, architecture, and performance. The results show large differences in reliability and scalability, with Istio and Kgateway standing out positively, while Nginx, Cilium, and Traefik suffered critical failures or severe scaling issues. For cloud engineers, this benchmark helps cut through marketing claims and highlights which controllers are production-ready.

⚙️ Infrastructure & DevOps

Google Cloud Run Adds Native Support for Docker Compose AI Deployments

Now in private preview, this simplifies moving multi-container AI apps from local to cloud with GPU support and persistent volumes. Cloud Run’s recent GPU GA and fast scaling make it a strong platform for agentic and LLM workloads.

Google Cloud Expands Cluster Director with GUI, Managed Slurm, and Anomaly Detection.

Users can launch optimized clusters with GPU, network, and storage setup in under a day, with built-in topology-aware scheduling and straggler detection. The updates aim to reduce setup time and improve performance for large-scale distributed training.

AWS has launched a developer preview of the API MCP Server, allowing foundation models to convert natural language into valid AWS CLI commands. This tool enables FM-powered agents to inspect and manage AWS resources securely through IAM-based permissions. It's open source and now available on GitHub for experimentation.

Amazon Bedrock AgentCore is now in preview, offering modular services to help developers run AI agents at scale with production-grade security and observability. It includes tools for session management, memory, API integration, web browsing, code execution, and identity control.

AWS has launched two new features for Lambda: console-to-IDE integration and remote debugging. Developers can now open Lambda functions directly in VS Code with a single click, and debug cloud-deployed functions live from their IDE, including access to VPCs and IAM roles.

Amazon ECS now supports native blue/green deployments, making it easier to roll out application updates safely without custom tooling. You can test new revisions in parallel, use lifecycle hooks for automated validation, and instantly roll back if needed, all with no end-user disruption.

🔐 Cloud Security

AWS Fixes Flaw That Allowed Full Org Takeover via Delegated Admins

Researchers found a way to take over entire AWS Organizations by combining misconfigured delegated admin accounts with an overly permissive managed policy. A user in a compromised account could gain control of every account, including the management account. AWS has released a fixed policy (v2), but the old version is still active if not manually replaced. Teams should audit delegated admin roles and update any remaining v1 policies immediately.

AWS IAM Action Classifications Updated. But Inconsistencies Remain

Fog Security found mismatches between AWS’s new programmatic IAM action listings and the older Service Authorization Reference (SAR) pages. Some actions have multiple classifications, others are missing or categorized differently across the two sources. These inconsistencies could affect IAM tooling and workflows. Teams using SAR data should review the differences before switching to the new programmatic references.

CDK Construct that syncs your sops secrets into AWS SecretsManager secrets.

The cdk-sops-secrets project helps developers securely sync SOPS-encrypted secrets into AWS Secrets Manager or SSM Parameter Store using CDK constructs. It supports JSON, YAML, dotenv, and binary formats, with features like batch uploads and automatic IAM permission generation. The tool also allows customization via a singleton Lambda provider.

Serverless Password Manager Built Entirely on AWS Free Tier

RunaVault is an open-source password manager using AWS Cognito, Lambda, DynamoDB, and KMS to store and share secrets securely. It’s built for zero-cost deployments under the AWS free tier, with features like MFA, RBAC, and client-side encryption.

S3 Security Scanner for Access and Ransomware Protection

YES3 is a Python-based tool that scans AWS S3 buckets for security misconfigurations, including public access, encryption gaps, versioning, and object lock issues. It also checks account-wide settings like public access blocks and logs findings in a readable format.

🔍 Observability & SRE

Google Cloud has rolled out a new Application Monitoring feature that auto-generates dashboards, logs, and metrics views for services defined in App Hub. The tool helps teams troubleshoot faster by surfacing golden signals and propagating labels across logs, metrics, and traces. It also integrates with Gemini Cloud Assist.

Microsoft has expanded Project Flash to give Azure users deeper, real-time visibility into VM availability disruptions. New features include a context-aware metric in Azure Monitor that distinguishes between platform- and user-triggered issues, and Event Grid integration for instant alerts.

Amazon EventBridge now supports enhanced logging to CloudWatch, S3, and Kinesis Firehose, helping teams debug event-driven apps more effectively. Users can choose log levels (error, info, trace), include event payloads, and track rule matches and invocation errors. This makes it easier to trace event flows and spot failures without custom tooling.

Amazon EventBridge now offers enhanced logging that tracks the full lifecycle of events, from receipt to success or failure, across CloudWatch, S3, or Firehose. Logs include rich metadata, latency breakdowns, and error details, helping engineers pinpoint issues in Lambda targets or API destinations.

AWS S3 Metadata now supports querying metadata for all objects, not just new or updated ones—via fully managed Iceberg tables. Live inventory tables and journal tables enable SQL-based queries to track storage usage, object changes, deletions, and lifecycle activity. This simplifies cost optimization, auditing, and ML pipeline prep by eliminating the need for manual scanning or S3 Inventory jobs.

Subscribe now

Share

Cheers,

Shreyans Singh

Editor-in-Chief

Which Call Paths Dominate at Runtime: Using Flame Graphs to Visualize it!

By Kaiwan N Billimoria

Analyzing workloads is something all engineers end up doing at some point or another (or it’s their job description!). An obvious reason is performance analysis; for example, CPU usage may spike at times, causing issues or even outages.

The need of the hour: observe, analyze, and figure out the root cause of the performance issue! Of course, that’s often easier said than done; this kind of work can bog down even experienced professionals...

Borrowing from Brendan Gregg’s wonderful presentation (though old, it’s still relevant):

In general, answering the ‘Who’ and the ‘How’ are simple(r):

• ‘Who?’: well-known tools like top (and its numerous variants – htop, atop, etc) help answer this question.

• ‘How?’: lots of system monitoring tools are available (vmstat, dstat, sar, nagios, cacti, nmon, iostat, nethogs, sysmon, etc.).

The harder questions tend to be the ‘Why?’ and ‘What?’:

• ‘Why?’: by generating a Flame Graph! (the topic of this short article)

• ‘What?’: Flame Graphs as well as plain old perf!

The following slide illustrates this (again, from Brendan Gregg):

Right. So what the heck’s this Flame Graph thingy? Let’s explore!

We’ll abbreviate Flame Graphs as FG.

There are several types of FGs (CPU, GPU, memory, off-cpu, etc.); here we keep the focus on just one: CPU FGs via Linux’s powerful perf CPU profiler.

The moment a tool can generate profiling data that includes stack traces, it implies that FGs can be generated! Thus, there are several tools besides perf that generate FGs:

• Windows: WPA, PerfView, Xperf.exe

• Linux: perf, eBPF, SystemTap, ktap

• FreeBSD: DTrace

• Mac OS X: Instruments

We’ll focus only on using Linux perf; it’s considered one of the best modern CPU profiling tools on the platform

Motivation for FGs

With perf, you can indeed profile your workload and see where exactly CPU usage shoots up. It’s easy: record something, get the report, and analyze it (well… it sounds easy at least).

Example:

• Record a system-wide profiling (-a option switch) session with stack chain / backtrace (--call-graph dwarf, old option was -g), frequency of 99 Hz, for 10 seconds:

sudo perf record -F 99 -a --call-graph dwarf -- sleep 10

(Instead of the -a option switch, you can use the -p PID option to profile a particular process. The generated perf.data file’s owned by root; do a chown to place its ownership under your account if you wish.)

• Get the perf report:

sudo perf report --stdio # or --tui

…

(Try it!).

This begs the question – so why not just use perf? Ah, that’s the thing: on non-trivial workloads, the report can be simply humongous, even going into dozens of (printed) pages! Are you really going to read through all of it, trying to spot the outliers?

Visualization with the CPU Flame Graph

It’s why we use the so-called Flame Graph (FG) – to visualize dense textual data and make sense of it; it’s so much clearer (so much more humane, literally).

Installation

First off, ensure both the perf utility and the FlameGraph scripts are installed.

Quick note: to install perf on Ubuntu/Debian, you typically need to be on a distro kernel (not a custom one).Why? Because – unusually for an app – it’s tightly coupled to the kernel it runs on! Assuming you’re on an Ubuntu/Debian distro, do this: sudo apt install linux-perf-$(uname -r) linux-tools-generic (even the linux-tools-generic package might be sufficient).

If you’re on a custom-built kernel, build perf (it’s easy): cd <kernel-src-tree>/tools/perf ; make .

Install FG from here or do (in an empty folder):

git clone --depth 1 https://github.com/brendangregg/FlameGraph.git

Steps to generate a Flame Graph

1. Profile the workload using perf:

perf record -F 99 --call-graph dwarf [-a]|[-p pid]

• -a: all cpus; in effect, if specified, the sample is system-wide

• -p: sample a particular process.

Generates the perf.data binary file.

2. Read from perf.data (default, else use -i <fname>) to convert the binary data to human-readable stack traces via perf script:

perf script > perfscript_out.dat

3. Generate the FG, a Scalable Vector Graphic (SVG) file:

The FG repo includes several stackcollapse-* scripts; we use the stackcollapse-perf.pl one:

cat perfscript_out.dat | FlameGraph/stackcollapse-perf.pl \

| FlameGraph/flamegraph.pl > out.svg

4. Open the SVG in a web browser, move the mouse over stack frames.

A Quick Test Run

We’ll assume you’ve installed both perf and the Flame Graph GitHub repo (the latter under your home dir).

1. Profile: record everything for 10s

sudo perf record -F 99 -a --call-graph dwarf -- sleep 10sudo chown ${LOGNAME}:${LOGNAME} perf.dataperf script > perfscript_out.datcat perfscript_out.dat | ~/FlameGraph/stackcollapse-perf.pl |~/FlameGraph/flamegraph.pl > out.svg

2. Open the SVG file in a web browser. Here’s a screenshot of the Flame Graph

Hmm, better if we zoom in… so I click on one of the rectangles on the lower-left (say on the gnome-shell one):

Ah, better.

Interpreting the Flame Graph

Some really key points regarding how to interpret the Flame Graph:

• Each rectangle represents a single stack frame; read it bottom-up.

  • The width is representative of the frequency of the function call.

  • The height is representative of the depth of the stack

• The order of rectangles from left-to-right is just alphabetical; it's not a timeline.

• The colors don’t signify anything special.

• You can (typically) use the browser Search (Ctrl-F) to search for a function by name.

• Click on a stack frame (a rectangle) to zoom into that tower. Click Reset Zoom (upper-left corner) to zoom back out.

In effect: the hottest code-paths – the ones that dominate - are the widest rectangles!

The top-edge – the rectangle at the very top - is the function on-CPU; beneath is ancestry (how it was invoked).

Here’s another FG I captured while SSH was running (truncated screenshot showing the interesting portion):

Interesting; the “towers” seem to be inverted! Yes, they’ve becomes top-down (downward-growing stacks) instead of bottom-up… they’re called icicles!

An option to the perf script command sets this up.

A fantastic thing about the FG is that both userspace and kernel-space functions are captured! It’s thus called a mixed-mode FG. For e.g., with the ‘ssh’ FG, you can clearly see the call path leading down to the kernel network protocol stack code – functions from the socket/INET layer sock_*(), followed by L4 tcp_*(), followed by the L3 ip_*() functions; even the invocation of the (network) device transmit – the dev_hard_start_xmit() and others – are visible!

My flamegrapher.sh wrapper scripts

Next, to make this a bit easier to use (no need to remember the syntax, easier options), I wrote a wrapper over the original Flame Graph scripts; the top-level one’s named flamegrapher.sh: https://github.com/kaiwan/L5_user_debug/tree/main/flamegraph (it forms a portion of my ‘Linux Userspace Debugging – Tools & Techniques’ training repo).

It’s Help screen reveals how you can – very easily! – use it to generate FGs:

$ ./flame_grapher.sh

Usage:

flame_grapher.sh -o svg-out-filename(without .svg) [options ...]
-o svg-out-filename(without .svg): name of SVG file to generate (saved under /tmp/flamegraphs/)

Optional switches:

[-p PID]: PID = generate a FlameGraph for ONLY this process or threadIf not passed, the *entire system* is sampled...

[-s <style>]: normal = draw the stack frames growing upward [default]icicle = draw the stack frames growing downward

[-t <type>]: graph= produce a flame graph (X axis is NOT time, merges stacks) [default]Good for performance outliers (who's eating CPU? using max stack?); works well for multi-threaded apps

chart= produce a flame chart (sort by time, do not merge stacks)

Good for seeing all calls; works well for single-threaded apps

[-f <freq>]: frequency (HZ) to have perf sample the system/process at [default=99]Too high a value here can cause issues

-h|-?: show this help screen.

Note:

• After pressing ^C to stop, please be patient... it can take a while to process.

• The FlameGraph SVG (and perf.data file) are stored in the volatile /tmp/flamegraphs dir; copy them to a non-volatile location to save them.

Notice a few points:

• The only mandatory option switch is -o fname; it generates an SVG file named fname.svg.

• There are two ‘types’ of FG’s we can generate:

  • graph [default]: Produce an FG (X axis is NOT time, merges stacks). This type’s good for performance outliers (who's eating CPU? using max stack?); works well for multi-threaded apps.

  • chart : Produce a flame chart – it’s sorted by time, do not merge stacks. Good for seeing all calls; works well for single-threaded apps.

• You can optionally specify a particular process (by -p PID) to profile, change the style to icicle, and set the profiling frequency.

• The metadata and the SVG is stored under /tmp; copy it to a non-volatile location if you want it saved!

(Do read README.md as well. Hey, this wrapper’s lightly tested; please help me (and everyone!) out by raising Issues, as and when you come across them!)

Tip: Try the speedscope.app site to interact with your FlameGraph!

Flame Graphs: Caveats/Issues

• Frame Pointers being present helps get good stack traces, BUT the -fomit-frame-pointer is the typical GCC flag passed!

  • Possible exception case is the Linux kernel itself; it has intelligent algorithms to emit accurate stack trace even in the absence of frame pointers.

• Symbols are required (can use a separate symbol file). A side effect of no symbols may be ill-formed (or close to zero) stack traces.

• VMs may not support the PMCs (performance measurement counters) that perf requires; in that case, FGs (or perf) don’t really work well.

Bonus material

B Gregg’s Linux Performance Observability Tools diagram across the stack!

Tips

• With [e]BPF becoming a powerhouse for many things, including observability, do look up equivalent eBPF tooling as well: https://www.brendangregg.com/ebpf.html (a similar diagram’s here!).

• Also be sure to check out B Gregg’s (and others) utility package wrappers: perf-tools and bpfcc-tools.

• Don’t ignore systemd’s systemd-analyze tool (boot-time).

• Perf: simply running sudo perf top is itself useful to find outliers; I keep a couple of aliases as well:

alias ptop='sudo perf top --sort pid,comm,dso,symbol 2>/dev/null'
alias ptopv='sudo perf top -r 80 -f 99 --sort pid,comm,dso,symbol \
--demangle-kernel -v --call-graph dwarf,fractal 2>/dev/null'

If you enjoyed reading this article, check out Kaiwan’s book Linux Kernel Programming!

He’s written today's newsletter on Bash vs Python in Cloud Infrastructure, and why shell scripting still wins in a bunch of real-world cases. He compares real tasks in both languages and shows how often Bash just gets you there faster, with less setup and fewer surprises.

Cheers,

Shreyans Singh

Editor-in-Chief

Bash vs Python in Cloud Infrastructure

by Donald Tevault

Python is a great programming language, and you can do a lot of awesome stuff with it.

However, you might at times find that Python is more than you need, or more than you can easily learn. For such jobs, you might want to consider shell scripting instead.

Let’s look at some specific reasons:

To begin with, Python might not be installed on every workstation, server, IoT device, or container that you need to administer.

On the other hand, every Linux, Unix, or Unix-like system that you’ll encounter has a shell already installed. Apart from bash, you’ll find Bourne Shell on BSD-type systems, and lightweight shells such as ash or dash on Linux for IoT devices.

Now, let’s say that you have a Linux-based IoT device and you need to parse through its webserver logs. The tools you’ll need to do this with shell scripting are already there, but Python likely isn’t.

Shell Script Portability versus Python Portability

The difference between bash and the other shells that I mentioned is that bash has some advanced features that the other shells lack. If you know that you’ll only need to run your scripts in a bash environment, then you can definitely take advantage of the extra bash features.

But, by avoiding the bash-specific features, you can create shell scripts that will run on a wide variety of shells, including bash, ash, dash, or Bourne Shell. Fortunately, that’s not as hard as it would seem. For example, you can create variable arrays in bash, but not in the other shells. If you need cross-shell portability but also need the benefits of using an array, you can easily create a construct that simulates an array, and that has the same functionality. It’s easy-peasy once you know how.

One portability problem with Python involves Python’s use of programming libraries that might or might not be installed on every device on which the Python script needs to run. In fact, you might have encountered this problem yourself if you’ve ever downloaded a Python script from GitHub. If you’re not a Python expert, it might take you a while to figure out how to install all of the required libraries.

With shell scripting, you don’t need to worry about libraries, because shell scripts use the command-line utilities that already come installed on pretty much every Linux, Unix, or Unix-like system. Another problem is that scripts that were created for the old Python 2 aren’t always compatible with the new Python 3.

Next, let’s talk about something that’s especially important to me personally.

The Shell Scripting Learning Curve versus the Python Learning Curve

If you’re a DevOps person, you’ve likely already mastered Python. But, if you’re more into systems administration, there’s a good chance that you haven’t had much experience with Python. Fear not, because even if you’re lousy with learning programming languages, as I am, you can still learn how to do some awesome things with old-fashioned shell scripting. Even if you do know Python, you might find that certain jobs can be accomplished more quickly and easily with shell scripting than with Python.

For example, here’s the script that I use to update and shut down the OpenMandriva workstation that I’m using right now:

#!/bin/bashdnf -y distro-sync && shutdown now

All this shell script contains is just the commands that I would normally run from the command line. With shell scripting, no coding skill at all is required for this.

Working with text files is way easier with shell scripting. Let’s take this text file with a listing of classic automobiles:

The fields in this file represent the make, model, year, mileage in thousands of miles, and U.S. dollar value of each car. Now, let’s say we want to sort this file alphabetically and save the output to a new file. Here’s how you could do it with Python:

#!/usr/bin/python 

def sort_file_content(in_path, out_path): 
    lines = [] 

    with open(in_path) as in_f: 
        for line in in_f: 
            lines.append(line) 

    lines.sort() 

    with open(out_path, 'w') as out_f: 
        for line in lines: 
            out_f.writelines(line) 

 
if __name__ == "__main__": 
    input_file = "autos.txt" 
    output_file = "sorted_autos.txt" 
    sort_file_content(input_file, output_file) 

Here’s how you’d do it with a shell script:

#!/bin/bash sort autos.txt > sorted_autos.txt

Either way, we get the same results, which look like this:

I think you see that doing this with shell scripting is way faster and easier.

Finally, let’s see how shell scripting can help us with cloud operations.

Shell Scripting for Cloud Operations

Let’s say that you have a web server that’s running on either a VPS or a remote IoT device, and you want a list of IP addresses of clients that have accessed it, along with status codes and number of bytes transferred. Here’s a Python script that you might use for that:

#!/usr/bin/python

import sys
from dataclasses import dataclass

@dataclass(frozen = True)
class LogEntry:
    ip_address : str
    n_bytes : int
    status_code : int

def main(args):
    file_path = args[0]
    entries = parse_log_file(file_path)
    for e in entries:
        print(e)

def parse_log_file(file_path):
    try:
        with open(file_path) as log_file:
            return [parse_log_line(line) for line in log_file]
    except OSError:
        abort(f'File not found: {file_path}')

def parse_log_line(line):
    try:
        xs = line.split()
        return LogEntry(xs[0], int(xs[9]), int(xs[8]))
    except IndexError:
        abort(f'Invalid log file format: {file_path}')

def abort(msg):
    print(msg, file = sys.stderr)
    exit(1)

if __name__ == '__main__':
    main(sys.argv[1:])

Here’s a bash script that does the same thing:

#!/bin/bash

echo "ip address, status code, number of bytes"
cut -d" " -f 1,10,9 /var/log/httpd/access_log

That’s right. A simple, two-line shell script can take the place of that entire Python script. At any rate, the output of the shell script will look something like this:

You can also create shell scripts to a`utomate management of your cloud services. For example, here’s a script that can start or stop an EC2 instance on Amazon Web Services:

#!/bin/bash
read -p "Enter the EC2 instance ID: " INSTANCE_ID

read -p "Do you want to start or stop the instance? (start/stop): " ACTION

if [[ "$ACTION" == "start" ]]; then
  echo "Starting instance $INSTANCE_ID..."
  aws ec2 start-instances --instance-ids $INSTANCE_ID
elif [[ "$ACTION" == "stop" ]]; then
  echo "Stopping instance $INSTANCE_ID..."
  aws ec2 stop-instances --instance-ids $INSTANCE_ID
else
  echo "Oops! Please type 'start' or 'stop'."
fi

When you run the script, just type in the instance ID at the first prompt, and then type either start or stop at the second prompt. This is a lot easier than typing the entire aws command every time you need to start or stop an instance. You can automate almost any other aws task in the same manner.

Conclusion

To be sure, shell scripting has its limitations. For large, complex programs that require high performance, Python, or perhaps even a compiled language such as C, would be much better. But as I’ve just demonstrated, there are many times when bash scripting is definitely a much better choice.

To learn more about shell scripting, check out The Ultimate Linux Shell Scripting Guide by Donald.

Hi again, Shreyans here.

Big thanks to Donald for putting this together. If you liked the piece, you’ll love his YouTube channel where he walks through practical Linux topics.

He’s also written two other books worth your time:

Linux Service Management Made Easy with systemd

Mastering Linux Security and Hardening

That’s all for now. Hope you found something useful in this issue!

Cheers,

Shreyans

Join Generative AI In Action now with a Full Event Pass for just $239.99—40% off the regular price—with code FLASH40. 

BOOK TODAY AT $239.99 $399.99 

Three Reasons Why You Cannot Miss This Event: 

-Network with 25+ Leading AI Experts

-Gain Insights from 30+ Dynamic Talks and Hands-On Sessions

-Engage with Experts and Peers through 1:1 Networking, Roundtables, and AMAs

Act fast—this FLASH SALE is only for a limited number of seats!

CLAIM NOW- LIMITED SEATS

Welcome to the latest edition of CloudPro!

Today we will talk about:

⭐Masterclass

AI agents invade observability: snake oil or the future of SRE?

I created DevOps Interview Preparation Lab based on Interviews from Microsoft, Airbnb, Accenture, and others

QA's Dead: Where Do We Go From Here?

Convert OpenTelemetry Traces to Metrics using SpanMetrics Connector

Reduce Network Traffic Costs in Your Kubernetes Cluster

🔍Secret Knowledge

SQLite on Rails

Just use Postgres

Why I still Self-Host my Servers

Essays on programming I think about a lot

A detailed guide to cron jobs

⚡Techwave

How Google fine-tuned Gemma model for Flipkart

AWS has launched Console to Code: tool that generates code

Bring your conversations to WhatsApp with AWS End User Messaging Social

Introducing pipe syntax in BigQuery and Cloud Logging

GCloud Database Center: AI-powered, unified fleet management solution preview now open to all customers

🛠️Hackhub

agnost-gitops: Open source GitOps platform running on Kubernetes clusters

kube-downscaler: Scale down Kubernetes deployments after work hours

AWS Mine: honey token system designed to generate AWS access keys

TinyStatus: A simple, customizable status page generator that monitors and displays the status of services on a responsive web page.

Litecli: A command-line client for SQLite databases, featuring auto-completion and syntax highlighting.

Cheers,

Shreyans Singh

Editor-in-Chief

Looking to build, train, deploy, or implement Generative AI? 

Meet Innodata — offering high-quality solutions for developing and implementing industry-leading generative AI, including: 

With 5,000+ in-house SMEs and expansion and localization supported across 85+ languages, Innodata drives AI initiatives for enterprises globally. 

Learn More

⭐MasterClass: Tutorials & Guides

AI agents invade observability: snake oil or the future of SRE?

This article explores how AI, particularly agentic AI, is transforming the field of observability and monitoring. Traditional monitoring tools use dashboards, alerts, and data insights to help developers and operators manage system health, but new AI agents are designed to act more like team members. These agents, powered by large language models (LLMs), can analyze operational data and automate tasks like incident response and maintenance.

I created DevOps Interview Preparation Lab based on Interviews from Microsoft, Airbnb, Accenture, and others

This hands-on lab is designed to help you prepare for DevOps interviews by walking you through key tools like Python web apps, Docker, Kubernetes, Helm Charts, GitHub Actions for CI/CD, and Ingress Controllers. It's practical, not theory-based, and helps you build a project from scratch through containerization, deployment, and CI/CD setup.

QA's Dead: Where Do We Go From Here?

 The concept of traditional QA (Quality Assurance) has evolved, shifting responsibility for software quality from a separate QA team to developers themselves. In the old model, QA was a distinct stage that came after development, causing delays, inefficiencies, and higher costs due to late bug detection. Now, with agile methodologies and advanced tooling, testing is integrated throughout the development process. Developers take ownership of quality, using tools like automated testing, CI/CD pipelines, and instant feedback mechanisms. QA isn't dead; instead, it has become an essential part of every developer's role, with QA professionals either moving into technical automation roles or higher-level strategic positions.

Convert OpenTelemetry Traces to Metrics using SpanMetrics Connector

The SpanMetrics Connector in OpenTelemetry converts trace data into actionable metrics, which is useful when robust tracing is in place but metrics instrumentation is lacking. It works by extracting metrics from spans (units of trace data) and aggregating them into key performance indicators like request counts, errors, and durations. This unified approach simplifies observability by reducing the need for separate instrumentation for traces and metrics. By configuring the connector, developers can easily generate custom metrics, optimize system performance, and enhance monitoring without increasing overhead or complexity.

Reduce Network Traffic Costs in Your Kubernetes Cluster

To reduce network traffic costs in a Kubernetes cluster, it's important to minimize cross-availability zone (AZ) traffic, which can increase latency and lead to higher data transfer costs. Strategies to reduce this include intelligent node placement, ensuring related pods are located in the same AZ to avoid unnecessary data transfer. Topology-aware routing ensures traffic is directed within the same AZ, while using local persistent volumes keeps data close to the pods accessing it. Pod topology spread constraints help evenly distribute pods across zones, further minimizing cross-AZ communication and improving both performance and cost-efficiency.

🔍Secret Knowledge: Learning Resources

SQLite on Rails

Running SQLite on Rails can provide good performance, but out-of-the-box it isn’t optimized for high-concurrency production environments. This is mainly due to SQLite’s single-write locking mechanism, which can cause errors and bottlenecks when multiple threads attempt to write at the same time. However, by fine-tuning configurations—like setting immediate transactions, adjusting busy timeouts, and managing connection pools—Rails apps can achieve resilient performance. Advanced techniques, such as using custom busy handlers and write-ahead logging (WAL), further enhance concurrency and minimize delays, making SQLite on Rails a viable production option.

Just use Postgres

When building a new application requiring persistent storage, Postgres should be your default choice. It highlights why other databases might not be ideal: SQLite is great for single-machine apps but limited for distributed systems, NoSQL databases like MongoDB require rigid access patterns, and newer databases like XTDB pose long-term risks. Postgres offers flexibility, scalability, and a rich ecosystem of tools, making it a reliable and efficient choice for most web applications without the trade-offs of other databases.

Why I still Self-Host my Servers

Two reasons: independence and learning. Hosting own services lets the author stay free from corporate control and subscriptions while teaching valuable skills that benefit his career as a software engineer. From managing a Proxmox cluster and Pi-Hole DNS servers to troubleshooting outages and hardware issues, the experience forces him to dive deeper into the technical aspects of system administration. This continuous learning has proven useful in handling complex distributed systems at work. Despite the challenges, like hardware failures and occasional crashes, the lessons learned make it worthwhile.

Essays on programming I think about a lot

This passage highlights several key programming essays that have deeply impacted the author's thinking and engineering approach. These essays cover various topics, from understanding complex systems, choosing stable technology, and managing abstractions, to hiring strong engineering teams and designing scalable distributed systems. The recurring theme is thoughtful, pragmatic decision-making in software engineering, advocating for simplicity, clear abstraction boundaries, and understanding the deeper layers of technology. Each essay provides timeless insights, shaping the author's work habits, and the list invites others to explore and reflect on these ideas for themselves.

A detailed guide to cron jobs

A cron job is a scheduled task or command in Unix-based systems, like Linux and macOS, that automates repetitive processes such as backups, email sending, or database updates. Cron jobs use a specific time-based syntax to determine when and how often the task should run. This guide explains how to set up, edit, and manage cron jobs, including the syntax, adding new jobs, and checking their logs. It also covers methods for monitoring cron jobs, such as using logs, monitoring tools, and email alerts to ensure tasks run as expected without system issues.

⚡ TechWave: Cloud News & Analysis

How Google fine-tuned Gemma model for Flipkart

The blog describes the process of fine-tuning Gemma, an instruction-tuned AI model, for a conversational shopping assistant. It starts with data preparation using a subset of Flipkart’s product catalog, filtering for clothing items and generating Q&A pairs based on product details. Fine-tuning was achieved using LoRA, a parameter-efficient method, with multiple iterations on both pre-trained and instruction-tuned models. The fine-tuning was scaled using multi-GPU setups on Google Kubernetes Engine (GKE). Hyperparameter tuning was also crucial to optimize model performance, ensuring the chatbot provides accurate, contextual responses.

AWS has launched Console to Code: tool that generates code

AWS has launched "Console to Code," a tool that simplifies the process of moving from prototyping in the AWS Management Console to writing production-ready code. This tool automatically captures actions taken in the console and generates code in formats like CLI, CloudFormation, and CDK, following AWS best practices. It helps users quickly create reusable, automation-friendly code without needing to manually write it, streamlining the transition from console use to Infrastructure-as-Code (IaC). This service is available for key AWS services like EC2, VPC, and RDS.

Bring your conversations to WhatsApp with AWS End User Messaging Social

AWS has introduced "End User Messaging Social," allowing developers to send messages to their users on WhatsApp, the world’s most popular messaging app. With this tool, developers can create rich, interactive messaging experiences that include multimedia content. WhatsApp can now be used alongside SMS and Push notifications, giving businesses multiple ways to reach their audience. Setting up WhatsApp messaging is easy, with options to create a new WhatsApp Business Account or link an existing one, all within the AWS console.

Introducing pipe syntax in BigQuery and Cloud Logging

Google Cloud has introduced a new "pipe syntax" in BigQuery and Cloud Logging, designed to simplify log data queries. This new syntax uses a pipe symbol (|>) to break down complex SQL queries into clear, easy-to-read steps, improving the readability and writability of log analysis tasks. With this innovation, users can quickly filter, aggregate, and explore log data, making it easier to extract insights. BigQuery’s enhanced performance features, like faster numeric search indexes and better handling of JSON data, further streamline log analysis. Pipe syntax is now available in preview.

GCloud Database Center: AI-powered, unified fleet management solution preview now open to all customers

Google Cloud has launched Database Center, an AI-powered solution that simplifies managing large, complex database fleets. It provides a unified interface for monitoring and optimizing databases like Cloud SQL, AlloyDB, and Spanner. Database Center helps businesses detect and address performance and security issues with proactive recommendations, ensuring smoother operations and better compliance with industry standards. It also includes AI-powered chat for quick troubleshooting and optimization insights, allowing users to improve performance, reduce costs, and strengthen security across their entire database landscape.

🛠️HackHub: Best Tools for Cloud

agnost-gitops: Open source GitOps platform running on Kubernetes clusters

Agnost GitOps is an open-source platform for continuous deployment (CD) on Kubernetes clusters. It automates the process of building, deploying, and managing applications by connecting your GitHub, GitLab, or Bitbucket repository. When you push new code, Agnost builds a Docker image using Kaniko and deploys it to your Kubernetes cluster.

kube-downscaler: Scale down Kubernetes deployments after work hours

Kube-downscaler is a Kubernetes tool designed to automatically scale down or pause workloads (like Deployments, StatefulSets, and HorizontalPodAutoscalers) during non-work hours, helping organizations save on cloud costs. It operates based on a configurable schedule of uptime and downtime, using Kubernetes annotations or command-line options.

AWS Mine: honey token system designed to generate AWS access keys

The "aws-mine" project is a honey token system designed to generate AWS access keys that can be strategically placed in various locations to lure and detect potential attackers. If someone attempts to use these keys, the system sends a notification within about four minutes, allowing you to investigate the source and assess whether the asset has been compromised.

TinyStatus: A simple, customizable status page generator that monitors and displays the status of services on a responsive web page.

It checks the status of HTTP endpoints, pings hosts, and monitors open ports, displaying results on a clean and responsive web page. The system is configured using YAML files, and it supports both light and dark themes, as well as incident history tracking.

Litecli: A command-line client for SQLite databases, featuring auto-completion and syntax highlighting.

Upon first use, LiteCLI generates a configuration file that can be customized for user preferences. It streamlines database interactions by predicting commands and formatting output, enhancing the command-line experience for SQLite users.

⭐Masterclass

[Sponsored] Become an AI Powered Professional. Free 3-hour ChatGPT and AI workshop for Professionals

Preemptible pods: Optimizing Kubernetes node utilization

Supercharge Your Kubernetes Workflow with Essential Tools: Starship, Kubectx, Kubecolor, and K9s

Exploring Helm template dictionary objects: Syntax evolution and best practices

Dockerizing a Golang API with MySQL and adding Docker Compose Support

Karmada: Deep dive into managing multiple AKS clusters

🔍Secret Knowledge

Zero Downtime Deployment in AWS with Tofu

Cron Jobs on Linux

How To Run Migrations Across 2,800 Microservices

Transform AWS exam generator architecture to open source

How to Run WebAssembly on Amazon EKS

⚡Techwave

Chrome Vulnerability Reward Program (VRP) has updated its rewards

How misconfigured AWS IAM roles using GitLab's OpenID Connect (OIDC) can allow unauthorized users to assume roles

Preview Release of the Migration Tool for the AWS SDK for Java 2.x

Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2

Unlock 1 Million RPS: Experience Triple the Speed with Valkey

🛠️Hackhub

kubeai: Private Open AI on Kubernetes

cyphernetes: A Kubernetes Query Language

chartdb: Free and open-source database diagrams editor, visualize and design your DB with a single query.

stack-auth: Open-source Auth0/Clerk alternative

mariadb-operator: Run and operate MariaDB in a cloud native way

💡Get 30% off on CloudPro Book of the Week: AWS DevOps Simplified

Cheers,

Shreyans Singh

Editor-in-Chief

Last Chance! For the next 48 hours only, save $150 on your full event pass!

Imagine being part of 10+ Power Talks, 12+ Hands-On Workshops, and 3 Interactive Roundtables—while networking with 30+ top industry leaders and hundreds of tech professionals from across the globe. This is your opportunity to dive into cutting-edge AI solutions at the Generative AI in Action 2024 Conference.
 
It's all happening on November 11-13 (LIVE, Virtual) - prices increase permanently on Saturday!

BOOK YOUR SEAT NOW before prices go up!

Use code LASTCHANCE40 at checkout

BOOK NOW AT $399.99 $239.99

⭐MasterClass: Tutorials & Guides

Preemptible pods: Optimizing Kubernetes node utilization

Preemptible Pods in Kubernetes enable efficient resource management by allowing you to assign priorities to different workloads through pod priority and preemption mechanisms. This means that critical applications are guaranteed the resources they need because higher-priority pods can preempt, or evict, lower-priority ones when resources are scarce. By implementing PriorityClasses and configuring pods accordingly, you ensure that essential services remain responsive and that your cluster optimizes node utilization.

Supercharge Your Kubernetes Workflow with Essential Tools: Starship, Kubectx, Kubecolor, and K9s

To enhance your Kubernetes workflow, using tools like Starship, Kubectx, Kubecolor, and K9s can significantly improve efficiency. Starship provides a customizable, fast shell prompt that shows key info like cluster and namespace, while Kubectx and Kubens allow quick switching between clusters and namespaces. Kubecolor adds color to kubectl output for better readability, and K9s offers a terminal-based UI to manage and visualize Kubernetes resources easily.

Exploring Helm template dictionary objects: Syntax evolution and best practices

Helm, the Kubernetes package manager, uses dictionary objects in its templating system to manage key-value pairs for application deployment. Initially, Helm syntax allowed for creating dictionaries in a single line, but this became cumbersome when handling many properties. Over time, a more efficient syntax evolved, using the `set` function to incrementally add properties to a dictionary without recreating it. Best practices for using Helm dictionaries include adding properties incrementally, avoiding reassignment to prevent data loss, maintaining consistent naming conventions, and thoroughly testing templates to ensure correct Kubernetes manifest generation.

Dockerizing a Golang API with MySQL and adding Docker Compose Support

Dockerizing a Golang API with MySQL simplifies the process of developing and testing APIs locally by containerizing both the API and database. First, you create a Dockerfile for the Go API using best practices like lightweight base images, multi-stage builds, creating a binary, and optimizing Docker layers. This ensures a smaller and more efficient container. Then, to streamline managing both the API and MySQL containers, Docker Compose is used. A `compose.yml` file sets up both services, ensuring the API only starts once the MySQL database is ready, avoiding connection issues. This setup makes local development smoother and easier to replicate.

Karmada: Deep dive into managing multiple AKS clusters

Karmada (Kubernetes Armada) is a tool that simplifies managing multiple AKS (Azure Kubernetes Service) clusters by treating them like a single entity. It helps deploy applications across clusters while handling tasks like scheduling, resource propagation, and ensuring consistency. Karmada’s components—such as the API Server, Controller Manager, Scheduler, and Agent—work together to automate the deployment process. It supports advanced strategies like multi-cluster deployments, disaster recovery, and canary releases.

🔍Secret Knowledge: Learning Resources

Zero Downtime Deployment in AWS with Tofu

Zero Downtime Deployment in AWS is a strategy to update applications without causing service interruptions. By leveraging tools like OpenTofu, Terraform, and AWS SAM, developers can ensure seamless updates. Techniques like instance refreshes in Auto Scaling Groups (using OpenTofu), immutable infrastructure (Terraform + Ansible), and advanced deployment strategies like Blue/Green and Canary deployments enable applications to be updated while keeping them available to users. These approaches allow for gradual testing, automated rollbacks, and maintaining reliability.

Cron Jobs on Linux

Cron jobs in Linux are scheduled tasks that automate running scripts or commands at specific times or intervals, managed by the cron daemon. Common use cases include backups, updates, and system health checks. Users can create, view, or edit cron jobs using the `crontab` command. Cron jobs are defined using a simple time-based syntax, where each job can run on a specific schedule (e.g., hourly, daily, or weekly). Cron jobs can be user-specific or system-wide, and their syntax supports flexible timing options like ranges, lists, and intervals.

How To Run Migrations Across 2,800 Microservices

To handle migrations across 2,800 microservices, we use a centrally driven approach where a single team manages the entire process. This allows us to keep libraries up-to-date, maintain consistency, and automate the bulk of the changes, reducing coordination overhead and minimizing risks of failure. Our strategy relies on a monorepo structure, consistent technology (like Go), and powerful mass deployment tooling. We start by wrapping old libraries, automate common updates, handle edge cases manually, and control rollouts via config changes to ensure smooth transitions without downtime.

Transform AWS exam generator architecture to open source

In this series, we aim to transform a serverless AWS architecture for an exam generator app into an open-source version. The original solution helps educators create curriculum-aligned assessments quickly, while students can take personalized quizzes with instant feedback. We'll replace key AWS services like Cognito, Lambda, DynamoDB, and Fargate with open-source alternatives and host everything on a Kubernetes cluster.

How to Run WebAssembly on Amazon EKS

The article outlines the process of setting up a Wasm environment on Amazon EKS using tools like HashiCorp Packer and Terraform to create custom Amazon Machine Images (AMIs) and manage the infrastructure. It details how to build an EKS cluster, deploy example workloads using different Wasm runtimes (Spin and WasmEdge), and check if everything is working correctly. Finally, it offers instructions for cleaning up the resources after running the applications.

⚡ TechWave: Cloud News & Analysis

Chrome Vulnerability Reward Program (VRP) has updated its rewards

Google's Chrome Vulnerability Reward Program (VRP) has updated its reward structure to encourage deeper research into Chrome's security vulnerabilities. As Chrome becomes more secure, finding impactful bugs has become harder. The new structure separates memory corruption bugs from other vulnerability types and offers higher rewards for more complex, well-documented reports, such as those demonstrating remote code execution (RCE) or memory corruption. The top reward for an RCE in a non-sandboxed process is now $250,000. These changes aim to incentivize thorough and high-quality bug reporting, ensuring Chrome remains secure.

How misconfigured AWS IAM roles using GitLab's OpenID Connect (OIDC) can allow unauthorized users to assume roles

The article by Nick Frichette explains how misconfigured AWS IAM roles using GitLab's OpenID Connect (OIDC) can allow unauthorized users to assume roles. This occurs when the trust policy lacks restrictions on which specific GitLab groups or projects can access the role. By default, the AWS Console creates a vulnerable trust policy, making it possible for any GitLab user to exploit the misconfiguration. The article walks through how to generate a GitLab OIDC token and use it to assume a misconfigured role, highlighting the risks of default settings in AWS.

Preview Release of the Migration Tool for the AWS SDK for Java 2.x

AWS has released a preview of a migration tool to help developers transition from AWS SDK for Java 1.x to 2.x, as 1.x is now in maintenance mode. This tool uses OpenRewrite, an open-source code refactoring tool, to automate much of the migration process. It currently supports most service SDK clients, except for AmazonS3Client, TransferManager, and DynamoDBMapper, and helps reduce the time and effort needed for the upgrade. Developers can use this tool with Maven or Gradle projects, choosing between preview (dryRun) or actual (run) modes to apply the changes.

Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2

Amazon’s Business Data Technologies (BDT) team is migrating from Apache Spark to Ray on Amazon EC2 to handle exabyte-scale data more efficiently. The switch is driven by the need to reduce data processing costs and time for their large business intelligence datasets. Apache Spark, though powerful, had started to show limitations with scalability and performance as their data grew. Ray, initially known for machine learning tasks, offered a more flexible and cost-effective solution with its distributed compute capabilities, reducing processing costs by 82% and improving data processing speeds significantly.

Unlock 1 Million RPS: Experience Triple the Speed with Valkey

Valkey 8.0, set for release in September 2024, introduces a new multi-threaded architecture that significantly boosts performance, increasing throughput by 230% to over 1 million requests per second and reducing latency by nearly 70%. This is achieved through an innovative I/O threading system, where dedicated worker threads handle tasks like reading commands and writing responses, freeing up the main thread to focus on executing commands. Valkey 8.0 also supports larger shards, improving performance for workloads that don't scale well horizontally, but comes with trade-offs like increased complexity in managing larger nodes.

🛠️HackHub: Best Tools for Cloud

kubeai: Private Open AI on Kubernetes

KubeAI is an open-source tool that allows users to run AI models like LLMs (Large Language Models), embeddings, and speech-to-text on Kubernetes. It provides an API compatible with OpenAI, letting users serve and scale models like Whisper and vLLM across CPU, GPU, and soon TPU infrastructure.

cyphernetes: A Kubernetes Query Language

Cyphernetes is a query language for Kubernetes inspired by Cypher (from Neo4j) that simplifies managing Kubernetes resources. Instead of complex `kubectl` commands, Cyphernetes lets users perform operations like finding and modifying deployments, services, and ingresses with clear, SQL-like syntax.

chartdb: Free and open-source database diagrams editor, visualize and design your DB with a single query.

ChartDB is an open-source, web-based tool for creating and editing database diagrams. With a single "Smart Query," users can instantly visualize their database schema, making it easy to understand and document database structures. It supports multiple databases like PostgreSQL, MySQL, and SQLite.

stack-auth: Open-source Auth0/Clerk alternative

ChartDB is a free, open-source tool for creating and editing database diagrams. It allows users to instantly visualize their database schema with a single query and supports databases like PostgreSQL, MySQL, and SQLite. Users can interactively edit schemas, export SQL scripts, and even use AI to generate migration scripts for switching between databases.

mariadb-operator: Run and operate MariaDB in a cloud native way

The MariaDB Operator allows users to manage MariaDB databases in a cloud-native environment using Kubernetes. It simplifies tasks like deploying and operating MariaDB instances through Custom Resource Definitions (CRDs), enabling features like high availability, automated backups, and flexible storage options.

⭐Masterclass

Infamous DevOps roadmap

Kubernetes Open Source Limits & Requests Configuration Optimization

A guide to modern Kubernetes network policies

Using Python Virtual Environments in Docker

How to terminate Go programs elegantly – a guide to graceful shutdowns

🔍Secret Knowledge

How Meta Enforces Purpose Limitation at Scale

Why I Use Nim Instead of Python for Data Processing

Convert OpenTelemetry Traces to Metrics using SpanMetrics Connector

What happens when bucket.grantRead() in AWS CDK

Preventing the Risk of Request Collapsing in Web Caching

⚡Techwave

Grafana Labs Soars Past $250M ARR and 5,000 Customers, Completes $270M funding round, and Named a Leader in the Gartner Magic Quadrant for Observability Platforms

CockroachDB retires its free "Core" version

OpenMetrics is Archived, Merged into Prometheus

Announcing Storage Browser for Amazon S3 for your web applications (alpha release)

Juniper jumps into Wi-Fi 7 with enterprise switches, access points

🛠️Hackhub

Kardinal: lightest-weight way to spin up dev and test environments in Kubernetes

Kubeblocks: control plane software that runs and manages databases, message queues on K8s.

Flipt: Enterprise-ready, GitOps enabled, CloudNative feature management solution

Kubecolor: Colorize your kubectl output

AWS-mine: AWS honey token manager

💡Recommended Learning: Continuous Integration Mastery with Jenkins

Cheers,

Shreyans Singh

Editor-in-Chief

⭐MasterClass: Tutorials & Guides

Infamous DevOps roadmap

This roadmap provides community-driven guides, resources, and roadmaps to help developers grow in their careers, focusing on different fields like DevOps, backend development, and various programming languages. It offers step-by-step instructions for learning new skills, tracking progress, and staying updated with industry best practices.

Kubernetes Open Source Limits & Requests Configuration Optimization

This article provides a step-by-step guide on using Kexa, an open-source tool for optimizing Kubernetes resource limits and requests through monitoring and alerting, with Grafana for visualization. It explains how to install Kexa using Helm, set up necessary credentials, connect it to databases like Postgres or MySQL, and configure rules to monitor CPU and memory consumption. It then walks through the setup of a Grafana dashboard to display and optimize pod performance.

A guide to modern Kubernetes network policies

In Kubernetes, network policies are rules that control traffic flow between pods in a cluster. They define which traffic is allowed to enter (ingress), exit (egress), or move between pods, helping secure communication within the cluster. These policies fall into two categories based on the OSI model: Layer 4 (L4) policies, which control traffic using IP addresses and ports, and Layer 7 (L7) policies, which offer finer control at the application level (e.g., HTTP routes). By combining both, Kubernetes can implement robust, zero-trust security models.

Using Python Virtual Environments in Docker

The author explains that despite the trend of simplifying Python Docker workflows by avoiding virtual environments, they continue using them for several key reasons. Virtual environments provide predictability, a well-defined structure, and consistency across projects, which simplifies communication and management in team environments. By isolating the Python environment, it helps prevent complex import issues and makes the codebase more reliable and easier to debug.

How to terminate Go programs elegantly – a guide to graceful shutdowns

By handling termination signals like SIGTERM, Go applications can stop accepting new requests while allowing in-flight processes to finish, utilizing tools such as `signal.NotifyContext` and `sync.WaitGroup` to manage concurrency. This approach helps maintain data integrity and smooth operations during shutdowns, particularly in orchestrated environments where unexpected terminations can otherwise lead to issues.

🔍Secret Knowledge: Learning Resources

How Meta Enforces Purpose Limitation at Scale

Meta enforces purpose limitation at scale using its Privacy Aware Infrastructure (PAI) through technologies like Policy Zones. Policy Zones ensure that data is processed only for its intended purposes by labeling and tracking data assets across systems. It integrates real-time checks during data flow, preventing unauthorized uses by monitoring the movement and processing of data in different environments like function-based or batch-processing systems. This approach provides granular control over data use while scaling across Meta’s complex infrastructure.

Why I Use Nim Instead of Python for Data Processing

The author chooses Nim over Python for data processing because it offers the simplicity of Python with the speed of C, making it ideal for handling large datasets without complex optimization. In a comparison of processing a 150 MB genome file, Nim significantly outperforms Python, running 30 times faster with nearly identical code. While Nim requires a few syntax changes, such as using `var` for variables and `echo` for output, its faster compilation and execution make it a powerful alternative for tasks like analyzing DNA sequences.

Convert OpenTelemetry Traces to Metrics using SpanMetrics Connector

The SpanMetrics Connector in OpenTelemetry allows you to convert trace data into actionable metrics, addressing the lack of native metrics support in some languages. It works by aggregating key metrics like request counts, errors, and durations (R.E.D metrics) from trace spans. By configuring it in the OpenTelemetry Collector, you can generate useful performance insights without adding extra instrumentation for metrics.

What happens when bucket.grantRead() in AWS CDK

When you call `bucket.grantRead()` in AWS CDK, it grants read permissions to an IAM role or user by either updating identity-based policies (attached to the IAM principal) or resource-based policies (attached to the S3 bucket). If the IAM role was created within the same CDK stack, identity-based policies are updated. However, if the role or bucket is just a reference (using interfaces like `IRole` or `IBucket`), CDK cannot modify existing policies, and the grant may not work.

Preventing the Risk of Request Collapsing in Web Caching

Request collapsing is a caching feature where multiple identical requests for the same resource are combined, so only one is sent to the origin server to reduce load. However, this can cause security issues when dealing with sensitive data, as the response to the first request might be mistakenly sent to other users who made the same request. Even if a server uses `Cache-Control: no-cache`, request collapsing may still send cached responses to multiple users. To prevent this, it's crucial to use strict cache policies, such as disabling caching for certain patterns and configuring both the cache and origin server to avoid caching sensitive data.

⚡ TechWave: Cloud News & Analysis

Grafana Labs Soars Past $250M ARR and 5,000 Customers, Completes $270M Primary and Secondary Transaction, and Named a Leader in the Gartner® Magic Quadrant™ for Observability Platforms

CockroachDB retires its free "Core" version

CockroachDB is evolving its self-hosted offering by retiring the free "Core" version and consolidating all users into a single "Enterprise" version that provides full access to its advanced features. This change, starting with version 24.3, ensures that individuals, students, and small businesses (under $10M annual revenue) can still use CockroachDB Enterprise for free with community support, while larger businesses will need a paid license.

OpenMetrics is Archived, Merged into Prometheus

The OpenMetrics project, originally created to spin off Prometheus' metrics format into an independent specification, has been archived and merged back into Prometheus as of July 2024. While OpenMetrics aimed to become a universal format for exporting metrics, it struggled to gain adoption outside the Prometheus ecosystem, where Prometheus had already become the de facto standard for cloud-native observability.

Announcing Storage Browser for Amazon S3 for your web applications (alpha release)

 Amazon S3 has released an alpha version of Storage Browser for S3, an open-source component that lets web applications provide a simple interface for users to browse, download, and upload S3-stored data. It integrates with AWS Amplify's JavaScript and React libraries, allowing developers to control access based on user identity and customize the design to fit their app's branding.

Juniper jumps into Wi-Fi 7 with enterprise switches, access points

Juniper has introduced new EX Series switches and Mist Wi-Fi 7 access points for enterprise wireless networks, offering higher speeds, lower latency, and broader range. The EX4400 switches support both Wi-Fi 6E and Wi-Fi 7 and are managed via the AI-powered Mist Cloud, which helps detect and resolve network issues. Juniper’s new AP47 Series access points offer advanced features like dual-5GHz or dual-6GHz operation and AI-based channel management.

🛠️HackHub: Best Tools for Cloud

kardinal: lightest-weight way to spin up dev and test environments in Kubernetes

Create lightweight, temporary development environments within a shared Kubernetes cluster, making testing and development more efficient. It allows developers to spin up tailored, on-demand "flows"—ephemeral environments that use minimal resources by deploying only the necessary services for feature development.

kubeblocks: control plane software that runs and manages databases, message queues on K8s.

KubeBlocks is an open-source control plane software designed to simplify the management of multiple database engines on Kubernetes (K8s). It uses a unified set of APIs to manage various types of databases, such as MySQL, PostgreSQL, Redis, and Kafka, reducing the need to learn individual database operators.

flipt: Enterprise-ready, GitOps enabled, CloudNative feature management solution

Flipt is a cloud-native, GitOps-enabled feature management solution designed to help organizations separate feature releases from deployments, allowing for safer, more controlled updates. It can be integrated into existing infrastructure to avoid third-party latency and is built with high-performance DevOps teams in mind.

kubecolor: Colorize your kubectl output

Kubecolor is a simple wrapper for the kubectl command-line tool that adds color to its output, making it easier to read and interpret. It enhances the standard kubectl by colorizing logs, tables, and other outputs without changing the actual content. Kubecolor supports custom themes, including options for light backgrounds and colorblind-friendly themes.

aws-mine: AWS honey token manager

aws-mine is a project designed to create "honey tokens" for AWS, which are fake AWS access keys placed in various locations to lure potential attackers. If someone uses these keys, the system sends a notification within about four minutes, allowing you to investigate the possible compromise. Built with AWS Amplify for easy deployment, users can manage their access through Amazon Cognito and receive alerts via Amazon SNS when the keys are accessed.

⭐Masterclass

A Guide to Kubernetes Network Policies

Dockerfile Instructions - ADD vs. COPY

How to add new worker node to existing Kubernetes cluster

How I Reduced Docker Image Size from 588 MB to Only 47.7 MB

Ambient mesh: Can sidecar-less Istio make your application faster?

🔍Secret Knowledge

Oops, I Deleted the AWS Auth Roles

Rising Incidents on Git Platforms

How Postgres stores data on disk

How We Integrate a New Service in Under 1 Hour for 25 Clusters

Eleventeen ways to delete an AWS resource

⚡Techwave

European grocery store becomes cloud services provider

IBM acquires Kubecost

Introducing Pulumi Insights 2.0

Linus Torvalds advises open-source developers to pursue meaningful projects, not hype

JFrog Extends GitHub Alliance to Provide Unified Dashboard

🛠️Hackhub

Apeman: AWS attack path management tool

Cyphernetes: A Kubernetes Query Language

Desed: A command-line tool for complex sed scripts

Kueue: Kubernetes-native Job Queueing

AWS CloudFormation Starterkit

💡Recommended Reading: Implementing GitOps with Kubernetes

Cheers,

Shreyans Singh

Editor-in-Chief

⭐MasterClass: Tutorials & Guides

A Guide to Kubernetes Network Policies

In Kubernetes, network policies control the traffic between pods, ensuring secure communication within the cluster. There are two main types: Layer 4 (L4) and Layer 7 (L7) policies. L4 policies manage traffic at the transport layer (e.g., TCP/UDP) based on IP addresses and ports, while L7 policies operate at the application layer (e.g., HTTP) with more fine-grained control over communication between services. L7 policies often require a service mesh like Linkerd, which adds features like mutual TLS (mTLS) for encrypted communication.

Dockerfile Instructions - ADD vs. COPY

`COPY` is simple and secure, only transferring files from the local build context to the image. In contrast, `ADD` offers extra functionality, such as downloading files from URLs or automatically extracting compressed archives. However, this added flexibility introduces complexity and potential security risks. Best practice recommends using `COPY` for most cases due to its straightforwardness, reserving `ADD` for situations where its unique features are necessary.

How to add new worker node to existing Kubernetes cluster

To add a new worker node to an existing Kubernetes cluster, start by setting up a new Ubuntu 24.04 instance and configuring its hostname and `/etc/hosts` file. Disable swap memory, load necessary kernel modules, and install containerd as the container runtime. Add the Kubernetes APT repository, then install Kubernetes components like kubeadm, kubelet, and kubectl. On the control plane node, generate a kubeadm join command with a token. Run this command on the new worker node to join the cluster. Finally, verify the addition by checking the nodes from the control plane using `kubectl get nodes`.

How I Reduced Docker Image Size from 588 MB to Only 47.7 MB

To significantly reduce a Docker image size, using multi-stage builds is key. In this case, a Flask app's image size was reduced from 588 MB to just 47.7 MB by switching to the lightweight Python 3.9-alpine image and using a multi-stage build approach. Multi-stage builds allow you to separate the build and runtime environments, keeping only essential runtime dependencies in the final image. Additionally, minimizing the number of layers by combining commands, using a `.dockerignore` file to exclude unnecessary files, and optimizing the Dockerfile structure contributed to this impressive 91.89% reduction.

Ambient mesh: Can sidecar-less Istio make your application faster?

Ambient mode in Istio, introduced in 2022, allows a sidecar-less architecture that can sometimes make applications faster. In traditional service meshes, adding latency is expected, but tests with ambient mode showed slightly improved performance in some cases, like the Bookinfo application's details service. This is partly because of more efficient connection handling and reduced syscalls in ambient mode, which offsets the overhead of extra hops via lightweight ztunnels.

🔍Secret Knowledge: Learning Resources

Oops, I Deleted the AWS Auth Roles

The author, while managing an EKS (Elastic Kubernetes Service) cluster using Terraform, accidentally deleted the AWS authentication roles, which are crucial for accessing the cluster. This resulted in losing access to the EKS cluster. The fix involved manually restoring access by modifying the EKS API access configuration via the AWS Console, re-adding the necessary admin roles, and regenerating the `aws-auth` config map.

Rising Incidents on Git Platforms

In 2023, incidents affecting popular DevOps platforms like GitHub, Bitbucket, GitLab, and Jira increased, with issues such as RepoJacking, security vulnerabilities, and performance disruptions. GitHub saw a rise in attacks, with hackers exploiting vulnerabilities and hosting malware. Atlassian products like Bitbucket and Jira faced security flaws, with Jira experiencing a significant increase in incidents. GitLab suffered from performance issues and security breaches, including a major Proxyjacking attack.

How Postgres stores data on disk

Postgres stores data on disk in a well-organized, file-based structure within a directory, typically located at `/var/lib/postgresql/data`. Inside this directory, you'll find folders like `base/`, where actual database data for each database is stored, and `pg_wal/`, which holds the Write-Ahead Log (WAL) files that help recover data after crashes. Each table and database object is ultimately represented by files in these directories. PostgreSQL uses clever abstractions to manage data, such as snapshots for transactions, dynamic shared memory for handling multiple processes, and special mechanisms like tablespaces for physically separating certain data.

How We Integrate a New Service in Under 1 Hour for 25 Clusters

The article describes how a team integrated a new service called Otterize across 25 clusters in under an hour, emphasizing that while the technical setup was quick, the lengthy licensing process took over four months. The integration involved automating several steps using GitOps and tools like Argo CD to avoid manual errors. Key tasks included creating an organization and environment, inviting users, integrating with Kubernetes, securely managing credentials, and deploying the setup through a script.

Eleventeen ways to delete an AWS resource

Our goal is to reduce AWS costs, but the deletion methods vary widely, often leaving users frustrated. They categorize deletion patterns, from simple one-click deletes to more complex confirmations that require typing specific phrases or acknowledging consequences. Ultimately, AWS should standardize its deletion processes to improve user experience and security, and they call for more data on user behavior during these actions.

⚡ TechWave: Cloud News & Analysis

European grocery store becomes cloud services provider

Lidl, through its parent company Schwarz Group, unintentionally entered the competitive world of cloud computing when it built its own cloud system in 2021 to meet internal needs. As other German businesses sought alternatives to U.S. and Chinese cloud providers, Schwarz Group recognized a demand for data services with a focus on European data privacy standards. This led to the creation of Schwarz Digits, which now provides cloud and cybersecurity services, attracting major clients like SAP and Bayern Munich. While competing with giants like Amazon and Google, Schwarz Digits differentiates itself with a focus on digital sovereignty and data protection.

IBM acquires Kubecost

IBM has acquired Kubecost, a startup that helps companies optimize and monitor their Kubernetes clusters for cost efficiency. Kubecost, known for its widely adopted Kubernetes cost management tool and its open-source project OpenCost, will enhance IBM’s FinOps capabilities. Kubecost will likely be integrated into IBM's FinOps Suite and potentially its OpenShift platform.

Introducing Pulumi Insights 2.0

Pulumi Insights 2.0 expands beyond just Pulumi-managed infrastructure to provide visibility into all cloud resources, offering powerful tools for assessing security, efficiency, and management. It introduces new features like comprehensive infrastructure scanning, visual explorers, and dashboards to help organizations manage their cloud environments more effectively. Insights 2.0 integrates with Pulumi’s Infrastructure-as-Code (IaC) tools, making it easier to bring unmanaged infrastructure under IaC.

Linus Torvalds advises open-source developers to pursue meaningful projects, not hype

At the Open Source Summit Europe, Linus Torvalds encouraged open-source developers to focus on meaningful projects rather than chasing trends and hype. While discussing the latest Linux kernel updates, he emphasized that progress in Linux remains steady, even if not always exciting, with a focus on reliability. Torvalds also praised the ongoing evolution of Linux and the wider open-source ecosystem, noting its democratizing effect for new developers.

JFrog Extends GitHub Alliance to Provide Unified Dashboard

JFrog and GitHub have expanded their partnership to provide developers with a unified platform for better security and productivity. This integration offers a consolidated view of project statuses and security through tools like GitHub's Copilot chat and JFrog’s Advanced Security features. Developers can now get insights on third-party packages, track vulnerabilities earlier, and navigate between code and the binaries it produces seamlessly.

🛠️HackHub: Best Tools for Cloud

Apeman: AWS attack path management tool

Project Apeman is an AWS attack path management tool that helps analyze and manage AWS security data. To set it up, you need Docker, Python, and a virtual environment. Once the system is initialized, Apeman gathers AWS account data, including authorization details and ARNs, which are then ingested into a graph database for analysis.

Cyphernetes: A Kubernetes Query Language

Cyphernetes is a Cypher-inspired query language for Kubernetes, simplifying complex Kubernetes operations with intuitive, SQL-like queries. It allows developers to easily manage Kubernetes resources by expressing relationships between them, such as connecting deployments to services and ingresses.

Desed: A command-line tool for complex sed scripts

Desed is a command-line tool designed to help debug and understand complex `sed` scripts. It allows users to step through their scripts, both forwards and backwards, preview how substitute commands will affect the pattern space, and set breakpoints to examine the program's state. Desed also supports hot reloading, so changes to the source code can be instantly applied without restarting the debugger.

Kueue: Kubernetes-native Job Queueing

Kueue is a Kubernetes-native job queueing system that manages when jobs start and stop based on a variety of factors, such as priorities and resource availability. It offers features like job management with FIFO strategies, resource fair sharing, dynamic resource reclaim, and integration with popular job types like BatchJob and Kubeflow training jobs.

AWS CloudFormation Starterkit

An AWS CloudFormation starterkit including CI/CD and dev tools that allow you to securely and quickly deploy CloudFormation stacks on your AWS account.