Email Delivery Challenges

"Loops, software factories and why engineers need to own the outer loop" My latest free article: https://lnkd.in/guHi2wDF ✏️ One fundamental shift in agentic engineering isn't just about faster code generation - it's moving from writing code to building the factory that writes it. That shift starts with loop engineering. Instead of prompting an LLM turn-by-turn, loop engineering is about designing the small system that drives the agent for you: gather context, take an action, check the result, and repeat. When you scale these loops inside a harness and feed them a queue of work, you get a software factory. But how you run that factory determines whether you scale your output or just scale your technical debt. In my latest article, I explore the two ways to operate: 1. The "Lit" Factory: This is the same automated pipeline, but human judgment is moved upstream to architecture and system design. Agents execute the inner loops (diagnosing, implementing, and testing), while engineers own the outer loop - reviewing decisions and guarding the gate. The lights stay on where mistakes are expensive. 2. The "Dark" Factory: Fully autonomous loops churning out and shipping code with zero human oversight. Code ships that no one has read. It feels incredibly fast at first, but it quietly maximizes comprehension debt - the widening gap between how much code exists and how much your team actually understands. The trap is assuming that harness engineering alone is enough. Unconstrained loops inevitably wander off track in complex, brownfield systems. The real bottleneck in a software factory isn't generation but as so many have been saying it's verification. Autonomy can only expand as far as you can reliably and cheaply verify it. Read the full breakdown on how to structure your loops, manage back pressure, and know exactly which checks to build into your factory. #ai #programming #softwareengineering

One rule I follow in microservices systems: All message consumers must be idempotent. What happens when a message is re-sent in a distributed system? This could happen for several reasons. - We retry the publish because we didn't receive an ACK - The broker can retry delivering a message due to a network failure The worst-case scenario is that we process the message twice. Unfortunately, this also means we can apply side effects more than once. There are two ways how you can solve this: 1. Make your operation idempotent. Some operations are naturally idempotent, so duplicate messages aren't a problem. 2. However, some operations are not naturally idempotent. In this case, you have to implement idempotency manually. You can use the Idempotent Consumer pattern to solve this. Here's how the Idempotent Consumer works: - Was the message already processed? - If yes, it's a duplicate, and there's nothing to do - If not, we handle the message and store the message ID You also need to run this operation inside a transaction to ensure consistency. If concurrency is an issue, you could also consider locking to prevent race conditions. Here's how you could implement idempotent consumers: https://lnkd.in/eeTy86CJ Have you used the Idempotent Consumer pattern before? Another option to consider is having an Inbox on the consumer side.

A mistake many data engineers make is thinking real-time = streaming pipelines! Whenever a stakeholder says they want data in real time, you shouldn't default to dreams of Flink, Kafka, and watermarks. You should clarify with precision what an acceptable amount of latency is for this use case. Many times when a stakeholder asks for real-time data, it can be solved with an hourly batch pipeline. The incremental benefit to jump from hourly batch to streaming isn't worth it because it impacts the homogeneity of your suite of pipelines and makes the overhead maintenance much higher! Sometimes stakeholders say real-time and what they mean is "predictable refresh rates." This is a sign you need to do better as a data engineer at setting SLAs for your pipelines about when they'll refresh.

If you’ve built software in a distributed system, you know the rhythm: write code, run unit tests locally, put changes up for code review, eventually merge, wait for CI/CD tests to run, get a failure an hour, either rollback your changes or if you’re feeling lucky just re-run the tests, spend an hour debugging whether it’s your fault or a flaky test, fix it, push again, wait again. Repeat. The outer loop was never a feature. It was a compromise, a concession to the fact that important tests were too slow, too expensive, and too unreliable to run anywhere else. 𝐀𝐈 𝐡𝐚𝐬 𝐜𝐡𝐚𝐧𝐠𝐞𝐝 𝐭𝐡𝐞 𝐞𝐪𝐮𝐚𝐭𝐢𝐨𝐧. It is now generating code at a pace that makes the toil from this rhythm unsustainable. The part that’s bottlenecking isn’t the code generation, it’s validating the code. AI didn’t create this problem. It exposed it. It made it more acute and urgent to fix. The outer loop is dead. The bottleneck was never writing code. It was knowing whether the code works. It’s time to fix that.

Real-time data ≠ Streaming data! When stakeholders ask for "Real-Time" data, 99% of Data Engineers immediately think: Kafka, Queues, low latency, Streaming Pipelines. And most of the time… that’s wrong. The biggest mistake? Coding the pipeline before clarifying the requirement. “Real-time” often just means: Hourly, Daily, ... Even Weekly! Sometimes a simple batch job is all that’s needed. So ask yourself: Do you really need Streaming Data for that? Before writing a single line of code, clarify: • Business impact of delays • Data consumption frequency • Expected data volume & growth • Who actually uses the data (and how) • Required data freshness (seconds, minutes, hours?) I love building complex systems too, for learning. But simplicity wins. 99% of the times. Build pipelines that are: ✔️ Cheap to run ✔️ Fit for purpose ✔️ Easy to maintain --- ♻️ Repost if you’ve over-engineered a pipeline before! Follow 👉🏻 José for practical Data Engineering content

Are your stakeholders arguing about commas instead of what truly matters? It might not be their fault. Here’s a familiar nightmare for comms professionals: You spend hours crafting content - getting the tone right, aligning with the objective, polishing every detail. You send it out for approval… …and what comes back? A flood of comments about word choices, commas, and the colour of a chart. This often starts with the wrong ask. When you say, “Please review and let me know if you have any feedback,” you’re inviting every type of input - useful or not. Let’s fix that. Here’s how to get more focused, higher-value feedback: 1️⃣ Ask for specific input Narrow the scope so stakeholders know exactly what their role is. The queen of Internal comms Joanna Parsons recommends asking: 👉 “Can you fact-check this for accuracy?” This shift keeps the review focused on what actually matters. 2️⃣ Set a clear deadline Always include a concrete date and time. And add this line: 👉 “If I don’t receive feedback by the deadline, I’ll consider the information approved and move forward with publishing.” This removes ambiguity and speeds up approvals. 3️⃣ Raise the stakes if necessary Some stakeholders can’t resist commenting on every detail. 👉 If that’s your reality, send the content as a PDF. It’s reviewable - just less editable. If major changes are truly needed, they’ll ask for an editable file. This naturally filters out low-value edits. Just to be clear: this isn’t about avoiding input. It’s about protecting everyone’s time. When you guide stakeholders toward the feedback that truly matters, you keep them out of the comma-and-colour weeds and focused on impact. Just make sure before tightening the feedback process that you understand your stakeholders - sometimes maintaining strong relationships matters more than speed. Image credit: E.S. Glenn for The New Yorker

The recent inadvertent exposure of classified U.S. military plans by top defense and intelligence leaders serves as a stark reminder that even the most capable cybersecurity tools and well-defined policies can be rendered meaningless if ignored or misused. In this case, senior leaders relied on the Signal messaging app to communicate sensitive data but unintentionally exposed critical information to unauthorized parties. The leaked details—time-sensitive plans for a military operation—could have not only placed personnel in greater danger but also undermined the mission by alerting adversaries to an imminent attack. While #Signal is a widely respected, consumer-grade, end-to-end encrypted communication tool, it does not provide the same level of security as classified government systems. National security organizations typically utilize Sensitive Compartmented Information Facilities (SCIFs) to safeguard classified data from leaks and eavesdropping. However, SCIFs and other highly-secure methods are not as convenient as less secure alternatives—such as personal smartphones. In this instance, Signal's encryption was not the issue; rather, the exposure occurred when an unauthorized individual was mistakenly added to the chat. This human error resulted in sensitive information being disclosed to a reporter. Lessons Learned: This incident highlights critical cybersecurity challenges that extend beyond the military and apply to organizations everywhere: 1.     Human behavior can undermine even the most robust security technologies. 2.     Convenience often conflicts with secure communication practices. 3.     Untrained personnel—or those who disregard security protocols—pose a persistent risk. 4.     Even with clear policies and secure tools, some individuals will attempt to bypass compliance. 5.     When senior leaders ignore security policies, they set a dangerous precedent for the entire organization. Best Practices for Organizations: To mitigate these risks, organizations should adopt the following best practices: 1.     Educate leaders on security risks, policies, and consequences, empowering them to lead by example. 2.     Ensure policies align with the organization’s evolving risk tolerance. 3.     Reduce compliance friction by making secure behaviors as convenient as possible. 4.     Recognize that even the strongest tools can be compromised by user mistakes. 5.     Anticipate that adversaries will exploit behavioral, process, and technical vulnerabilities—never underestimate their persistence to exploit an opportunity. #Cybersecurity is only as strong as the people who enforce and follow it. Ignoring best practices or prioritizing convenience over security will inevitably lead to information exposures. Organizations must instill a culture of cybersecurity vigilance, starting at the top, to ensure sensitive information remains protected. #Datasecurity #SCIF #infosec

As more services move online, the public sector necessarily holds -- and uses -- far more data, much of it personal. Episodes like the 2018 SingHealth cyberattack are a reminder that public trust depends not just on convenience, but on strong governance and security. In practice, the boundary between “public” and “private” delivery is blurred. Many frontline and “last mile” services -- especially in social support -- depend on trusted external partners that have community relationships and specialised expertise. The challenge is that, today, when agencies need to share data with such partners, they often have to rely on consent or a common-law “public interest” basis, which can be slow and legally uncertain even for clearly public-spirited programmes. These amendments aim to create a clearer statutory framework for sharing data with trusted external partners, while importing familiar PSGA-style safeguards -- such as documented, scoped authorisation by the responsible Minister or delegate that specifies the purpose, the partner(s), and the data to be shared, and does not override other legal or contractual restrictions. On accountability, the intent is also clearer: external partners remain subject to the PDPA for personal data, and the amended framework would add offences and deterrents so that individuals handling shared government data in partner organisations face consequences for unauthorised disclosure or misuse, including for non-personal data that the PDPA would not cover. The key challenge will be implementation. Government agencies generally have more mature governance, training and cybersecurity processes than many smaller partner organisations. If more sensitive data is to be shared to improve service delivery, there should be commensurate investment in partner capability -- clear minimum standards, practical support, and proportionate compliance expectations -- so partners are not given new responsibilities without the capacity to carry them out safely. https://lnkd.in/gDwY4Btb

After years building event-driven systems. Here are the top 4 mistakes I have seen: 1. Duplication Events often get re-delivered due to retries or system failures. Without proper handling, duplicate events can: • Charge a customer twice for the same transaction. • Cause duplicate inventory updates, messing up stock levels. • Create inconsistent or broken system states. Solution: • Assign unique IDs to every event so consumers can track and ignore duplicates. • Design event processing to be idempotent, ensuring repeated actions don’t cause harm. 2. Not Guaranteeing Order Events can arrive out of order when distributed across partitions or queues. This can lead to: • Processing a refund before the payment. • Breaking logic that relies on correct sequence. Solution: • Use brokers that support ordering guarantees (e.g., Kafka). • Add sequence numbers or timestamps to events so consumers can detect and reorder them if needed. 3. The Dual Write Problem When writing to a database and publishing an event, one might succeed while the other fails. This can: • Lose events, leaving downstream systems uninformed. • Cause mismatched states between the database and event consumers. Solution: • Use the Transactional Outbox Pattern: Store events in the database as part of the same transaction, then publish them separately. • Adopt Change Data Capture (CDC) tools to track and publish database changes as events automatically. 4. Non-Backward-Compatible Changes Changing event schemas without considering existing consumers can break systems. For example: • Removing a field might cause missing data for consumers. • Renaming or changing field types can trigger runtime errors. Solution: • Maintain versioned schemas to allow smooth migration for consumers. • Use formats like Avro or Protobuf that support schema evolution. • Add adapters to translate new schema versions into older ones for compatibility. "Every schema change is a test of your system’s resilience—don’t fail it." What other mistakes have you seen out there?