DEV Community: Anguishe

The Alert Never Fired Because the Loop Skipped the Last Line of the File

Anguishe — Fri, 19 Jun 2026 17:58:09 +0000

We kept a plaintext file of hostnames, one per line, and a monitoring script read the file and pinged each host every five minutes. When a host failed to respond, the script sent an email alert. The system had been running for months and it worked — we had caught three actual outages with it, which gave us real confidence in the setup.

app-07 was added to the list on a Thursday afternoon. The engineer who added it was using VS Code on a Mac, and VS Code by default does not add a trailing newline to a file when you append to it using certain editing workflows. The file had ended in a newline before the edit. After the edit, the last line — app-07 — had no trailing newline.

app-07 went down the following Sunday afternoon at 2:17pm. The monitoring script ran at 2:20, 2:25, 2:30, all the way through the evening. No alert ever fired. The on-call engineer found out at 8pm when a client emailed. The system had been down for almost six hours.

When I looked at the script, the bug was immediately obvious once I knew what to look for. But I had written that script, I had tested it, and I had been looking at the monitoring confirmation emails for months without ever noticing. The confirmation email listed the hosts it checked. app-07 was never in the list. I had been reading those emails without actually counting the hosts. I just scanned for the OK lines and moved on.

Why read drops the last line

read returns a success exit status when it reads a line and finds the newline that terminates it. When the file does not end in a newline, read still populates the variable with the final line's content, but it returns a non-zero (failure) exit status because it hit end-of-file before finding a terminator. A while read host loop checks the return status to decide whether to execute the loop body. On the final, newline-less line, read puts app-07 into host and then returns failure. The while loop sees failure and exits without running the body. The content is there. The variable is populated. The loop throws it away.

This behavior is documented in the POSIX spec for read. It is not a bash quirk. Any POSIX shell handles the missing-final-newline case this way. A plain while read line loop is incorrect for any file you do not personally control the formatting of, which in practice means nearly any file.

The fix is one extra clause:

#!/bin/bash
# Script: check-hosts.sh
# Purpose: ping every host in a file, including a newline-less final line
# Usage: ./check-hosts.sh hosts.txt
set -euo pipefail

CHECK="✓"
CROSS="✗"
HOST_FILE="${1:?Usage: check-hosts.sh <host-file>}"

while IFS= read -r host || [[ -n "$host" ]]; do
  [[ -z "$host" || "$host" == \#* ]] && continue
  if ping -c1 -W2 "$host" >/dev/null 2>&1; then
    echo "$CHECK up:   $host"
  else
    echo "$CROSS down: $host"
  fi
done < "$HOST_FILE"

|| [[ -n "$host" ]] says: if read returned failure but the variable is non-empty, run the loop body anyway. That is precisely the leftover-final-line case. read failed because it hit end-of-file, but it populated host with app-07 before returning. The || catches it. app-07 gets pinged.

What IFS= and -r actually do

You see while IFS= read -r line written in every correct read-loop example, and it is worth being specific about what each piece prevents because both have their own failure mode.

IFS= sets the field separator to empty for the duration of the read command. Without it, read strips leading and trailing whitespace from each line. A hostname like app-07 (with leading spaces, which some editors produce) becomes app-07, which might be correct. An indented config value, a Python-style YAML string, a log line that starts with spaces for alignment — all of these are silently modified. Setting IFS= tells read to take the line exactly as it appears.

-r prevents read from interpreting backslash sequences. Without -r, a line like C:\temp\logs has its backslashes consumed as escape characters and arrives as C:templogs. This matters less for hostname files and enormously for any script that processes Windows paths, config files that use backslash as a line-continuation character, or log files from mixed-OS environments. The -r flag is essentially free protection; there is no reason not to include it.

A bare read line without either flag silently mangles both whitespace and backslashes. The script works correctly on clean input and produces wrong output on input with edge cases. The wrong output does not produce an error. You find out when the data that mattered was the indented or backslash-containing kind.

The subshell trap that kills your counters

This is the one that is most likely to make you question your sanity:

# This looks correct. It is not.
fails=0
cat hosts.txt | while IFS= read -r host; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done
echo "Total failures: $fails"   # Always prints 0

The pipe creates a subshell for the right side. The while loop runs inside that subshell. fails increments correctly inside the subshell. When the subshell exits, the parent shell's fails is still 0, because the increment happened in a different process. The parent echo sees the original value.

This catches people because the loop body itself works — the pings happen, the increment logic is correct — but any state the loop was supposed to accumulate for later use is silently discarded. I spent forty minutes on a version of this problem before I remembered that pipes create subshells. It is the kind of thing that feels like a bash bug until you understand that it is behaving exactly as documented.

The fix is to redirect the file into the loop instead of piping into it:

fails=0
while IFS= read -r host || [[ -n "$host" ]]; do
  ping -c1 -W2 "$host" >/dev/null 2>&1 || ((fails++))
done < hosts.txt
echo "Total failures: $fails"   # Now correct

done < hosts.txt feeds the file to the loop's stdin without a pipe. The loop runs in the current shell. fails accumulates in the current shell. The echo sees the real count.

Why for loop is wrong for this

# Never do this — iterates words, not lines
for line in $(cat hosts.txt); do
  ping -c1 "$line"
done

$(cat hosts.txt) is command substitution. Bash captures the text output and word-splits it on IFS — spaces, tabs, newlines. For a file with one hostname per line and no spaces in the hostnames, this accidentally produces the right behavior. For any file with spaces — log lines, config values, paths with spaces, anything a non-developer might have generated — it splits lines into fragments and each fragment becomes a loop iteration.

There is no version of for line in $(cat file) that is correct for reading lines. The right tool is always while IFS= read -r line || [[ -n "$line" ]]; do ... done < file. The for loop is the right tool for iterating a known list that you control directly, not for reading file content.

The monitoring system, after the fix

After the app-07 incident we made three changes. The obvious one was fixing the read loop with the || [[ -n "$host" ]] guard. The second was adding a sanity check at the top of the script that counted the lines in the hosts file and compared it to the number of hosts the loop actually processed — a mismatch meant the file was malformed or something else was wrong. The third was adding a nightly email that included the count of hosts checked, not just the status of each one, so a future addition to the file that somehow got lost would show up as "expected 12 hosts, checked 11."

The confirmation email count was something I should have had from the start. If I had been looking at "checked 11/12 hosts" instead of a list of OK lines, I would have noticed app-07 missing on the first night. The monitoring was working. The observability of the monitoring was not.

Full version with CSV-field parsing, comment-skipping, and the subshell-safe redirect form: https://bashsnippets.xyz/snippets/bash-read-file-line-by-line

To iterate a list of files rather than a file's contents, reach for a for loop instead, and wrap anything that acts on what it reads in set -euo pipefail. More at https://bashsnippets.xyz

A For Loop Skipped Every File With a Space and Called the Backup a Success

Anguishe — Thu, 18 Jun 2026 19:29:49 +0000

The nightly backup looped over for f in $(ls /data/exports) and copied each file to a backup volume. It exited clean every night. For three weeks, green exit codes, no errors, nothing in the logs to suggest anything was wrong. The backup script had been written by someone who left the company six months before I started, and it had never been tested against files with spaces in their names because the original export directory only ever had files like Q3.xlsx and report-final.xlsx.

Then someone generated Q3 final.xlsx.

It took three weeks because that file was generated once a quarter and the next time someone needed it was a quarter later. The person who needed it was the CFO. The CFO does not particularly enjoy being told that the backup system that was supposed to protect the quarterly export has been silently failing for an indeterminate period of time and we are not sure which other files it missed.

I know the specific backup had been running for three weeks because that was the date the file appeared in the exports directory. Every night since then, the loop had been splitting Q3 final.xlsx into two items — Q3 and final.xlsx — trying to copy two paths that did not exist, logging two harmless "no such file or directory" lines that nobody read, and moving on. Every file without a space in its name backed up fine. The script looked correct because most of the time it was.

Why $(ls) word-splits

When you write for f in $(ls /data/exports), bash runs ls, captures its text output, and splits it on IFS — the internal field separator, which defaults to spaces, tabs, and newlines. Filenames are just text in the output of ls. A file named Q3 final.xlsx is one filename, but ls outputs it as the string Q3 final.xlsx, and bash splits that string on the space into two separate items before the loop ever starts.

This is not a bug in ls. This is not a bug in bash. This is exactly what $( ) does to any command's output — it captures text and bash processes it as text. The problem is that filenames are not reliably text-safe; they can contain any character except null and the path separator. Spaces are common. Tabs are less common but legal. Newlines are technically legal. Treating command output as a filename list breaks the moment any of those show up.

The fix is to stop treating command output as a filename list and let bash build the list from the filesystem directly:

#!/bin/bash
# Script: backup-exports.sh
# Purpose: copy export files without losing ones with spaces in their names
# Usage: ./backup-exports.sh
set -euo pipefail

CHECK="✓"
CROSS="✗"
SRC_DIR="/data/exports"
DEST_DIR="/backup/exports"
shopt -s nullglob   # empty glob expands to nothing, not the literal pattern

for f in "$SRC_DIR"/*.xlsx; do
  if cp -- "$f" "$DEST_DIR/"; then
    echo "$CHECK backed up: $(basename "$f")"
  else
    echo "$CROSS failed: $(basename "$f")"
  fi
done

for f in "$SRC_DIR"/*.xlsx asks bash to expand the glob. Bash talks directly to the filesystem and gets back a properly-separated list of matching paths. No command output, no text splitting, no ambiguity about what the separator is. A file named Q3 final.xlsx stays one item because it was never turned into text and split back apart. The -- before "$f" tells cp to stop reading flags, so a filename that starts with a dash does not get interpreted as a cp option.

The second half: quoting on use

The glob fixes the loop header. Quoting fixes the point of use. These are separate.

for f in "$SRC_DIR"/*.xlsx; do
  cp $f "$DEST_DIR/"    # WRONG — $f re-splits here
  cp "$f" "$DEST_DIR/"  # RIGHT — the quotes prevent the split
done

Even with a correct glob, an unquoted $f in the cp command re-splits on spaces. Bash has already expanded the variable to Q3 final.xlsx, but when you use it unquoted, the shell processes word splitting again on that value and cp receives two arguments: Q3 and final.xlsx. The quotes around "$f" tell bash to pass the entire value as a single argument. The rule is: glob over parse to build the list, quote on use to keep each item intact.

I have seen this mistake repeated in scripts at three different companies. In each case the scripts had been running for months or years and the problem was invisible because the majority of files had no spaces. The ones that did — client names, quarterly reports, anything a non-technical person named — were being silently skipped or mishandled.

Ranges and counters: where the brace trap hides

n=5

# This does NOT produce a range — it produces the literal string {1..5}
for i in {1..$n}; do echo "attempt $i"; done

# This works — C-style, evaluates variables at runtime
for ((i = 1; i <= n; i++)); do echo "attempt $i of $n"; done

Brace expansion happens before variable expansion in bash's order of operations. By the time $n is replaced with its value, the brace expansion step has already passed and {1..$n} is just a string. This is the kind of thing that fails silently in a test environment where the count is small and hardcoded, and causes weird output in production where it is a variable.

The C-style loop is the correct form any time the bound is a variable. It evaluates at runtime and handles arithmetic naturally. If the count is genuinely fixed at write time, {1..10} works fine. If it is ever going to be a variable, use for (( )).

Arrays: the original bug wearing a different hat

servers=("web-01" "db primary" "cache-02")

# WRONG — word-splits "db primary" into two iterations
for s in ${servers[@]}; do
  ping -c1 "$s"
done

# RIGHT — quotes keep each element intact
for s in "${servers[@]}"; do
  ping -c1 "$s"
done

"${servers[@]}" with the quotes and [@] is the form that preserves each element as a single item regardless of what is in it. Without the quotes, bash word-splits the array expansion and db primary becomes two separate loop iterations — neither of which is a real hostname. This is exactly the same word-splitting mechanism as the $(ls) problem, just manifesting in arrays instead of command output.

Once you internalize that word-splitting happens wherever an unquoted variable or expansion appears, the rule becomes one rule instead of several: always quote expansions. The specific context changes; the mechanism does not.

The nullglob case

shopt -s nullglob
for f in /data/exports/*.xlsx; do
  echo "$f"
done

Without shopt -s nullglob, if no .xlsx files exist, the glob *.xlsx does not expand — it stays as the literal string *.xlsx. The loop runs once with f set to the literal string /data/exports/*.xlsx. Your script then tries to process a file with that exact path, which does not exist, and produces an error or silently does nothing depending on what you do with it.

Setting nullglob tells bash to expand a non-matching glob to nothing (an empty list), so the loop simply does not run. This is almost always the right behavior when you are iterating files that might not exist.

What happened to the Q3 export

We recovered it from the CFO's local machine, where she had downloaded it before the backup was supposed to preserve it. The fix to the script took four minutes. The conversation about why the backup system had been failing silently for three weeks took longer. The monitoring that we added afterwards — a nightly check that the backup directory has at least as many files as the source directory — took another twenty minutes.

The monitoring should have been there from the start. So should the glob. So should the set -euo pipefail that would have made the copy failures loud instead of silent. These are things you add before something breaks, and the only reason to know you need them is to have seen, or caused, or read about what happens when they are missing.

Full examples with the safe glob, counter, C-style loop, array form, and nullglob guard: https://bashsnippets.xyz/snippets/bash-for-loop-examples

For reading a file's lines one at a time, a for loop is the wrong tool — use while IFS= read -r — and wrap any loop that touches real files in set -euo pipefail. The rest is at https://bashsnippets.xyz

find . -delete Ran Before the Filter and Emptied the Whole Tree

Anguishe — Wed, 17 Jun 2026 23:59:08 +0000

I meant to delete the .cache files under a data directory. The server had been running for two months and the cache layer had grown to about 14GB. The application team told me it was safe to purge it — they'd rebuilt the cache logic and the old files were just dead weight. I typed find /data -delete -name "*.cache" because I was moving fast and I figured the order of arguments to find did not matter. It does. find evaluates its expression left to right, and -delete is not a filter — it is an action. It fired on every path find walked, starting at /data itself, and -name "*.cache" never got a chance to narrow anything. By the time I hit Ctrl-C the tree was roughly 80% gone.

That was not a test server.

The restore from backup took forty minutes. The application was down for those forty minutes. The postmortem was a forty-five minute conversation with people who did not particularly enjoy having it. I have run hundreds of find commands since then and I verify the expression order before every single one that has any destructive action attached to it — not because I've forgotten the rule, but because the cost of forgetting it once is not recoverable with an apology.

Why the order is the program

find does not have a flag parser that groups tests and actions separately. It walks a directory tree and evaluates its arguments as a logical expression, left to right, short-circuiting on false. When you write:

find /data -name "*.cache" -delete

it evaluates -name "*.cache" first on each path. If the name does not match, it short-circuits and -delete never runs on that path. When you write:

find /data -delete -name "*.cache"

it evaluates -delete first. -delete always succeeds — it removes the path and returns true, which means the expression continues to -name. The name check runs after the deletion, on a file that no longer exists, which is meaningless. The effect is that everything gets deleted and nothing is filtered.

This is not a bug. It is exactly how the man page says find works. It is just not how anyone instinctively reads a command the first few times they use it.

The rule is: tests filter, actions act, and actions must come after the tests that are supposed to narrow them. Write it in that order every time.

The quoting trap, right behind the ordering one

Here is the one that is even more subtle, because it causes the wrong behavior and produces no error at all:

# WRONG — the shell expands *.cache before find ever sees it
find /data -name *.cache

# RIGHT — quote the pattern so find does the matching
find /data -name "*.cache"

If there happens to be a single .cache file in your current working directory when you run this, the shell expands *.cache to that one filename and passes it to find -name as a literal string. find then searches for files named exactly that, everywhere under /data. It finds some, it finds none, but it is definitely not doing a wildcard search. If there are multiple .cache files in your current directory, find receives too many arguments for -name and errors out with something confusing about paths needing to precede the expression.

Either way you did not get what you intended, and if you added -delete, you deleted the wrong things silently.

Quoting the pattern is the fix. The quotes prevent the shell from expanding the glob so find receives the literal *.cache pattern and handles the wildcard itself, across the directory tree you pointed it at.

The age sign that everyone reverses at least once

find /var/log -name "*.log" -mtime +30   # older than 30 days
find /var/log -name "*.log" -mtime -1    # modified within the last day
find /var/log -name "*.log" -mtime 30    # exactly day 30 (almost never what you want)

+30 is older than thirty days. -1 is within the last day. A bare 30 means precisely thirty days ago, which is almost never the thing you're trying to match. The sign convention is the opposite of what feels natural — you want files "older than" a threshold and the intuitive symbol for "bigger number" is +, but a lot of people read +30 as "in the last thirty days" the first time they see it.

I have reversed this twice in production. Once I kept the wrong logs. Once I deleted logs I needed for an audit the following week. Neither was catastrophic but both were embarrassing, and both happened under the kind of mild time pressure where double-checking the man page feels slower than it actually is. The builder I built for this labels the output as "older than" or "within the last" in plain English next to the value, which removes the one decision in a find -delete job most likely to go wrong when you are already stressed.

The -exec batching difference nobody explains

# Runs the command once per file — slower, one PID per file
find /data -name "*.cache" -exec rm {} \;

# Batches files into one invocation — faster, one rm for many files
find /data -name "*.cache" -exec rm {} +

\; runs the command once per file. + collects as many paths as it can and passes them all to one invocation of the command, the same way xargs does. For something like rm, which accepts multiple arguments, the + form is faster and produces less process overhead. For something like a custom script that must process exactly one file at a time, \; is correct.

Most resources either do not mention this difference or mention it once in a reference table. The practical consequence is real — on a directory with ten thousand files, \; spawns ten thousand processes. The + form spawns a handful. On a cleanup job that runs in cron, the difference shows up in CPU load.

The preview step I skipped the day I broke things

The most reliable way to avoid the ordering and quoting mistakes is to build the command in two steps. First, run it with -print instead of -delete:

find /data -name "*.cache" -print

Read every line of that output. Confirm the list is what you intended. Then, and only then, swap -print for -delete. This adds maybe thirty seconds to the workflow. It would have saved me forty minutes and a postmortem. I skipped it because I was confident. Confidence is not a useful substitute for verification when the operation is irreversible.

The find command builder enforces this by showing a warning whenever you select -delete or -exec as the action, and offering to generate the -print version of the command first. It is the kind of nudge I would have appreciated having that day.

What the builder actually does

It assembles the expression in the legally correct order — tests first, action last — so you cannot accidentally replicate the mistake I made. You set the starting path, add tests in whatever order feels natural to you (name, type, age, size, exclude path), and pick an action. The output command always has the tests before the action, regardless of the order you clicked things in.

Every active flag gets a plain-English description inline. -mtime +30 reads as "modified more than 30 days ago." -name "*.cache" reads as "name matches the glob *.cache, quoted so find handles the wildcard." The -exec {} + form is the default when you pick -exec, with a note explaining why it is faster.

The whole point is to get from "I need to find and delete files matching these conditions" to a verified, copy-paste command without the thirty-second loop of man-page reading and second-guessing that I used to do and, on one bad morning, skipped.

What I upgraded while I was at it

The same discipline applies to the other tools. The rsync command builder now has presets for the three setups people build most often — local backup, push to remote, mirror — because those cover maybe ninety percent of rsync jobs and getting the flags right from scratch every time is where the dangerous ones like --delete get misapplied. The cron builder previews the next five run times after you build an expression, because a cron job I once deployed ran at 3am UTC instead of 3am local time and I did not notice until it fired on the wrong schedule for a week. You can also paste an existing crontab line and get the human-readable schedule back. The chmod builder now accepts an octal you paste in and sets the checkboxes — two-directional, because reading a file's permissions and understanding what they mean is just as common a task as setting them from scratch.

These are not features I planned in advance. They are all things I needed at 2am and did not have. That is the pattern most of this site runs on.

Build a find command with tests ordered before actions and every flag explained: https://bashsnippets.xyz/tools/find-command-builder

If you are scoping files before searching or transforming them, the whole pipeline is documented in Bash Text Processing: find, grep, sed, and awk. The rest of the free tools are at https://bashsnippets.xyz/tools

A for Loop Skipped 23 Files and Called It a Successful Backup

Anguishe — Sun, 14 Jun 2026 17:26:03 +0000

The backup ran every night at 2am and emailed me a green "847 files archived" summary. I'd built it, tested it against my own home directory where every file was named like report_2024.csv, watched it sail through, and shipped it. For weeks the summary said everything was fine.

Then a coworker asked me to restore a file. Q3 forecast.xlsx. It wasn't in the archive. Neither was Annual Review FINAL.pdf, or meeting notes (draft).md, or any of the other 23 files someone had named the way normal humans name files — with spaces in them. The backup had been quietly skipping the most important files on the share for a week, and the nightly email had been telling me the whole time that nothing was wrong.

Here's the line that did it:

for file in $(ls "$DIR"); do
  echo "Processing $file"
done

The problem is word splitting, and it's invisible until the day it isn't. An unquoted $(ls ...) splits its output on every space, so Q3 forecast.xlsx doesn't arrive as one filename — it arrives as two iterations, Q3 and forecast.xlsx. Neither one exists on disk. The loop tries both, finds nothing, shrugs, and moves on without a single error. The script "succeeds" because from its point of view it did exactly what it was told.

I've now got two habits burned in, and I don't write a loop without both. The first: glob the directory, never parse ls.

for file in "$DIR"/*; do
  [[ -e "$file" ]] || continue   # the glob yields a literal '*' on an empty dir
  echo "Processing $file"
done

"$DIR"/* hands you each real path as a single unit. No subshell, no string to re-split, no ls output to misread. The [[ -e "$file" ]] || continue guard covers the one quirk of globbing: when a directory is empty, * expands to the literal character *, and without the guard you'd try to process a file named *.

The second habit: quote every expansion, every time. "$file", never $file. The day a filename has a space in it, the unquoted version splits into two arguments and your command operates on paths that were never there.

Arrays follow the exact same rule, and the distinction that matters is [@] versus [*]:

servers=("web-01" "db-prod 02" "cache-03")

for host in "${servers[@]}"; do
  echo "Connecting to $host"   # three clean iterations, the space survives
done

"${servers[@]}" in double quotes gives you one word per element — db-prod 02 stays whole. "${servers[*]}" joins everything into a single string and is almost never what you want in a loop. Drop the quotes on either and you're back to the bug that ate my backup.

When you genuinely need a counter — walking app.log.1 through app.log.9, numbering batches, counting retries — use the C-style form. Do not reach for for i in {1..$n}; brace expansion runs before variable expansion, so $n never gets substituted and you loop once over the literal text {1..$n}. Ask me how I know.

for ((i = 1; i <= MAX_ROTATED; i++)); do
  log="$LOG_DIR/app.log.$i"
  [[ -f "$log" ]] || continue
  echo "Scanning $log"
done

And reading a file line by line — for line in $(cat file) is wrong in two directions at once. It splits on whitespace instead of newlines, so a line with spaces becomes several iterations and blank lines vanish, and it globs, so a line containing * expands to filenames in your current directory. The form that survives real files:

while IFS= read -r line; do
  echo "Line: $line"
done < "$INPUT_FILE"

IFS= stops bash from trimming leading and trailing whitespace. -r stops it from eating backslashes. One line per iteration, exactly as written.

The production script I keep on the site ties all of this together — globs the directory, quotes every use, counts successes and failures separately, and exits non-zero if anything failed. That last part is the one people skip, and it's the one that matters: a loop that processes files but always exits 0 is how you end up with a nightly email that says 847 when the real number is 824.

The cost of my mistake was a week of bad backups and the specific discomfort of a coworker finding the gap before my own tooling did. I'm not interested in repeating that, so the quoting is muscle memory now. Yours might as well be too.

Full script with all four loop forms — files, arrays, counters, and line-by-line reads — production-ready and ShellCheck-clean: https://bashsnippets.xyz/snippets/bash-for-loop-examples

If your loops are the thing crashing scripts, the Bash Error Handling snippet pairs with this one, and the rest of the library is at https://bashsnippets.xyz

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

Anguishe — Fri, 12 Jun 2026 16:27:11 +0000

For three weeks a deployment pipeline reported every step green and shipped a build that had failed to compile on every single run. The build step ended in npm run build | tee build.log so the output could be archived. That pipe is the whole story: bash returns the exit status of the last command in a pipeline, which was tee, and tee always succeeds at copying text. The compiler's non-zero exit got thrown away the instant the pipe handed off. The error was sitting right there in build.log. GitHub Actions saw exit code 0, painted the step green, and deployed the broken artifact. Nobody read the log, because the checkmark said there was nothing to read.

That's the defining property of bash in CI, and it's why I treat pipeline scripts differently from anything I run in a terminal: a silent failure can present as success. On a server you watch a command fail in front of you. In a pipeline, a swallowed exit code produces a green checkmark over broken code, and the gap between "the logs show an error" and "the pipeline reports failure" is exactly where outages are born. I wrote the full guide because I've now been burned by every variation of this, and there's a consistent set of habits that close the gap.

There are four failure modes that are specific to CI and barely ever bite you at an interactive prompt. Exit codes swallowed by a pipe — the story above, any command | tee, command | grep, command | sort. Shell provisioning differences — ubuntu-latest gives you bash 5.x, macos-latest gives you bash 3.2 from 2007, and a script using associative arrays or ${var,,} passes on one runner and throws a syntax error on the other in the same workflow. Environment variable gaps — CI sets variables you don't control and omits ones you assume exist, and without set -u a missing $DEPLOY_TARGET becomes an empty string and does something quietly wrong. Interactive-shell assumptions — CI runs a non-interactive, non-login shell that never sources your .bashrc, so a command that works when you type it dies with command not found because the thing that defined it was never loaded.

The header that closes most of these is short, and every line earns its place:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

set -e exits the moment any command fails, so the step exits non-zero and the workflow actually registers a failure. set -u treats an unset variable as an error, so a typo'd $DPLOY_TARGET dies immediately instead of expanding to nothing and corrupting a path. set -o pipefail makes a pipeline return the first non-zero exit among its commands rather than only the last — that one flag is the direct fix for the | tee bug that ran green for three weeks.

Secrets deserve their own paragraph because CI hands you env: values and secrets: values identically — the shell can't tell them apart, the only difference is that Actions masks the secret's literal string in the log. The trap is that the moment you transform a secret (base64-decode it, slice it, interpolate it), the transformed value no longer matches the mask and prints in clear text. Validate required values up front with ${VAR:?} so a missing secret fails at startup with a clear message instead of on line 47 with a cryptic permission denied, and be very careful with set -x in any step that touches a secret.

The pipe-exit-code problem is worth one concrete tool beyond pipefail: PIPESTATUS is an array holding the exit code of every command in the last pipeline, read immediately after:

npm run build | tee build.log
build_rc="${PIPESTATUS[0]}"   # npm's code, not tee's
[[ "$build_rc" -eq 0 ]] || { echo "build failed: $build_rc" >&2; exit "$build_rc"; }

pipefail has one well-known false positive — grep returns 1 when it finds no matches, which is often fine, and under pipefail plus set -e that aborts the script. Absorb it deliberately with || true only where a non-match is genuinely acceptable, and nowhere else, because blanketing every command in || true just reinvents the silent-success problem you're trying to kill.

Docker has its own landmine: every entrypoint script must end with exec "$@". Without it, your script stays PID 1 and your app runs as a child, so when the orchestrator sends SIGTERM on docker stop or a rolling deploy, the signal hits the script, which doesn't forward it, and after the grace period the orchestrator escalates to SIGKILL — abrupt termination, dropped connections, lost in-flight work. exec "$@" replaces the shell with your app so it becomes PID 1 and receives signals directly. The guide pairs this with a wait_for dependency-check pattern and a trap, and the Bash trap & Signal Handler Builder generates the exact signal block an entrypoint needs.

Deploys get the same treatment: deploy into a fresh timestamped directory, flip a current symlink atomically with ln -sfn so traffic never sees a half-written release, keep the last several releases so rollback is just re-pointing the symlink, run a health check after the swap and fail the deploy if it doesn't pass, and stamp the git SHA into the release so "what's running right now" always has an answer. And when a step fails and the logs won't say why, set -x around just the suspect section shows you each command with its variables expanded — a doubled slash or an empty segment in the trace is usually your bug standing in plain sight.

The full guide is the field manual version of all of this — the four failure modes, the safe header, secret validation with a validate_env function, PIPESTATUS and pipefail, Docker entrypoints, the atomic-symlink deploy script with rollback and health check, debugging with set -x, and a production-ready checklist at the end: https://bashsnippets.xyz/guides/bash-scripting-for-ci-cd-pipelines

If your pipeline scripts are dying on the small stuff first — unquoted loops, bad argument parsing, missing traps — the snippet library that feeds into this guide is at https://bashsnippets.xyz

My Script Crashed and Left a Lock File Behind. Every Run After That Refused to Start.

Anguishe — Thu, 11 Jun 2026 16:56:45 +0000

A backup script of mine created a lock file on startup so two copies couldn't run at once — sensible. Then one night it hit an error partway through, set -e killed it on the spot, and it died without ever reaching the line that removes the lock. The lock file sat there. Every scheduled run for the next three days started up, saw the lock, printed "already running," and exited immediately. No backups ran. The cron job was firing perfectly on time and doing nothing, and the only symptom was an absence — backups that simply weren't there — until I went looking and found a stale lock from Tuesday.

The fix is a trap. A trap registers a cleanup handler that runs when the script exits for any reason — clean finish, set -e failure, Ctrl+C, kill. Put the lock removal in an EXIT trap and it runs no matter how the script dies. The lock would have been gone the instant that backup crashed, and the next run would have started fine.

So why did I write the script without one? Because I could never remember the syntax cold. Single quotes or double? Which signals? Does EXIT fire on exit 1 or only on a clean finish? How do I get the exit code inside the handler? Every time I needed a trap I ended up with three browser tabs open, reading the same Stack Overflow answers, second-guessing the quoting. Under pressure, in the middle of fixing something else, that friction is exactly when people skip the trap entirely — which is how I ended up with the stale lock in the first place.

So I built the thing I kept wishing existed: a Bash trap & Signal Handler Builder. You pick the signals you want to handle, check off the cleanup actions you need, and it writes a correct, ShellCheck-clean trap block you paste into your script. No tabs, no second-guessing the quoting.

It covers the signals that actually come up. EXIT — fires on every exit, the one that should carry your cleanup. ERR — fires when a command fails under set -e, the one that logs the exact failing line with $LINENO. INT — Ctrl+C. TERM — what kill, systemctl stop, and docker stop send. HUP — terminal closed or SSH dropped. PIPE — writing to a closed pipe. Each one has a one-line reminder of when it fires and what it's good for, because half the battle is just remembering that TERM is the one Docker sends.

The cleanup actions are the things people forget until a crash makes them care: remove temp files (with the TMPFILE=$(mktemp) declaration wired in up top), remove a lock file — the exact failure that bit me — stop background jobs the script started, log the exit reason with the code, and restore the terminal cursor. Tick the ones you need and they land in the handler.

The generated code isn't a toy snippet. It single-quotes the trap so the handler resolves when the signal fires instead of at definition time — the SC2064 gotcha most hand-written traps get wrong. It includes an idempotency guard so that when ERR and EXIT both fire on the same failure, your cleanup runs exactly once instead of twice. The ERR handler captures $LINENO so you find out which line actually blew up. I ran the generator's output through ShellCheck across a dozen different configurations while building it, and every one comes back clean — the whole point was that you can paste it and trust it, not paste it and then go debug the thing that was supposed to save you debugging.

There are two copy buttons, because there are two situations. "Copy trap block only" when you've already got a script and just want to drop the traps in. "Copy complete script header" when you're starting fresh and want the shebang, set -euo pipefail, the CHECK/CROSS vars, the resource declarations, and the handler all assembled in the right order. It only declares the variables the generated code actually uses, so you never paste in a CROSS you never reference and get a ShellCheck warning for your trouble.

The lock-file incident cost me three days of silently missing backups and ten minutes of feeling foolish when I found the cause. The trap that would have prevented it is four lines. The reason I didn't have those four lines was pure friction — I couldn't recall the syntax fast enough to be bothered in the moment. This tool removes the friction, which is the only thing that was ever standing between me and a correct script.

Build your trap block here — pick signals, pick cleanup actions, copy clean code: https://bashsnippets.xyz/tools/bash-trap-builder

If you want the full picture of where traps fit — strict mode, cleanup, the failure modes that make them necessary — the Bash Error Handling snippet is the companion read, and the rest of the tools are at https://bashsnippets.xyz/tools

I Packaged the Scripts I Copy to Every New Server Into a $9 Toolkit. Here's What's In It and Why.

Anguishe — Tue, 09 Jun 2026 15:44:19 +0000

Every time I provision a new server — whether it's a $5 DigitalOcean droplet for a side project or a client's production box — there's a set of scripts I copy to /opt/scripts before I do anything else.

Not after the app is deployed. Not after the first incident. Before I touch the application at all. Before I configure nginx. Before I set up the database. The monitoring layer goes in first because the first time you need it, you needed it yesterday.

Disk monitoring that fires before the outage. A backup pipeline with automatic retention. SSL certificate checks that run daily at 8am so I'm not finding out from a user's email. A service watchdog that restarts nginx or Postgres within 60 seconds of a crash instead of six hours later when someone notices the site is down.

These scripts took me about two years of production incidents to build. Not because they're complicated — they're not. Each one is 15-50 lines. Because each one was built in response to a specific failure where I didn't have the thing I needed and had to build it under pressure at a bad time of day.

I open-sourced the basic versions on BashSnippets.xyz. Those are free and they'll stay free. But the versions I actually run in production are different from the tutorial versions in ways that matter — and that gap is what I packaged into the toolkit.

What's Different Between the Free Snippets and the Toolkit

The free snippets on the site are single-purpose scripts that each solve one problem. They're complete. They work. If all you need is a disk space check, the free version does that.

The toolkit versions are built as a system.

There's a shared library — bashlib.sh — with 31 functions that every script sources. Logging, color output, email alerts, error handling, lock file management, threshold checks, dry-run support. Instead of each script reimplementing its own log() function and its own error handling and its own email logic, they all call bashlib.sh and get consistent behavior across the board.

That means when I change how logging works, it changes everywhere. When I add Slack webhook support to the alert function, every script that calls alert() gets Slack notifications without any changes to the script itself. The shared library is the infrastructure layer that turns six standalone scripts into a cohesive system.

The free snippets don't have this because a shared library adds a dependency — bashlib.sh has to exist at a known path, the scripts have to source it at startup, and if someone downloads one script without the library it breaks. That's fine for a toolkit you install as a package. It's bad for a tutorial page where someone wants to copy-paste one script and have it work immediately. Both approaches are correct for their context.

What's in the Box

6 production scripts:

Each one follows the same structure — set -euo pipefail, sourcing bashlib.sh, named variables for every threshold and path (no magic numbers buried in command pipelines), comments explaining not just what each line does but why it exists, and explicit non-zero exits on every failure path.

The scripts cover disk space monitoring, database backup with retention, SSL certificate expiry checking across multiple domains, service watchdog with automatic restart, log rotation and cleanup, and system health reporting.

bashlib.sh — the shared library (31 functions):

This is the part I'm most particular about. Functions for timestamped logging with severity levels. Color-coded terminal output that degrades gracefully when piped to a file (no escape codes in your log files). Email and webhook alerting. Lock file acquisition with stale-lock detection. Dry-run mode that every function respects. PID file management. Threshold comparison helpers. Configuration file loading.

Every function is documented with a usage comment. Every function handles its own error cases. The library passes ShellCheck with zero warnings at every severity level.

template.sh:

A starter template that sources bashlib.sh, sets up traps, parses arguments, and has placeholder sections for your own logic. When I need a new script on a server, I copy this template, fill in the business logic, and the error handling and logging are already done.

52-page field guide (PDF):

Not an API reference. Not a man page reformatted as a PDF. A field guide — structured as the things you need to know in the order you need to know them when you're setting up automation on a new server.

Covers the why behind every pattern in the scripts. Why set -euo pipefail and what each flag actually prevents. Why traps on EXIT instead of just INT. Why lock files need stale detection. Why backup retention has to be a separate step from the backup itself. Why SSL monitoring should be independent of your renewal tool.

Each section includes the real failure scenario that motivated the pattern, because "best practice" without the consequence attached is advice that gets skipped.

Why $9

I thought about this for a while. The scripts are worth more than $9 to anyone who's going to use them — preventing one 4am incident pays for the toolkit immediately. But I also know what it's like to be the person running a $5 VPS on a budget, and I wanted the price to be low enough that buying it doesn't require approval from anyone or a second thought.

$9. MIT license. Unlimited personal and commercial use. You can deploy these on client servers, modify them however you want, include them in your own automation. No subscription. No upsell. No "starter tier" with premium features behind another paywall.

The free snippets on the site are not going away. They're not a demo. They're complete, working scripts that I actively maintain. The toolkit is for people who want the production layer — the shared library, the integrated system, the field guide that ties it together — and want it in one download instead of building it themselves.

Who This Is For

If you're managing one or more Linux servers and you don't already have automated monitoring, backups, and alerting set up — this is the fastest path to having all three. Copy the scripts to the server, edit the config variables at the top of each file, add the cron entries from the guide, and you have a monitoring layer that didn't exist 10 minutes ago.

If you already have these things set up and you built them yourself, you probably don't need this. You might find something useful in bashlib.sh — the stale lock detection or the dry-run mode — but the scripts themselves won't tell you anything new.

If you're learning bash and want to see how production scripts are structured differently from tutorial scripts, the field guide is probably the most useful part. The "why" sections explain patterns that most bash tutorials skip because they're not relevant to a single-file script running on a laptop.

Every Script Passes ShellCheck at Zero Warnings

This one matters to me. ShellCheck is the static analysis tool for bash. It catches the bugs that work on happy-path input and break on edge cases — unquoted variables that split on whitespace, pipelines that swallow exit codes, deprecated syntax that newer bash versions handle differently.

Every script in the toolkit, including bashlib.sh, passes ShellCheck at the strictest severity level with zero warnings. Not "a few style notes we decided were fine." Zero. I treat ShellCheck warnings the same way I treat compiler warnings in C — they are bugs I haven't hit yet, and ignoring them is technical debt with a guaranteed due date.

If you run ShellCheck on these files and get output, something went wrong and I want to know about it.

The Link

→ bashsnippets.xyz/starter-kit

$9. Instant download. 6 production scripts + bashlib.sh shared library (31 functions) + template.sh + 52-page field guide. ShellCheck-clean. MIT license.

Already have scripts and need somewhere to run them? DigitalOcean droplets start at $4/month and you can get $200 in free credit to start:

→ Get $200 free credit — DigitalOcean

The free snippet library with 17+ scripts and 7 interactive tools is at:

→ bashsnippets.xyz

6 small things we shipped across the BashSnippets tools this week

Anguishe — Tue, 09 Jun 2026 03:49:07 +0000

Nobody announces small features. You ship them, they're in there, and the people who find them either notice or they don't. I want to start documenting these because some of them are the kind of thing that makes a tool actually worth using day-to-day instead of being something you visit once and close.

Six updates across six tools this week. None of them are headline features. All of them are fixes for specific annoyances that came up in my own usage, which is the only real source I trust for "does this actually help."

1. Bash Boilerplate Generator — Trap & Cleanup Handler

There's a now a toggle for trap handling. When on, the generated template includes:

cleanup() {
  # Remove temp files, release locks, undo partial changes
  rm -f "$TMPFILE"
}

trap cleanup EXIT INT TERM

Why this matters: trap cleanup EXIT means the cleanup() function runs no matter how the script exits. Normal completion. Ctrl+C. An uncaught error. A kill signal. The cleanup function runs. Every time.

Without this pattern, scripts that create temp files leave them behind when they're interrupted. Scripts that acquire lock files leave them locked. Scripts that make partial changes — creating a directory, writing part of a config — leave the system in an inconsistent state because the cleanup code at the end of the script never ran.

The trap-on-exit pattern is not optional for anything running unattended. It's the thing that separates "runs fine when nothing goes wrong" from "also recovers gracefully when something does." I've seen this pattern left out of boilerplate generators enough times that I wanted to make it explicit and easy to include.

The generated stub includes a TMPFILE=$(mktemp) line as a concrete example of something that needs cleanup. Replace it with whatever state your script actually manages.

→ bashsnippets.xyz/tools/bash-boilerplate-generator

2. Bash Exit Code Lookup — Export as `case` Statement

After looking up an exit code, there's a button that generates a ready-to-paste case $? in ... esac block with the explanation inline as a comment.

Before this, the lookup gave you the meaning of the exit code and you had to write the case handler yourself. That's not hard — the case syntax is simple — but it's friction. You looked up what 126 means ("command found but not executable — check permissions"), now you have to translate that into:

case $? in
  0)   echo "Success" ;;
  1)   echo "General error" ;;
  126) echo "Command found but not executable — check file permissions" ;;  # ← the one you just looked up
  127) echo "Command not found — check PATH or typo" ;;
  *)   echo "Unknown exit code: $?" ;;
esac

The export button generates that block for you, with the specific code you looked up highlighted in the right place and the explanation preserved as a comment. Paste it directly into your script. No rewriting the syntax from scratch.

The case template includes the four most common exit codes (0, 1, 126, 127) plus the wildcard catch-all. Delete the ones you don't need.

→ bashsnippets.xyz/tools/bash-exit-code-lookup

3. Cron Job Builder — Next 5 Run Times

This is the one I wanted for myself.

Enter any cron expression — say, 0 3 * * 1-5 — and it now immediately shows the next five times it will fire:

Next run times for: 0 3 * * 1-5

1.  Mon Jun 09 2026  03:00:00
2.  Tue Jun 10 2026  03:00:00
3.  Wed Jun 11 2026  03:00:00
4.  Thu Jun 12 2026  03:00:00
5.  Fri Jun 13 2026  03:00:00

The problem this solves: cron expressions are easy to get subtly wrong. 0 3 * * * fires at 3am. But 3am what — your server's local time or UTC? And is your server set to UTC? And does */6 mean every 6 hours starting at midnight, or starting at the first minute it's defined? And does 0 9 * * 1 fire on Monday or Sunday, depending on which cron implementation counts week starts?

Without something that shows you the actual fire times, you add the job, wait until the next day, and find out whether your assumptions were right or wrong. If they were wrong, you change it and wait another day. It can take three or four days to confirm a cron expression fires when you want it to, purely because of iteration time.

Showing the next five run times eliminates that cycle entirely. You know immediately whether your expression does what you think it does. Change it, verify it, add it to your crontab — all in under a minute.

→ bashsnippets.xyz/tools/cron-job-builder

4. Chmod Calculator — World-Writable Warning and Live Symbolic Mirror

Two updates here bundled together because they're related.

World-writable warning:

Enabling any world-write bit (the w in the "others" column) now shows a warning banner:

⚠️ World-writable permissions mean any user on the system can modify this file or directory. This is rarely correct. Verify this is intentional.

This is the chmod 777 trap. I've seen chmod 777 applied as a "quick fix" for permission errors more times than I can count. It works immediately, which is why people do it. The fact that it means "literally every user and process on this system can write to this file" gets lost in the urgency of making the error go away.

The warning doesn't prevent you from setting world-writable permissions. It just makes sure you've seen the words "any user on the system can modify this" before you click confirm. If you intended that, the warning is just noise. If you didn't, it might save you.

Live symbolic ↔ octal mirror:

Every permission you configure now shows both forms simultaneously:

Octal:    chmod 755
Symbolic: chmod u=rwx,go=rx

Both update in real time as you toggle permissions. This is useful for two reasons: you see the octal code for when you need to type it quickly in a terminal, and you see the symbolic form for when you need to express the same permission in a script where the explicit form is easier to read and maintain. Neither is "more correct" — they're the same thing expressed two ways, and having both removes the mental step of converting between them.

→ bashsnippets.xyz/tools/chmod-permissions-builder

5. PATH Debugger — Duplicate Entry Detector

Duplicate entries in $PATH accumulate over years. Every time a tool's installer adds itself to your PATH, every time you add an entry to .bashrc, every time a package manager prepends its bin directory to your environment — the list grows. On machines that have been around for a while, or on developer laptops where tools get installed and removed and reinstalled, you can end up with /usr/local/bin listed four times.

Duplicate entries don't usually cause obvious breakage. The correct binary still runs. It just runs with slightly more PATH resolution overhead on every command, and the duplicate entries make it harder to reason about which version of a tool will actually be picked when you have multiple installed. If /usr/local/bin appears before /usr/bin, the local install wins. When it appears four times, you've lost track of the actual resolution order.

The debugger now flags repeated entries and generates a one-liner to deduplicate and export a clean PATH:

# Generated dedup command:
export PATH=$(echo "$PATH" | tr ':' '\n' | awk '!seen[$0]++' | tr '\n' ':' | sed 's/:$//')

That pipeline: converts PATH from colon-separated to newline-separated, uses awk to keep only the first occurrence of each entry, converts back to colon-separated, and strips the trailing colon. The resulting PATH has the same resolution order as the original but with all duplicates removed.

Add that line to the bottom of your .bashrc and your PATH will deduplicate itself on every new shell session.

→ bashsnippets.xyz/tools/path-debugger

6. ShellCheck Error Decoder — Inline Script Scanner

ShellCheck is the best static analysis tool for bash scripts. If you're writing bash and you're not running your scripts through ShellCheck, you're missing real bugs. I'll say that plainly.

The problem is that ShellCheck error codes (SC2086, SC2046, SC1091, etc.) are meaningful if you know what they mean and opaque if you don't. When ShellCheck tells you SC2086: Double quote to prevent globbing and word splitting, that makes sense. When it just gives you SC2086 in isolation, you have to look it up.

The inline script scanner adds a paste box where you put your bash script directly. The tool maps recognized SC error codes to the lines in your script where they appear — not in the abstract documentation, but in your specific code. You see the line, the SC code, and the explanation together.

Important caveat I want to be direct about: this is not a replacement for running actual ShellCheck. The real ShellCheck tool parses bash properly, handles edge cases, and catches issues that a pattern-matcher won't. If you're writing scripts that matter, install ShellCheck and run it directly:

# Install
apt install shellcheck       # Debian/Ubuntu
brew install shellcheck      # macOS

# Run
shellcheck yourscript.sh

What the decoder in BashSnippets tools is useful for: understanding what a specific SC code means in the context of your own code rather than in a reference page. It's a learning tool and a quick lookup, not a substitute for the real thing.

→ bashsnippets.xyz/tools/shellcheck-error-decoder

All tools are free, no login, no account required. The full tools index is at:

→ bashsnippets.xyz/tools

→ bashsnippets.xyz

I was handed a server with no docs and no idea what it was listening on. One command fixed that.

Anguishe — Tue, 09 Jun 2026 00:04:46 +0000

A client handed me SSH access to a server they'd been running for two years.

No documentation. No handoff notes. No "here's what's running and why." Just a root password in a LastPass share and a Slack message that said "it hosts our web app, let us know if you need anything."

I didn't know what the web app was. I didn't know what was installed. I didn't know what ports were open to the internet, what services were running, or what this machine had been doing quietly for 730 days before I touched it.

This is not unusual. This is actually most of the "inherited server" situations I've been in. The person who set it up is gone, or was a contractor, or was the founder who also did DevOps because someone had to, and the documentation lived entirely inside one person's head and left with them.

The first thing I do on an unfamiliar server is not grep logs or check running services. It's figure out what the machine is actually reachable on from the outside. Not what anyone says it's supposed to be doing. What it is doing. What ports are in LISTEN state, what process owns each one, and whether that process is bound to localhost or to every network interface on the machine.

I ran netstat -an first, because that's what I knew at the time. I got back 200 lines of connection state. TIME_WAIT entries for connections that had already closed. ESTABLISHED entries for active sessions. CLOSE_WAIT entries from something that hadn't cleaned up properly. All of it technically useful for debugging specific connection issues. None of it useful for getting a fast security picture of what's actually listening for new connections.

Finding the ports in LISTEN state in netstat -an output feels like searching for a specific ingredient in a paragraph of text. It's all there, but you have to read every line.

Then someone showed me ss.

The Command

ss -tlnp

That's the whole thing.

-t — TCP only (use -u for UDP, or -tu for both)

-l — listening sockets only (filters out ESTABLISHED, TIME_WAIT, everything else)

-n — numeric output (don't resolve port numbers to service names, don't do reverse DNS)

-p — show the process name and PID holding each socket

Output on a typical web server:

State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       128     0.0.0.0:22           0.0.0.0:*          users:(("sshd",pid=1234,fd=3))
LISTEN  0       511     0.0.0.0:80           0.0.0.0:*          users:(("nginx",pid=5678,fd=6))
LISTEN  0       511     0.0.0.0:443          0.0.0.0:*          users:(("nginx",pid=5678,fd=7))
LISTEN  0       128     0.0.0.0:5432         0.0.0.0:*          users:(("postgres",pid=9012,fd=5))

Port 22 — SSH. Expected.

Port 80, 443 — nginx. Expected.

Port 5432 — Postgres. Bound to 0.0.0.0.

That last one.

0.0.0.0:5432 means Postgres was accepting connections from any IP address on the internet. Not just localhost. Not just the application server. Any IP. Port 5432, wide open, reachable from anywhere.

It had a strong password. It had been running that way for two years without an incident anyone knew about. But "strong password" is not a substitute for "not exposed to the internet." A service that should only accept connections from localhost or from your application tier has no business being bound to 0.0.0.0.

One command. Three seconds. Caught a misconfiguration that had been sitting there since the server was provisioned.

What the Output Is Actually Telling You

The column that matters is Local Address. This is where the socket is bound — who it will accept connections from.

0.0.0.0:5432    → Externally reachable on all IPv4 interfaces
127.0.0.1:5432  → Localhost only — not reachable from outside
:::5432         → All IPv6 interfaces (equivalent to 0.0.0.0 for IPv6)
::1:5432        → IPv6 localhost only

If a service should only be accessed from the same machine — a database, an internal cache, a monitoring agent — it should be bound to 127.0.0.1. If you see it bound to 0.0.0.0 and you don't have an explicit reason for that, that's a conversation to have with whoever configured the service.

For the Postgres case, the fix is one line in postgresql.conf:

listen_addresses = 'localhost'

Restart Postgres, confirm the bind address changed with ss -tlnp again, done. That configuration change is a 30-second fix. The two years it had been wrong before someone checked is the part that should concern you.

Why `ss` Instead of `netstat`

ss replaced netstat on most Linux distributions when iproute2 superseded net-tools around 2016. On a fresh Ubuntu or Debian install, netstat isn't installed by default — it's in the net-tools package which isn't pulled in automatically anymore. That's why you've probably SSH'd into a server, typed netstat, and gotten command not found.

ss reads kernel socket tables directly instead of going through /proc/net/tcp. It's faster on machines with thousands of connections, and it's maintained. net-tools has been in maintenance-only mode for years. For new work, use ss.

That said, netstat -tlnp does the same thing if you have it installed. The flags are identical. If you're on an older system or have net-tools available, either works.

The Process Name Caveat

Run ss -tlnp without sudo and you'll see the ports and bind addresses, but the Process column will be empty for sockets owned by other users. You'll see the port, you won't see what's holding it.

sudo ss -tlnp

With sudo, you see everything — the socket, the process name, the PID, the file descriptor number. That's the version to run when you're doing a security audit on a machine you have root on.

The `lsof` Alternative

lsof -i -P -n | grep LISTEN gives you the same information from a different angle — process name first, port second.

nginx    5678 root   6u  IPv4  12345  0t0  TCP *:80 (LISTEN)
nginx    5678 root   7u  IPv4  12346  0t0  TCP *:443 (LISTEN)
sshd     1234 root   3u  IPv4  12347  0t0  TCP *:22 (LISTEN)
postgres 9012 postgres 5u IPv4 12348  0t0  TCP *:5432 (LISTEN)

Useful when you know the service name and want to find its ports. Less useful when you want a port-first view of everything listening. I use ss for the initial audit and lsof when I'm tracking down a specific process.

What I Actually Do With Ports I Can't Identify

When I see a port in LISTEN state and the process name isn't immediately obvious — java, python3, something that isn't a clear service name — I do three things:

# 1. Find the full command that started the process
ps aux | grep <PID>

# 2. Check what package installed the binary
dpkg -S /path/to/binary    # Debian/Ubuntu
rpm -qf /path/to/binary    # RHEL/CentOS

# 3. Check what the process has open (files, connections, everything)
sudo lsof -p <PID>

Most of the time it's something benign — a monitoring agent, a language runtime for an app, a service the previous admin installed and forgot about. Occasionally it's something that shouldn't be there at all. You don't know until you check.

Building This Into Your Server Checklist

Every time I take over a server now, this is the third command I run. After uptime (how long has it been running) and df -h (is it about to run out of disk), it's sudo ss -tlnp (what is this machine telling the internet it's listening on).

Takes three seconds. Has caught real problems more than once. The Postgres 0.0.0.0 situation is the one I remember most clearly, but it's not the only one — I've found web apps listening on non-standard ports that were supposed to be internal, Redis instances bound to 0.0.0.0 on development machines that got promoted to production without anyone auditing the config, SSH running on a second non-standard port "for convenience" that had been left open after a migration.

None of those were catastrophic on their own. All of them were things the people running those servers didn't know were there.

Three seconds to find out. I don't know why I didn't run this on every server from the beginning.

Full script with UDP ports, both TCP and UDP in one pass, lsof cross-reference, and notes on what to do when you can't identify a process:

→ bashsnippets.xyz/snippets/list-open-ports-linux

→ bashsnippets.xyz

certbot had been auto-renewing my certs for two years. Until it wasn't.

Anguishe — Sun, 07 Jun 2026 05:08:19 +0000

certbot had been running quietly on my server for almost two years without a single issue.

Automatic renewals. Silent cron job. I never thought about it. Set it up once, tested it once, and mentally moved it to the "solved" column of my infrastructure checklist. That column felt good. certbot goes in there and stays there.

Then I checked my site on a Monday morning and got the red browser warning.

Not "connection is slow." Not a timeout. The full-page block. Chrome's red lock icon. "Your connection is not private." NET::ERR_CERT_DATE_INVALID in small gray text underneath, in case you wanted the clinical version of bad news.

I SSHd in immediately. The cert had expired three days earlier.

certbot had been running, technically. The cron job was firing. No errors in /var/log/syslog that I was watching. But the HTTP challenge was failing silently — something to do with a port conflict during renewal. A service I'd added a few months back was binding to port 80 for its own health check, and certbot couldn't complete the ACME challenge because port 80 wasn't free during the brief window it needed it. The renewal attempt failed. certbot logged it somewhere I wasn't watching. The cron job reported success because the cron job ran — it just happened to run a certbot process that quietly gave up.

No alert email. No log message in any file I had on my radar. Just a quiet failure, three renewal cycles in a row, and then an expired certificate.

I found out from a user who emailed me. Not from monitoring. Not from a script. From a stranger who was kind enough to say "hey, your SSL is broken" instead of just closing the tab.

The renewal fix took five minutes once I found the port conflict. What I didn't have — what I wish I'd had — was a script that told me the cert was about to expire before it happened. Something running every morning, checking the actual live certificate the way a browser would check it, and reporting "you have 18 days left — go fix this."

Turns out openssl can do this in one command. It's been able to do this for years. I just never looked.

The Core Command

echo | openssl s_client -connect yoursite.com:443 -servername yoursite.com \
  2>/dev/null | openssl x509 -noout -enddate

Run that and you get back something like:

notAfter=Aug 14 12:00:00 2026 GMT

That's the raw expiry date straight from the live certificate your server is presenting to the world. Not what certbot thinks. Not what your local files say. What a browser actually sees when it connects.

To turn that into days remaining:

EXPIRY=$(echo | openssl s_client -connect yoursite.com:443 -servername yoursite.com \
  2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

DAYS=$(( ($(date -d "$EXPIRY" +%s) - $(date +%s)) / 86400 ))

echo "$DAYS days remaining on SSL cert"

That arithmetic converts two Unix timestamps (the expiry date and now) to seconds, subtracts them, and divides by 86400 to get days. It's ugly but it works.

Three Non-Obvious Things I Ran Into

1. The -servername flag is not optional on SNI hosts.

SNI — Server Name Indication — is how a single IP address serves multiple domains with different SSL certificates. Most shared hosting and most modern VPS setups use it. Without -servername yoursite.com, openssl doesn't know which certificate to request. It reads the server's default certificate instead of yours.

I spent twenty minutes getting clean results from this command before I realized I was checking the wrong cert. My server's default certificate was still valid. My production domain's certificate was the one expiring. I would have gotten a false "everything is fine" reading and missed the problem entirely. Always include -servername.

2. The echo | pipe is required for non-interactive use.

Without it, openssl s_client waits for stdin input after establishing the connection. In a cron job, that means the script hangs forever, waiting for input that never comes, holding the connection open until something kills it. The echo sends empty input immediately, which closes the connection after the certificate handshake completes. This is one of those things where the command works perfectly in your terminal and completely silently breaks in cron. The echo | is what makes it safe to automate.

3. The date -d syntax is Linux-only.

macOS uses date -j -f "%b %d %T %Y %Z" "$EXPIRY" +%s instead of date -d "$EXPIRY" +%s. If you're building a script that needs to run on both platforms, you'll need a conditional. The full version of the script at the link below handles this with an OS check. If you're only ever running this on Linux servers — which is probably most of you — you don't need to care about this.

The Full Script

#!/bin/bash

CHECK="✓"
CROSS="✗"

# --- Configuration ---
DOMAINS=(
  "yourdomain.com"
  "anotherdomain.com"
)
WARN_DAYS=30      # Alert if fewer than this many days remain
PORT=443

# --- Check each domain ---
for DOMAIN in "${DOMAINS[@]}"; do
  EXPIRY=$(echo | openssl s_client \
    -connect "${DOMAIN}:${PORT}" \
    -servername "$DOMAIN" \
    2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

  if [[ -z "$EXPIRY" ]]; then
    echo "$CROSS Could not retrieve cert for $DOMAIN — is the server reachable?"
    continue
  fi

  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null)
  NOW_EPOCH=$(date +%s)
  DAYS=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

  if [[ "$DAYS" -lt 0 ]]; then
    echo "$CROSS $DOMAIN — CERT EXPIRED ${DAYS#-} days ago"
  elif [[ "$DAYS" -lt "$WARN_DAYS" ]]; then
    echo "$CROSS $DOMAIN — WARNING: $DAYS days remaining (expires $EXPIRY)"
  else
    echo "$CHECK $DOMAIN — OK: $DAYS days remaining"
  fi
done

Sample output with two domains:

✓ yourdomain.com — OK: 74 days remaining
✗ anotherdomain.com — WARNING: 11 days remaining (expires Sep 2 12:00:00 2026 GMT)

Why 30 Days as the Threshold

Let's Encrypt certificates are 90 days. certbot typically renews at 60 days remaining — 30 days before the default renewal window. If my monitoring fires at 30 days, I have a full renewal cycle's worth of time to figure out whatever certbot is failing on before anything breaks.

That's the math. 30 days is not conservative for its own sake — it's exactly one full renewal attempt window on a 90-day cert. If you're using a 1-year cert from a paid CA, you probably want 60 days. If you're on Let's Encrypt, 30 days gives you two full automated renewal attempts before you hit zero.

Adding an Email Alert

If you want to be notified instead of (or in addition to) just logging, add this to the warning block:

if [[ "$DAYS" -lt "$WARN_DAYS" ]]; then
  echo "SSL cert for $DOMAIN expires in $DAYS days" | \
    mail -s "CERT EXPIRY WARNING: $DOMAIN" you@youremail.com
fi

This requires mailutils or sendmail to be configured on your server. If you don't have email set up, the cron log version is enough — as long as you actually read the log, which is its own challenge.

The Cron Setup

0 8 * * * /home/user/check-ssl-expiry.sh >> /var/log/ssl-check.log 2>&1

Every morning at 8am. Logs to /var/log/ssl-check.log. If something fires, the line is there when you check in the morning. If everything is green, the log file is a quiet record that your certs are healthy.

I run this across all my domains every day. 30-day warning threshold. On a Let's Encrypt setup that gives two full auto-renewal cycles before anything breaks. No more Monday morning surprises.

The cert that expired on me was a $0 certificate on a $5 server. The cost wasn't the problem. The embarrassment of a user telling me before my own monitoring did — that was the part I wasn't willing to repeat.

Full script with multi-domain array, configurable threshold, email alert variation, macOS compatibility, and cron setup instructions:

→ bashsnippets.xyz/snippets/check-ssl-certificate-expiry

→ bashsnippets.xyz

set -euo pipefail — the Line That Would Have Saved Me from Deleting the Wrong Directory

Anguishe — Wed, 03 Jun 2026 21:40:44 +0000

Here's a script that will ruin your day:

#!/bin/bash
cd /nonexistent/folder
rm -rf *
echo "Done"

The cd fails because the folder doesn't exist. Bash ignores the failure. rm -rf * runs in whatever directory you were already in. The script prints "Done." You have no idea anything went wrong until you notice your files are gone.

This isn't a contrived example. This is the actual failure mode of every bash script that doesn't have error handling. Bash's default behavior is to keep running after errors. Every command after a failure executes as if everything is fine. It's the most dangerous default in the entire language.

Two lines fix it.

The Template

#!/bin/bash
set -euo pipefail

trap 'echo "Error on line $LINENO — script stopped." >&2' ERR

# Your script starts here

Add this to the top of every script you write. Every single one. No exceptions.

What Each Flag Does

-e (errexit) — Stop the script immediately if any command exits with a non-zero status. This is the one that prevents the cd + rm disaster above. If cd fails, the script stops right there. rm never runs.

-u (nounset) — Treat any undefined variable as an error. Without this, $UNDEFINED_VAR silently becomes an empty string. Imagine this:

BACKUP_DIR=""  # Oops, forgot to set this
rm -rf "$BACKUP_DIR"/*

That expands to rm -rf /*. That's your entire filesystem. With -u, bash catches the empty variable and stops before the rm ever runs.

-o pipefail — Make a pipeline fail if ANY command in it fails, not just the last one. Without this:

cat nonexistent-file.txt | grep "pattern" | wc -l

cat fails, but grep gets empty input, and wc -l counts zero lines and exits successfully. The pipeline reports success even though the first command failed. With pipefail, the pipeline's exit status is the failure from cat, so -e catches it.

The `trap` Line

trap 'echo "Error on line $LINENO — script stopped." >&2' ERR

This tells bash: "When an error happens, before you exit, run this command." It prints which line number failed and writes to stderr (>&2). Without this, the script just stops silently and you have to guess which line caused it.

In a 200-line script, "it failed somewhere" is useless. "Error on line 47" is actionable.

The Real-World Difference

Without error handling:

#!/bin/bash
cd /tmp/build
make
make install
echo "Build complete"

If cd fails (the directory doesn't exist), make runs in the wrong directory. If make fails (compilation error), make install installs whatever was there from last time. The script prints "Build complete" regardless.

With error handling:

#!/bin/bash
set -euo pipefail
trap 'echo "Error on line $LINENO" >&2' ERR

cd /tmp/build
make
make install
echo "Build complete"

If cd fails, the script stops and prints "Error on line 5." If make fails, same thing. echo "Build complete" only runs if every single command before it succeeded.

The One Gotcha

set -e changes how you write conditional logic. This fails:

set -e
grep "pattern" file.txt  # exits 1 if pattern not found
echo "This never runs"

grep returns exit code 1 when it finds no matches. With -e, that's treated as an error and the script stops. But you didn't want it to stop — you just wanted to check if the pattern exists.

The fix is to use it in a conditional:

set -e
if grep -q "pattern" file.txt; then
  echo "Found it"
else
  echo "Not found"
fi

When a command is part of an if condition, -e doesn't trigger on its exit code. This is the standard pattern.

Or suppress the exit code explicitly:

grep "pattern" file.txt || true

The || true means "if grep fails, run true instead" — and true always succeeds, so -e doesn't trigger.

Where to Go from Here

Every script on BashSnippets.xyz uses set -euo pipefail and the CHECK="✓" / CROSS="✗" pattern. If you want a head start, the Bash Boilerplate Generator builds a complete script template with error handling, logging, and cleanup traps already configured. Pick your options, copy the output, and start writing from a safe foundation.

Full breakdown with more examples, the trap pattern, and common pitfalls:

bashsnippets.xyz/snippets/bash-error-handling.html

If you only take one thing from this: add set -euo pipefail to the top of every script. It takes 3 seconds and prevents the kind of failures that take hours to recover from.

I Aliased 'syscheck' to 7 Lines of Bash and Now I Run It on Every Server I SSH Into

Anguishe — Mon, 01 Jun 2026 01:38:40 +0000

Every time I SSH into a server, the first thing I want to know is: how's it doing?

Not a deep dive. Not a monitoring dashboard. Just the basics: how long has it been running, how much RAM is left, is the disk getting full, what's the IP. Five things that take five separate commands to check — hostname, uptime, free, df, hostname -I — and I'm tired of typing all five every single time.

So I put them in one script and aliased it to syscheck. Now I SSH in, type one word, and know the state of the machine in 2 seconds.

The Script

#!/bin/bash
# Quick System Info Report
# Prints key stats at a glance.
# Alias to 'syscheck' in your .bashrc for fast access.

echo "=== Quick System Check ==="
echo "Host    : $(hostname)"
echo "Uptime  : $(uptime -p)"
echo "RAM     : $(free -h | awk '/Mem/{print $3"/"$2}')"
echo "Disk /  : $(df -h / | awk 'NR==2{print $3"/"$2}')"
echo "IP      : $(hostname -I | awk '{print $1}')"
echo "========================="

Output looks like:

=== Quick System Check ===
Host    : my-server
Uptime  : up 3 days, 4 hours
RAM     : 1.2G/2.0G
Disk /  : 8.3G/25G
IP      : 192.168.1.42
=========================

Seven lines. No dependencies. Every command used here ships pre-installed on every Linux distribution I've ever touched.

What Each Line Actually Does

hostname — prints the machine name. Obvious, but when you're managing 4 servers with different roles, seeing "staging-web" vs "prod-db" at the top saves you from running the wrong command on the wrong box.

uptime -p — the -p flag gives you human-readable output like "up 3 days, 4 hours" instead of the default uptime output which includes load averages and the current time and is harder to parse at a glance.

free -h | awk '/Mem/{print $3"/"$2}' — free -h shows memory in human-readable format (GB instead of bytes). The awk part grabs the "used" and "total" columns from the "Mem:" line and formats them as 1.2G/2.0G. You see at a glance whether you're at 60% RAM or 95%.

df -h / | awk 'NR==2{print $3"/"$2}' — same idea but for disk. df -h / checks the root partition, and awk pulls used and total. If this says 23G/25G you know you need to clean up soon.

hostname -I | awk '{print $1}' — prints the machine's IP address. The awk grabs just the first one in case there are multiple network interfaces.

The Alias Setup (The Part That Makes It Worth It)

The script is useful. The alias is what makes you actually use it.

nano ~/.bashrc

Add at the bottom:

alias syscheck='/home/user/syscheck.sh'

Or if you want the whole thing inline without a separate file:

alias syscheck='echo "=== Quick System Check ===" && echo "Host    : $(hostname)" && echo "Uptime  : $(uptime -p)" && echo "RAM     : $(free -h | awk '"'"'/Mem/{print $3"/"$2}'"'"')" && echo "Disk /  : $(df -h / | awk '"'"'NR==2{print $3"/"$2}'"'"')" && echo "IP      : $(hostname -I | awk '"'"'{print $1}'"'"')" && echo "========================="'

Then reload:

source ~/.bashrc

Now type syscheck on any terminal session on that machine and you get the full report instantly. I put this in .bashrc on every server I set up. It's the first thing I configure after SSH key access.

Extending It

Add CPU load:

echo "CPU     : $(top -bn1 | grep 'Cpu(s)' | awk '{print $2}')% used"

Add the top 3 processes by memory:

echo "Top RAM : $(ps aux --sort=-%mem | awk 'NR<=4{print $11}' | tail -3 | tr '\n' ' ')"

Add the OS version:

echo "OS      : $(cat /etc/os-release | grep PRETTY_NAME | cut -d'"' -f2)"

I keep the base version minimal on purpose. When I need deeper visibility, I have separate scripts for CPU and RAM monitoring and disk space warnings. syscheck is the quick glance. Those are the deep dive.

Where This Fits in the Workflow

I SSH into a server. I type syscheck. If everything looks normal, I do whatever I came to do. If RAM is at 95% or disk is nearly full, I investigate before touching anything else.

It's the 2-second sanity check that prevents the "oh no, why is everything slow" surprise 20 minutes into a debugging session when you finally think to check resources and realize the disk filled up an hour ago.

Full script, alias setup, and more extension ideas:

bashsnippets.xyz/snippets/quick-system-info-report.html

DEV Community: Anguishe

The Alert Never Fired Because the Loop Skipped the Last Line of the File

Why read drops the last line

What IFS= and -r actually do

The subshell trap that kills your counters

Why for loop is wrong for this

The monitoring system, after the fix

A For Loop Skipped Every File With a Space and Called the Backup a Success

Why $(ls) word-splits

The second half: quoting on use

Ranges and counters: where the brace trap hides

Arrays: the original bug wearing a different hat

The nullglob case

What happened to the Q3 export

find . -delete Ran Before the Filter and Emptied the Whole Tree

Why the order is the program

The quoting trap, right behind the ordering one

The age sign that everyone reverses at least once

The -exec batching difference nobody explains

The preview step I skipped the day I broke things

What the builder actually does

What I upgraded while I was at it

A for Loop Skipped 23 Files and Called It a Successful Backup

The Pipeline Was Green for Three Weeks. It Had Been Shipping a Build That Never Compiled.

My Script Crashed and Left a Lock File Behind. Every Run After That Refused to Start.

I Packaged the Scripts I Copy to Every New Server Into a $9 Toolkit. Here's What's In It and Why.

What's Different Between the Free Snippets and the Toolkit

What's in the Box

Why $9

Who This Is For

Every Script Passes ShellCheck at Zero Warnings

The Link

6 small things we shipped across the BashSnippets tools this week

1. Bash Boilerplate Generator — Trap & Cleanup Handler

2. Bash Exit Code Lookup — Export as case Statement

3. Cron Job Builder — Next 5 Run Times

4. Chmod Calculator — World-Writable Warning and Live Symbolic Mirror

5. PATH Debugger — Duplicate Entry Detector

6. ShellCheck Error Decoder — Inline Script Scanner

I was handed a server with no docs and no idea what it was listening on. One command fixed that.

The Command

What the Output Is Actually Telling You

Why ss Instead of netstat

The Process Name Caveat

The lsof Alternative

What I Actually Do With Ports I Can't Identify

Building This Into Your Server Checklist

certbot had been auto-renewing my certs for two years. Until it wasn't.

certbot had been running quietly on my server for almost two years without a single issue.

The Core Command

Three Non-Obvious Things I Ran Into

The Full Script

Why 30 Days as the Threshold

Adding an Email Alert

The Cron Setup

set -euo pipefail — the Line That Would Have Saved Me from Deleting the Wrong Directory

The Template

What Each Flag Does

The trap Line

The Real-World Difference

The One Gotcha

Where to Go from Here

I Aliased 'syscheck' to 7 Lines of Bash and Now I Run It on Every Server I SSH Into

The Script

What Each Line Actually Does

The Alias Setup (The Part That Makes It Worth It)

Extending It

Where This Fits in the Workflow

2. Bash Exit Code Lookup — Export as `case` Statement

Why `ss` Instead of `netstat`

The `lsof` Alternative

The `trap` Line