Dsecurity

How I Stressed My SQLite Job Queue to 5,000 Continuous Tasks on an Android Phone (And Why It Outperformed the Cloud)

DSECURITY — Tue, 26 May 2026 15:59:42 GMT

Every developer building a side project or a home automation pipeline eventually hits the same roadblock. You have a script running in the cloud (maybe a web scraper, a webhook handler, or an AI agent), and you need it to trigger a physical action on a local device—like an old Android phone running Termux, or a Raspberry Pi behind a strict home firewall.

The standard industry advice is immediate: "Just spin up Celery and back it with RabbitMQ or Redis." But for independent developers, indie hackers, and hobbyists, that answer feels terrible. Deploying and maintaining a heavy, memory-hungry message broker infrastructure for low-to-medium workloads is massive operational overkill. It’s expensive,introduces vendor lock-in if you go the managed cloud route, and forces you to manage background daemons that eat up system resources.

The alternative? Opening inbound network ports or setting up reverse SSH tunnels, which is a security nightmare.

That is why I built Intent Bus—a zero-infrastructure job coordination system powered entirely by a minimal Flask core and a local SQLite database. Instead of maintaining persistent stateful connections or heavy external brokers, your cloud script simply writes tasks to an HTTP endpoint, and your cross-device edge workers safely poll for jobs using basic outbound HTTP requests. It gives you atomic locking, priority scheduling, exponential backoffs, and dead-letter queues out of the box with zero operational friction.

But when you tell developers you built a concurrent message queue on top of SQLite, the skepticism is instant. "SQLite has a single-writer bottleneck." "It will thrash the disk." "It can't scale under real worker contention."

We just launched live on Product Hunt, and my co-maintainer Zan and I decided to stop talking about theory. We decided to break it. Here is how we stress-tested Intent Bus across the cloud, the limits we hit, and the completely unexpected way an old smartphone stole the show.

The Testing Strategy: Finding the Ceiling

To find exactly where our architectural trade-offs would give out, we established a rigorous profiling matrix across three distinct deployment environments:

PythonAnywhere (Free Tier): A standard python hosting environment.
Render (Free Tier Docker Container): A typical lightweight cloud container setup.
Android 12 Device (Termux via local ARM CPU & Flash Storage): True edge hosting.

We subjected the broker to progressively brutal worker/job ratios to watch the latency curve and catch database locks or lease drops in real-time.

Profile	Concurrency Workload
Low	5 active workers polling and fulfilling 50 total jobs.
Medium	15 active workers polling and fulfilling 500 total jobs.
Heavy	40 active workers polling and fulfilling 2,000 total jobs.
Extreme	100+ active workers polling and fulfilling 5,000 continuous jobs.

Phase 1: The Cloud Baseline (PythonAnywhere vs. Render)

We kicked off our benchmarks on a PythonAnywhere Free Tier instance. The results were an immediate failure under any real load. Because the free tier utilizes single-threaded request handling, concurrent workers polling the queue quickly created an unrecoverable request backlog. At medium load, it hit 100% network errors. Single-threaded processes are completely non-viable for worker polling loops.

Next, we shifted to a multi-threaded Docker deployment on Render's free tier. This is where the SQLite architecture started to shine.

On Medium load, Render sailed through with a 99.5% success rate, averaging 11.30 jobs/sec with a tight P99 worker latency of just 0.517 seconds.
When we pushed it to Heavy load (40 workers, 2,000 jobs), the system held steady at a 99.34% success rate and maintained a 14.18 jobs/sec throughput.

Because we engineered Intent Bus around a strictly optimized SQLite Write-Ahead Logging (WAL) configuration, the database handled concurrent reads comfortably. Under heavy writer contention, our locking mechanism gracefully degraded—tail latencies extended out to a P99 of 2.891 seconds, but the queue never crashed, never locked up, and leaked exactly zero jobs.

Phase 2: The Android Experiment (Pushing to the Extreme)

With a stable cloud baseline of ~14 jobs/sec, we decided to run an unconventional experiment: What happens if we host the central message broker right on the edge? We spun up the Intent Bus server inside Termux on a standard Android 12 phone, using its internal ARM processor and mobile flash storage as the entire infrastructure backbone.

The results completely blew past our expectations.

When we unleashed the Heavy load test (40 workers, 2,000 jobs) against the smartphone server, the mobile hardware didn't just keep up—it completely crushed the cloud container, clocking an astonishing 28.04 jobs per second with a P99 latency of 2.556 seconds. Because the physical phone flash storage didn't have to contend with the shared cloud hypervisor overhead found on free-tier containers, SQLite local disk I/O operations executed at blistering speeds.

Pushing to the Absolute Brink: 5,000 Continuous Jobs

Emboldened by the phone's performance, we threw our absolute worst-case scenario at it: The Extreme Profile. We unleashed over 100 concurrent worker loops, firing a non-stop barrage of 5,000 jobs over a sustained 4.5-minute beating (267 seconds).

This is what real system engineering looks like when it hits a hardware wall:

Graceful Degradation: As the mobile flash storage and ARM chip finally throttled under the sustained heat and massive file-write contention, our average throughput dropped from its peak down to a sustained 18.52 jobs/sec. Tail latencies spiked heavily, pushing the P99 worker response out to 9.002 seconds.
Absolute Structural Integrity: Despite the hardware bottleneck, look at the error counters. Zero network drops. Zero lease losses. Zero publish rejects. Even when the device was gasping for air, the protocol's locking logic and database state transitions remained flawless.
The Final Score: The system completed the marathon with a 98.89% success rate over 5,000 continuous tasks, proving it is virtually indestructible under load.

Architecture Takeaways: Why This Matters for Indie Devs

We didn't build Intent Bus to replace Kafka, RabbitMQ, or high-throughput enterprise systems. If you are building a multi-region corporate platform processing tens of thousands of requests per second, you need a traditional distributed broker architecture.

But if you are an indie hacker, a DevOps engineer tying personal scripts together, or an automation enthusiast coordinating a heterogeneous fleet of edge machines, these benchmarks change things.

They prove that a lightweight, properly optimized SQLite WAL backend is fundamentally robust enough to handle thousands of concurrent jobs with zero data corruption. It means you can completely bypass the cost and complexity of cloud server infrastructure. You can literally pull an old phone out of a drawer, install Termux, and host an incredibly reliable, self-contained, bulletproof automation queue entirely for free.

We are Live on Product Hunt!

Intent Bus is 100% open-source and written in Python. While we provide a dedicated Python SDK to handle signature generation and worker loops automatically, the underlying protocol is lightweight enough to interact with using nothing but pure bash and curl.

Our code is fully verified, our CI/CD pipelines are green, and our Product Hunt launch is officially live right now! If you want to check out the codebase, audit our security protocol updates, or leave us your honest feedback, check out the links below:

👉 Leave a Review on Product Hunt: Review

👉 GitHub Main Server:Intent-Bus

👉 GitHub SDK Repo:Python-SDK

I’ll be hanging out in the Product Hunt comments section all day to answer your questions—I'd love to hear your thoughts on our SQLite implementation, our capability routing, or our edge-computing performance results!

Building a zero-infra job queue with SQLite (and stress-testing it to 14 jobs/sec)

DSECURITY — Mon, 25 May 2026 10:17:16 GMT

I needed to trigger automation scripts on an old Android phone from a cloud VPS, without opening ports or spinning up Redis. So I built a job queue out of Flask and SQLite.

Here is a deep dive into what it is, how SQLite handles the concurrency, and what happened when I tried to stress-test it in production.

What it is (and what it isn't)

Intent Bus is designed to be the missing link between simple cron jobs and heavy enterprise message brokers.

What it aims for:

Giving indie hackers and home-lab enthusiasts actual execution guarantees (retries, dead-letter queues) without needing to deploy or maintain broker infrastructure.
Running anywhere. You can deploy the server on a free cloud tier, and run workers on a Raspberry Pi, a Termux phone, or a cheap VPS.

What it is not:

It is not a Kafka replacement. It is not meant for microservices processing 10,000 transactions per second. If you have enterprise scale, use enterprise tools.

The Core Features

Cron is strictly fire-and-forget, with no coordination and silent failures. To fix that, Intent Bus provides:

Feature	Description
Atomic Locking	Prevents double-claiming. A cryptographic lease token is issued to the claiming worker.
Reliable Delivery	If a worker crashes or drops offline, the lease expires and the job requeues with exponential backoff.
Dead-Letter Queue	Failed jobs (e.g., after 3 attempts) are archived for inspection rather than deleted.
Capability Routing	Workers advertise what they can do; jobs require what they need (e.g., require a worker with `ffmpeg`).
Priority Queues	High-priority intents are always claimed and processed first.

The SQLite Atomic Lock

The biggest challenge with using SQLite for a queue is concurrent writes. If 40 workers poll the server at the exact same millisecond, they can't all claim the same job.

Intent Bus solves this by strictly enforcing SQLite's WAL (Write-Ahead Logging) mode and relying on the UPDATE ... RETURNING clause introduced in SQLite 3.35.0.

When a worker asks for a job, the server executes a single atomic transaction:

UPDATE intents
SET
    status = 'claimed',
    claimed_at = :now,
    claim_expires_at = :lease_exp,
    claim_token = :token,
    claim_attempts = claim_attempts + 1
WHERE id = (
    SELECT id FROM intents 
    WHERE status = 'open' AND run_at <= :now
    ORDER BY priority DESC, created_at ASC
    LIMIT 1
)
RETURNING id, payload, claim_token;

Because SQLite serializes writes, this guarantees that only one worker will ever receive the ephemeral claim_token required to fulfill or fail that specific job.

The Stress Test: A Tale of Two Deployments

I wanted to see exactly where SQLite would choke under concurrent worker load, so I wrote an enterprise-style stress harness.

Attempt 1: The PythonAnywhere Disaster I originally deployed the architecture to PythonAnywhere's free tier. The results were catastrophic.

When I launched the high-intensity concurrent test, it didn't even manage to start. I stepped it down to the medium-intensity test (15 concurrent workers fighting for 500 jobs), and it threw 100% network errors.

Why? Because PythonAnywhere's free tier routes requests through a single-threaded Gunicorn worker. The concurrent polling requests bottlenecked at the WSGI layer, queued up, and timed out before SQLite even knew what hit it. It wasn't a database lock issue; it was a server concurrency issue.

Attempt 2: Docker on Render I moved the exact same SQLite file and Flask app to a basic Docker container on Render, configured with multiple application threads (--threads 4).

Render's free tier aggressively puts your container to sleep when idle, meaning the very first request takes about 30 seconds to wake the server up (the dreaded cold start). But once it's awake? It handles the load beautifully.

Here is the benchmark data from the Render Docker deployment:

Configuration	Workers	Jobs	Success	P99 Latency	Throughput
Light	5	50	100%	0.594s	3.72 j/s
Medium	15	500	98.75%	0.517s	13.27 j/s
Heavy	40	2000	99.01%	2.586s	13.62 j/s

Even with 40 concurrent workers furiously hammering the single SQLite file for write locks, there were 0 network errors, 0 lost leases, and 0 rate limit crashes. It comfortably sustained over 13 jobs per second.

Limits & The Theoretical Future

Right now, the system is tuned for safety. Payloads are strictly limited to 8KB to prevent database bloat, and busy_timeout PRAGMAs are set to allow workers to wait gracefully for locks.

Could it go further? In theory, yes. If you moved off a free cloud container onto a dedicated VPS with NVMe storage, and tuned SQLite for pure speed (e.g., PRAGMA synchronous=OFF), you could probably push 50-100 jobs a second. Furthermore, the architecture is simple enough that you could easily swap the get_db() call to point to PostgreSQL if you needed horizontal scaling.

But honestly, if you hit that scale, you should just bite the bullet and deploy Redis. The entire philosophy of Intent Bus is avoiding that infrastructure jump until it is absolutely necessary.

The server runs on Docker, Raspberry Pi, or any free cloud tier. Workers run anywhere that speaks HTTP — including Termux.

Core Server: GitHub - Intent Bus
Python SDK: GitHub - Intent Bus SDK

I would love to hear your feedback. Has anyone else experimented with using SQLite for job queueing? What edge cases did you run into?

I ditched Redis and built a job queue on SQLite. Here's what actually broke.

DSECURITY — Tue, 12 May 2026 03:22:35 GMT

I wanted to trigger a notification on my Android phone whenever a script finished running on a cloud server.

The obvious answer is Firebase, or Redis with a worker, or RabbitMQ. I looked at all of them. Every option meant spinning up something I would need to maintain forever just to move a handful of jobs between two devices.

So I built my own. A tiny HTTP job queue backed by Flask and SQLite. I called it Intent Bus.

This is not a story about why SQLite is underrated. You've read that article. This is a story about what happened when I actually tried to break it under concurrent load — and the bugs I found in my own test before I even got to the database.

The Core Problem: Atomic Claiming

The hardest part of any job queue is making sure exactly one worker claims each job. With Redis you get SETNX. With RabbitMQ you get acknowledgements baked in. With SQLite you get... a file.

My first instinct was a SELECT followed by an UPDATE. That's wrong. Between those two statements, ten workers running concurrently can all select the same job and all think they claimed it.

The fix is a single atomic statement using BEGIN IMMEDIATE and SQLite's RETURNING clause:

BEGIN IMMEDIATE;

WITH candidate AS (
    SELECT id FROM intents
    WHERE status = 'open'
      AND run_at <= :now
      AND expires_at > :now
      AND claim_attempts < max_attempts
      AND namespace = :namespace
      AND (publisher = :key OR visibility = 'public')
      AND (target_worker IS NULL OR target_worker = :worker_id)
    ORDER BY priority DESC, run_at ASC
    LIMIT 1
)
UPDATE intents
SET status = 'claimed',
    claimed_at = :now,
    claim_expires_at = :now + :timeout,
    claimed_by = :claimer,
    claim_attempts = claim_attempts + 1
WHERE id = (SELECT id FROM candidate)
  AND (status = 'open' OR (
    status = 'claimed' AND claim_expires_at < :now
  ))
RETURNING id, goal, payload, claim_attempts;

BEGIN IMMEDIATE acquires a write lock before any reads happen. The CTE selects the candidate, the UPDATE applies the condition a second time as a guard, and RETURNING hands back the result in the same statement. Either you get a row back or you don't. No race condition possible.

RETURNING requires SQLite 3.35+. The server refuses to start on older versions.

What the Queue Actually Does

Before the stress test, a quick summary of what I built:

Jobs have a goal (string), payload (JSON), namespace, and visibility (private or public)
Workers poll /claim, execute, and call /fulfill or /fail
Failed jobs are requeued with exponential backoff: now + (backoff_base × 2^attempts) + jitter
After max_attempts failures, jobs move to a dead-letter queue
Workers can advertise capabilities (X-Worker-Capabilities) and jobs can require them
Optional HMAC-SHA256 signing for replay protection

The whole server is a single flask_app.py. No external dependencies beyond Flask and Werkzeug.

The Stress Test

I wanted to know how SQLite WAL mode handles concurrent writers in practice, not in theory.

I wrote a test that runs three phases.

Phase 1A: Idempotency

Send the same job twice with the same Idempotency-Key header. The server should create exactly one job.

The server computes a SHA-256 hash of the request body and stores it alongside the idempotency key. On the second request, the hashes match and the cached response is returned. The second write never touches the intents table.

Result: one job created. ✓

Phase 1B: Max Attempts

Create a job with max_attempts: 2. Claim it and fail it twice. The third claim attempt should return 204 No Content because the job is now dead.

This one exposed a bug in my test, not the server.

After the first /fail, the server sets run_at = now + (backoff_base × 2^1) + jitter = now + ~10 seconds. The job is open but not yet claimable. My test immediately tried to claim again, got 204, and reported ✓ PASS — but that was a false positive. The job wasn't dead. It was just in backoff.

I had to add a time.sleep(15) between attempts to let the backoff window clear. After that fix, the test genuinely passed.

This was useful to catch. In production, a worker that hammers /claim immediately after /fail will always get 204 during the backoff window — which is correct behaviour, but looks like an empty queue if you're not expecting it.

Phase 2: Concurrent Publishers and Consumers

50 concurrent publishers. 50 concurrent consumers (plus 5 extra to ensure the queue drained completely).

Both pools used ThreadPoolExecutor with max_workers=2 against the live PythonAnywhere server.

[RESULTS] Time Elapsed: 82.48 seconds
[RESULTS] Published: 50/50
[RESULTS] Claimed & Fulfilled: 50/50
[RESULTS] Errors/Collisions: 0

Zero dropped jobs. Zero duplicate claims.

The 82 seconds is almost entirely network latency — PythonAnywhere is in Chicago, I'm in Bangalore, and each signed HTTP request takes around 0.8 seconds round trip. At max_workers=2, requests mostly run sequentially. The elapsed time is not a concurrency bottleneck, it's 100 requests × 0.8s.

I ran the same test with max_workers=20. The time dropped to under 20 seconds. Errors stayed at zero. BEGIN IMMEDIATE serializes writes correctly — competing claims block and wait rather than corrupting each other.

What Actually Surprised Me

SQLite WAL mode handles read-write concurrency better than I expected.

WAL (Write-Ahead Log) mode lets readers and writers operate concurrently without blocking each other. Readers never block writers. Writers only block other writers. For a job queue where claims are the critical write path, this is ideal.

The busy_timeout pragma is important.

Without it, SQLite throws SQLITE_BUSY immediately when a second writer tries to acquire a lock. I set PRAGMA busy_timeout = 30000 — SQLite will retry for up to 30 seconds before giving up. Combined with WAL mode, this handles brief contention cleanly and degrades gracefully to a 503 on real overload.

Exponential backoff is genuinely necessary.

Without jitter, failed jobs from multiple workers all requeue at the same future timestamp. They all become claimable at the same moment, causing a thundering herd — every worker races to claim simultaneously. The + random.uniform(0, 2) in the backoff formula spreads retries across a 2-second window and flattens the spike.

I found the bug in my own test before I found one in the server.

The backoff false-positive in Phase 1B was caught because I actually read the stress test output carefully. If I had just seen ✓ PASS and moved on, I would have shipped a test that reported correct behaviour when the backoff was working as intended — and I would have had no way to distinguish that from "job is dead."

Lesson: when writing a test for timing-dependent behaviour, always verify the reason for the result, not just the result.

What It's Not

SQLite is single-writer. That's a constraint you accept when you choose it.

This is not a replacement for Kafka, RabbitMQ, or Celery at scale. For hundreds of jobs per day with dozens of workers, SQLite is genuinely good enough and operationally simpler than anything else. For thousands of concurrent writers or horizontal scaling across multiple nodes, you need PostgreSQL at minimum.

The upgrade path is contained: get_db() is the only function that talks to SQLite directly. Swapping the backend is one function.

Where It Is Now

The server is live at dsecurity.pythonanywhere.com. There's a Python SDK on PyPI:

pip install intent-bus

from intent_bus import IntentClient

client = IntentClient(api_key="your_key")

# Publish from a cloud server
client.publish("notify", {"message": "deploy finished"})

# Worker on your phone (Termux)
client.listen("notify", handler=lambda p: print(p["message"]))

GitHub: github.com/dsecurity49/Intent-Bus

The next real test is getting external developers running workers in their own environments. If you're running small automations and want to try it, tester keys are available via GitHub Discussions.

The part I'm still most curious about: what breaks at max_workers=50 on a free-tier PythonAnywhere instance. I haven't pushed it there yet.