Many solo operations need a daily import, a report refresh, or a queue of records to review. These jobs need clear inputs, durable status, and diagnosable failures more than they need an autonomous planner. A deliberately small Python command-line program can expose the operational seams: how a job is submitted, what persists, what gets retried, and which action still requires a person.
Start with a per-project virtual environment for dependency isolation. Python’s venv documentation describes environments with their own installed package sets; isolation is useful even when the first prototype uses only the standard library.[14] Build an argparse command with enqueue, list, and later run-one subcommands rather than hard-coding a script that does everything when imported. argparse supports options and subcommands and parses arguments into a namespace.[12]
A minimal durable intake, not a complete queue
This illustrative intake stores one JSON payload per caller-chosen ID. It does not process jobs, lock workers, authenticate callers, or prevent a second external side effect; those would be separate design steps.
import argparse
from contextlib import closing
import json
import sqlite3
import sys
def main():
parser = argparse.ArgumentParser()
parser.add_argument("db")
sub = parser.add_subparsers(dest="action", required=True)
add = sub.add_parser("enqueue")
add.add_argument("job_id")
add.add_argument("payload_json")
sub.add_parser("list")
args = parser.parse_args()
with closing(sqlite3.connect(args.db)) as db:
with db:
db.execute("CREATE TABLE IF NOT EXISTS jobs ("
"id TEXT PRIMARY KEY, payload TEXT NOT NULL, "
"status TEXT NOT NULL DEFAULT 'pending')")
if args.action == "enqueue":
payload = json.dumps(json.loads(args.payload_json),
sort_keys=True, separators=(",", ":"))
try:
db.execute("INSERT INTO jobs(id, payload) VALUES (?, ?)",
(args.job_id, payload))
except sqlite3.IntegrityError:
row = db.execute("SELECT payload FROM jobs WHERE id = ?",
(args.job_id,)).fetchone()
if row is None: # An unrelated constraint failed.
raise
if row[0] != payload:
print("conflict: job ID already has a different payload",
file=sys.stderr)
return 2
print("unchanged: job ID and payload already exist")
else:
print("enqueued: new job ID")
else:
for row in db.execute("SELECT id, status FROM jobs ORDER BY id"):
print(*row)
return 0
if __name__ == "__main__":
sys.exit(main())
The primary key prevents a second row for the same ID. This example reports a new insertion, an unchanged resubmission (exit 0), or a different payload for an existing ID (exit 2); it neither overwrites the old payload nor prints either payload in the conflict message. JSON object key order is normalized before comparison. The example takes payload_json on the command line only for brevity: sensitive values can leak through shell history or process listings, so use a protected input channel in a real tool. Keep the database path in a reviewed, application-owned directory with appropriate permissions and a documented backup policy; SQLite is a local disk database, not an automatically distributed or encrypted service. Python’s sqlite3 guide recommends parameter placeholders rather than formatting untrusted values into SQL.[11]
A real runner should claim one pending job, record an attempt, perform a bounded operation, and record success or a typed failure. Specify the retry policy per action. A read-only fetch can often be retried after a timeout; a payment, message, or content publish may have completed even if the client missed the response. Reconcile with the destination or use its documented idempotency support before retrying. An attempts counter and status row alone cannot guarantee exactly-once external effects.
Bound the network and preserve useful evidence
For a job that fetches an approved URL, use an explicit timeout and destination allowlist, validate response size and format, and close the response. urllib.request.urlopen accepts a timeout for blocking operations such as connection attempts; that does not establish an overall job deadline, nor does it make redirects and downloaded content trustworthy.[16] Pass through a reviewed proxy and certificate policy when the deployment requires one. Do not let an AI model or an untrusted CSV row choose an arbitrary internal endpoint: the job’s input schema should refer to a known resource or restricted host, not a universal URL fetcher.
Use Python logging with a stable job ID, phase, outcome, and short error category.[13] Avoid logging API tokens or raw customer records. Record enough to answer whether the failure happened before the request, after a response, or during persistence. Put credentials in an approved secret store or runtime environment, not in JSON payloads or command-line examples that may enter shell history. Control log retention just as deliberately as database retention.
Test the seams before adding concurrency
Replace the HTTP boundary in tests, then exercise malformed JSON, duplicate IDs, a blocked destination, a timeout, and a repeated run after an ambiguous response. Python’s unittest.mock can replace parts of the system under test and assert how they were used.[15] Keep one explicit function for each external action so these tests do not need a real provider or paid calls. Test database state with a temporary file and examine both the success and failure paths.
Only add parallel workers when a measured backlog warrants them. At that point define a claim protocol, SQLite contention behavior, an attempt limit, and shutdown semantics; a loop that selects pending rows is not a safe multi-worker queue by default. If the job begins to branch on uncertain text, an agent can propose a classification or draft, but the CLI’s validation, storage, and external-action boundary should remain independently testable. The useful advance is not more Python packages: it is making each failure observable and each side effect deliberate.
Sources
- [11] https://docs.python.org/3/library/sqlite3.html
- [12] https://docs.python.org/3/library/argparse.html
- [13] https://docs.python.org/3/library/logging.html
- [14] https://docs.python.org/3/library/venv.html
- [15] https://docs.python.org/3/library/unittest.mock.html
- [16] https://docs.python.org/3/library/urllib.request.html




