Tag: Automation

Last weekend I wrote about the watchdog that now writes its own investigation tasks when a regression test on the Pi starts failing, so that Claude Code picks them up without me being the messenger. I finished that post by saying nothing had failed since it went in, so no real investigation task had been raised yet.

That turned out not to be true. The first real one had been raised just before midnight the night before the post went out. I just hadn’t noticed it yet. This post is about that task: what broke, what Claude found, why it stopped and asked me rather than fixing it, and what the fix ended up being.

The feature that broke

The failing check belonged to one of the sillier things I’ve built. The Pi Zero in the house with a Unicorn HAT HD LED matrix has a small web control page, and in September I added an “Aircraft overhead” button to it. Press it, and the panel scrolls the closest plane currently in the sky near home, something like:

• EAI55SK · AT76 · EDI→DUB · 8200ft · 8.5mi ESE

That’s the callsign, the aircraft type, the route, the altitude, and the distance and compass bearing from home. It was built in a few steps:

• The aircraft data comes from free community ADS-B feeds, run by volunteers with receivers. adsb.lol was the main feed, with adsb.fi as the fallback if it errored. (My first choice, airplanes.live, now refuses requests unless you register.)
• The radius started at 5 miles, which mostly produced “No aircraft within 5 mi”, so a day later it went up to 10.
• The route isn’t part of the ADS-B signal at all, only the callsign is. So the route is looked up separately against adsbdb.com, a free community database of airline schedules, and simply left out if there’s no match.

It lives in RailData, the Flask app on the Pi that has become home to most of my small odds and ends, and like everything else it has regression tests. One of them simply asks the aircraft endpoint for an answer and expects to get one within the suite’s 20-second limit.

Saturday night, 23:20

Here’s how it played out, using the timestamps in the task note and the logs:

• 23:20. The aircraft check failed for the first time. The test got an empty response instead of an answer.
• 23:34. It failed again on the next run. Two failures in a row is the rule for a failure counting as real, so the watchdog wrote an investigation task into the vault’s Inbox folder.
• Just before midnight. InboxProcessor picked the task up, Claude Code investigated, wrote up its findings, and marked the task as blocked, waiting for me.
• Sunday evening. I read it, wrote down my decision, and the fix went in shortly after 20:20.

Nothing was emailing me every 15 minutes in the meantime, which was exactly the point of last week’s changes.

What Claude found

The investigation was good, and I mean that in the sense that it went past the symptom. The service itself was healthy: running, no errors, every other check passing. The problem was upstream.

• The main feed had become very slow. Timing adsb.lol directly from the Pi gave 11 seconds, then a time-out at 20 seconds, then a time-out at 40 seconds with nothing back at all, then 23 seconds, then 4. Over the same period adsb.fi answered in under a fifth of a second.
• The fallback worked, but too late. Each request waited out adsb.lol’s 8-second timeout before trying adsb.fi, and then the route lookup ran on top of that.
• The timeouts didn’t mean what they looked like. This was the subtle bit. Python’s urlopen timeout applies to each socket operation, not to the whole request, so a server that sends data very slowly can keep a request going well past the number in the code. Add up the feed, the fallback and the route lookup, and the worst case was over 20 seconds.
• Only the first request was slow. The logs showed the first aircraft request in each failing run taking around 23–25 seconds, and the ones after it finishing almost together, because by then the cache had been filled.

Re-running the RailData tests on their own, everything passed. Claude pointed out that this was only because adsb.lol happened to fail quickly that time, and that it would probably keep flapping for as long as the feed stayed slow.

Was the test wrong?

The task tells Claude to consider whether the test is the problem, because a few failures over the summer were false positives. Its answer here was “not really”. A 20-second limit isn’t unreasonable: anyone pressing that button on the control page would also have been waiting 20 seconds or more. The test was reporting a real problem, it just happened to be one caused by somebody else’s server.

That mattered, because the easiest way to make the failure go away was to give that one check a longer timeout. Claude listed that as an option, and recommended against doing it on its own because it would hide real slowness. That’s the rule I care about most, never weaken a test to hide a real fault, being followed when it would have been the quick way out.

Why it stopped and asked

The watchdog’s task defines a simple fix quite narrowly, and one of the things that rules a fix out is “more than one plausible fix”. This had four:

• Swap the feeds, so the fast one goes first. A one-line change, but it moves the load and the dependency onto adsb.fi.
• Cap the total time, with a shorter timeout for the main feed and an overall deadline, or query both feeds at once and take the first good answer.
• Skip the route lookup when time is short, so it can’t add its few seconds on top of a slow fallback.
• Only relax the test, which it didn’t recommend.

It suggested the first two together, changed nothing, set the task to blocked and left it in the Inbox with a Findings section and a Blocked section. That is precisely the behaviour I’d asked for, and it’s also what I’d have wanted from a person: here’s the evidence, here are the options, here’s what I’d do, your call.

My part was one line. I added a Decision heading to the task note saying “Implement option1 and 2”, and put it back in the queue for InboxProcessor to pick up.

The fix

• adsb.fi is now the main feed, with a 4-second timeout, and adsb.lol is the fallback with 8 seconds.
• The whole lookup has a hard deadline of 10 seconds by default, covering both feeds and the route lookup together. Because urlopen‘s own timeout can’t be trusted to stop a slow trickle of data, each request now runs in a background thread and the code only waits for it until the deadline. If it isn’t finished by then, it’s abandoned.
• The route lookup is skipped if there’s less than a second left, and that skip isn’t cached, so the route is tried again next time.

It went in the same way as any other change: a commit pushed to the RailData repository, the README updated, the service restarted at a time that didn’t clash with any of the scheduled jobs, and a changelog note in the vault with instructions for reversing it.

The results were what I’d hoped for. A fresh request came back in about 0.35 seconds from adsb.fi. With both feeds deliberately made unreachable, the request gave up after exactly 10.0 seconds and still answered with a sensible error rather than hanging. The full regression suite passed: 43 components passed, none failed, and 3 were skipped. Then I got one email with a subject starting “Fixed:”, and the task was archived.

What I’ve learned

• Blocking is a good outcome. I’d half expected the first real case to be a test false positive or a container that needed a restart, something Claude could just fix. Instead it was a genuine judgement call, and the most useful thing it did was stop and lay out the options rather than pick one for me.
• The investigation is the valuable part. I’d never have timed the feeds at midnight. By the time I looked on Sunday the root cause, the evidence and a recommendation were all written down in the task, and my contribution took seconds.
• Free services can change under you. Nothing in my code changed to cause this. A volunteer-run feed got slow, which is entirely fair for something I don’t pay for, and the right response was to stop depending on it being fast.
• A timeout isn’t a deadline. I’d have assumed an 8-second timeout meant a request couldn’t take longer than 8 seconds. It can. If something has to answer within a time limit, the limit needs to be enforced around the whole thing.
• Check before you say “nothing yet”. Last week’s post said no real task had been raised. One had, about thirteen hours before it was published. I’ll take the correction.

A note on how this post was written: it was drafted by Claude Code, which went through the investigation task note itself, the RailData repository’s code and git history, the InboxProcessor history and the changelog notes in my vault, so that the timeline and numbers here could be checked against what actually happened. Claude also chose the two photos from my Flickr account. It wrote the draft into WordPress as a draft through the WordPress API, and I then reviewed and edited it before publishing.

In my last post I walked through InboxProcessor, the job that picks up task notes from my Obsidian vault every 15 minutes and hands them to Claude Code. One of the examples was a failing regression test: the tests emailed me every 15 minutes about Watchtower, I wrote a task from a chat asking Claude to investigate, and it found and fixed the cause while I got on with something else.

The obvious next question was why I was the one writing that task. This week I took myself out of that step. When a test on the Pi starts failing, the watchdog now writes the investigation task itself, and Claude picks it up.

How the watchdog got here

The watchdog has grown in steps, each one prompted by something going wrong:

• July: run the tests unattended. Paperless crash-looped for about 12 hours after an automatic image update, and nothing told me. The regression suite would have caught it, but nobody was running it. So a systemd timer now runs the whole suite every 15 minutes and emails me if anything fails.
• September: fix the obvious thing itself. After the NAS was powered down, ten containers sat stopped for about two days because they failed when they were first created, and Docker’s restart policy can’t recover a container that never started. The watchdog spotted it every 15 minutes and emailed me every 15 minutes, but did nothing about it. Now it restarts any container that should be running but isn’t before it runs the tests, and sends a softer warning email to say it did.
• October: investigate everything else. That’s this post.

The suite has grown alongside it, from 38 test scripts at the start of the month to 46 now.

From an email to a task

Until now a failure produced an email, and the rest was up to me: read it, work out whether it mattered, then either fix it or ask Claude to. In practice, that last step was usually me writing a task note asking Claude to look into it, which is exactly the sort of thing that can be automated.

So after every run, the watchdog now passes the test output to a small Python script. When a check has started failing, the script writes a task note into the vault’s Inbox folder addressed to Claude Code, and InboxProcessor picks it up on its next run, within 15 minutes. The task note contains:

• The failing checks, exactly as the test suite reported them, plus when they first failed.
• Where to look. Re-run just the affected tests, then look at the watchdog’s log, the test script itself, the service’s logs, and the service’s notes in the vault.
• A reminder that the test might be the problem. Several failures over the summer were false positives, such as the Watchtower race in the last post, so Claude is told to consider that as well as whether the service is actually broken.
• A decision to make: is this a simple fix, or does it need me?

Simple fix, or “Action needed”

The most important part of the task is the definition of a simple fix. It has to be small, low-risk and easily reversible, and within the conventions everything else already follows. It must not need credentials, a login or re-authentication. It must not delete data, cost money or touch anything public-facing. It must not involve a design choice I’d want to make myself. And it must not interrupt any of the scheduled jobs listed in the vault at the time it would be done. The examples given are a test false positive, a crashed container that just needs starting again, or a typo in a config file.

Everything else needs me, and the task spells out that this includes expired tokens or sessions, anything Claude isn’t confident about, and anything with more than one plausible fix.

The two outcomes look like this:

• Simple: fix it properly. The same way as any other change: commit and push to the project’s repository, keep the README current, update the test if needed, run the full suite rather than just the failing test, write a changelog note in the vault, email me with a subject starting “Fixed:”, and archive the task.
• Not simple: write it up and stop. Add a Findings section (the root cause and the evidence) and a Blocked section (the proposed fix or fixes, and exactly what I need to decide or do), mark the task blocked, leave it in the Inbox, and email me with a subject starting “Action needed:”.

Either way, I should get exactly one email about it, and the root cause should be written into the task note.

What it may never do

A few things are ruled out whatever the investigation finds:

• Never make a test pass by switching it off or weakening it to hide a real fault. This is the one I care about most. An AI asked to “make the failing test pass” has an easy way out, and taking it would quietly undo the whole point of having tests.
• Never reboot the Pi, and never take a Docker service down. Stopping a service on purpose is how I tell the auto-heal to leave it alone, so Claude doing it would confuse both of them.
• Leave the things I’ve deliberately parked alone. A couple of services are broken or stopped on purpose while I decide what to do with them. If one of those turns out to be the cause, the task blocks and says so.

I should be honest that these are instructions in the task, not hard limits. InboxProcessor runs Claude with full autonomy, as I explained last time, so the boundary between a simple fix and something that needs me is enforced by the prompt rather than by permissions. It’s the same trade-off as every other InboxProcessor task, and the same safety nets apply: the time limit, the lock, and alerts on anything other than a clean finish.

What counts as “new”

An investigation every 15 minutes for the same broken thing would be worse than useless, so most of the script is about deciding when a failure is genuinely new:

• It must fail twice in a row. A check only counts once it has failed on two consecutive runs. That filters out blips, such as the first run after a reboot, when containers are still starting. That case prompted this whole change: after a reboot on Thursday I asked for an Inbox task whenever an error was seen for the first time.
• Numbers don’t make it new. Each failing check is identified by the component and the check name, with any digits ignored, so a check reporting a changing count doesn’t look like a fresh failure each time.
• One open task per failure. While a task for that check is still in the Inbox, whether pending, in progress or blocked, nothing new is raised. Once it’s closed, the check only gets another task if it has passed in between and then failed again.
• One outage, one task. Everything that newly fails in the same run shares a single task, so if the NAS drops and thirty checks fail at once, Claude gets one investigation, not thirty.

Fewer emails

Once that logic existed, it was obvious the alert emails should follow it too. Until this week, a failing check sent a critical email on every run, every 15 minutes, for as long as it stayed broken. Now the critical email goes once, with the task, when a failure is confirmed. It lists only the new failures and names the task. Then it stays quiet while the check keeps failing, and the next email I get is Claude’s “Fixed” or “Action needed”.

There are a few deliberate exceptions. If the test suite itself crashes, there’s nothing to judge, so it always emails. If the vault can’t be reached, the email still goes out, saying the task couldn’t be written, and the task is written on a later run. The “containers were restarted” warning from auto-heal hasn’t changed.

That’s a trade-off too. A check that fails on and off but never twice in a row now never emails at all. It still shows up in the watchdog’s log and on the regression tests tab of my dashboard, but I’ve accepted that in exchange for not being nagged.

Testing the thing that watches the tests

Naturally it has its own regression test. It feeds made-up test output into the script, with a temporary state file and a temporary inbox (never the real vault), and checks the rules: a one-run blip is ignored, a confirmed failure raises exactly one task, repeats are suppressed, a new task is only raised after the check has cleared, the alert goes out once, and an unreachable vault is handled. It started with 10 checks and grew to 15 when the email throttling went in. A task generated by the script was also confirmed to be picked up by InboxProcessor’s real task-finding code, and the full suite passed after each change.

What I can’t tell you yet is how it does in real life. Nothing has failed since it went in, so no real investigation task has been raised. The end-to-end check is still an unticked box in the changelog, waiting for the next genuine failure. I’m looking forward to it more than I’d like to admit.

What I’ve learned

• Look for the step where you’re just the messenger. I was reading an email and then writing a task that said, in effect, “look into this”. Removing that step needed no new service, just one script joining two things that already existed.
• Write down what “simple” means. “Fix it if it’s easy” is too vague. A specific list (no credentials, no deletion, no design choices, nothing that interrupts a scheduled job) gives Claude something it can actually check against.
• Deciding when to stay quiet is most of the work. The investigation prompt was quick to write. The rules about what counts as new, and when not to email, took most of the effort and most of the tests.

A note on how this post was written: it was drafted by Claude Code, which went through the regression test repository (the README, the new script and its git history) and the changelog notes in my vault, so that everything here could be checked against what was actually built. It wrote the draft into WordPress through the WordPress API, and I then reviewed and edited it before publishing.