When the Tests Fail, Claude Investigates

In my last post I walked through InboxProcessor, the job that picks up task notes from my Obsidian vault every 15 minutes and hands them to Claude Code. One of the examples was a failing regression test: the tests emailed me every 15 minutes about Watchtower, I wrote a task from a chat asking Claude to investigate, and it found and fixed the cause while I got on with something else.

The obvious next question was why I was the one writing that task. This week I took myself out of that step. When a test on the Pi starts failing, the watchdog now writes the investigation task itself, and Claude picks it up.

How the watchdog got here

The watchdog has grown in steps, each one prompted by something going wrong:

• July: run the tests unattended. Paperless crash-looped for about 12 hours after an automatic image update, and nothing told me. The regression suite would have caught it, but nobody was running it. So a systemd timer now runs the whole suite every 15 minutes and emails me if anything fails.
• September: fix the obvious thing itself. After the NAS was powered down, ten containers sat stopped for about two days because they failed when they were first created, and Docker’s restart policy can’t recover a container that never started. The watchdog spotted it every 15 minutes and emailed me every 15 minutes, but did nothing about it. Now it restarts any container that should be running but isn’t before it runs the tests, and sends a softer warning email to say it did.
• October: investigate everything else. That’s this post.

The suite has grown alongside it, from 38 test scripts at the start of the month to 46 now.

From an email to a task

Until now a failure produced an email, and the rest was up to me: read it, work out whether it mattered, then either fix it or ask Claude to. In practice, that last step was usually me writing a task note asking Claude to look into it, which is exactly the sort of thing that can be automated.

So after every run, the watchdog now passes the test output to a small Python script. When a check has started failing, the script writes a task note into the vault’s Inbox folder addressed to Claude Code, and InboxProcessor picks it up on its next run, within 15 minutes. The task note contains:

• The failing checks, exactly as the test suite reported them, plus when they first failed.
• Where to look. Re-run just the affected tests, then look at the watchdog’s log, the test script itself, the service’s logs, and the service’s notes in the vault.
• A reminder that the test might be the problem. Several failures over the summer were false positives, such as the Watchtower race in the last post, so Claude is told to consider that as well as whether the service is actually broken.
• A decision to make: is this a simple fix, or does it need me?

Simple fix, or “Action needed”

The most important part of the task is the definition of a simple fix. It has to be small, low-risk and easily reversible, and within the conventions everything else already follows. It must not need credentials, a login or re-authentication. It must not delete data, cost money or touch anything public-facing. It must not involve a design choice I’d want to make myself. And it must not interrupt any of the scheduled jobs listed in the vault at the time it would be done. The examples given are a test false positive, a crashed container that just needs starting again, or a typo in a config file.

Everything else needs me, and the task spells out that this includes expired tokens or sessions, anything Claude isn’t confident about, and anything with more than one plausible fix.

The two outcomes look like this:

• Simple: fix it properly. The same way as any other change: commit and push to the project’s repository, keep the README current, update the test if needed, run the full suite rather than just the failing test, write a changelog note in the vault, email me with a subject starting “Fixed:”, and archive the task.
• Not simple: write it up and stop. Add a Findings section (the root cause and the evidence) and a Blocked section (the proposed fix or fixes, and exactly what I need to decide or do), mark the task blocked, leave it in the Inbox, and email me with a subject starting “Action needed:”.

Either way, I should get exactly one email about it, and the root cause should be written into the task note.

What it may never do

A few things are ruled out whatever the investigation finds:

• Never make a test pass by switching it off or weakening it to hide a real fault. This is the one I care about most. An AI asked to “make the failing test pass” has an easy way out, and taking it would quietly undo the whole point of having tests.
• Never reboot the Pi, and never take a Docker service down. Stopping a service on purpose is how I tell the auto-heal to leave it alone, so Claude doing it would confuse both of them.
• Leave the things I’ve deliberately parked alone. A couple of services are broken or stopped on purpose while I decide what to do with them. If one of those turns out to be the cause, the task blocks and says so.

I should be honest that these are instructions in the task, not hard limits. InboxProcessor runs Claude with full autonomy, as I explained last time, so the boundary between a simple fix and something that needs me is enforced by the prompt rather than by permissions. It’s the same trade-off as every other InboxProcessor task, and the same safety nets apply: the time limit, the lock, and alerts on anything other than a clean finish.

What counts as “new”

An investigation every 15 minutes for the same broken thing would be worse than useless, so most of the script is about deciding when a failure is genuinely new:

• It must fail twice in a row. A check only counts once it has failed on two consecutive runs. That filters out blips, such as the first run after a reboot, when containers are still starting. That case prompted this whole change: after a reboot on Thursday I asked for an Inbox task whenever an error was seen for the first time.
• Numbers don’t make it new. Each failing check is identified by the component and the check name, with any digits ignored, so a check reporting a changing count doesn’t look like a fresh failure each time.
• One open task per failure. While a task for that check is still in the Inbox, whether pending, in progress or blocked, nothing new is raised. Once it’s closed, the check only gets another task if it has passed in between and then failed again.
• One outage, one task. Everything that newly fails in the same run shares a single task, so if the NAS drops and thirty checks fail at once, Claude gets one investigation, not thirty.

Fewer emails

Once that logic existed, it was obvious the alert emails should follow it too. Until this week, a failing check sent a critical email on every run, every 15 minutes, for as long as it stayed broken. Now the critical email goes once, with the task, when a failure is confirmed. It lists only the new failures and names the task. Then it stays quiet while the check keeps failing, and the next email I get is Claude’s “Fixed” or “Action needed”.

There are a few deliberate exceptions. If the test suite itself crashes, there’s nothing to judge, so it always emails. If the vault can’t be reached, the email still goes out, saying the task couldn’t be written, and the task is written on a later run. The “containers were restarted” warning from auto-heal hasn’t changed.

That’s a trade-off too. A check that fails on and off but never twice in a row now never emails at all. It still shows up in the watchdog’s log and on the regression tests tab of my dashboard, but I’ve accepted that in exchange for not being nagged.

Testing the thing that watches the tests

Naturally it has its own regression test. It feeds made-up test output into the script, with a temporary state file and a temporary inbox (never the real vault), and checks the rules: a one-run blip is ignored, a confirmed failure raises exactly one task, repeats are suppressed, a new task is only raised after the check has cleared, the alert goes out once, and an unreachable vault is handled. It started with 10 checks and grew to 15 when the email throttling went in. A task generated by the script was also confirmed to be picked up by InboxProcessor’s real task-finding code, and the full suite passed after each change.

What I can’t tell you yet is how it does in real life. Nothing has failed since it went in, so no real investigation task has been raised. The end-to-end check is still an unticked box in the changelog, waiting for the next genuine failure. I’m looking forward to it more than I’d like to admit.

What I’ve learned

• Look for the step where you’re just the messenger. I was reading an email and then writing a task that said, in effect, “look into this”. Removing that step needed no new service, just one script joining two things that already existed.
• Write down what “simple” means. “Fix it if it’s easy” is too vague. A specific list (no credentials, no deletion, no design choices, nothing that interrupts a scheduled job) gives Claude something it can actually check against.
• Deciding when to stay quiet is most of the work. The investigation prompt was quick to write. The rules about what counts as new, and when not to email, took most of the effort and most of the tests.

A note on how this post was written: it was drafted by Claude Code, which went through the regression test repository (the README, the new script and its git history) and the changelog notes in my vault, so that everything here could be checked against what was actually built. It wrote the draft into WordPress through the WordPress API, and I then reviewed and edited it before publishing.