Most automation looks great on the happy path. The job runs, it does its thing, everyone is happy. The trouble starts the day it runs when you did not expect it to - a retry fires, a deploy restarts it, someone kicks it off manually while the scheduled one is already going. If running twice does something bad, you have built a quiet little time bomb.
So the bar I hold for any scheduled job is simple: I should be able to run it twice by accident and have nothing go wrong.
Idempotency is the whole game
The fix is making the job idempotent, which is a fancy word for a plain idea - running it again lands you in the same place, not a worse one. That usually means checking whether a piece of work was already done before doing it, keying actions to something stable so duplicates collapse instead of stacking, and never assuming this is the first time you have seen a given item.
A job you can safely run twice is a job you can trust to run unattended. One you cannot is one you will be babysitting forever.
Reconcile, do not just fire and forget
The other half is knowing whether the job actually did what it was supposed to. So I like a reconciliation step - something that compares what should be true against what is true and flags the gap, rather than trusting that because the job did not crash, it must have worked. Silent partial failures are the worst kind, because you find out about them from a confused user, not a log.
How I make a scheduled job trustworthy:
- Make it idempotent so a second run is harmless, not destructive.
- Key actions to stable identifiers so duplicates collapse instead of stacking.
- Add a reconciliation pass that checks the result, not just the exit code.
- Assume it will run when you do not expect it, because eventually it will.
The goal is automation you can stop thinking about. That only happens when running it twice is a non-event, and you have a way to confirm it actually worked.