Quota prober
The tick pass reads account quota from the quota caches and never probes an account itself. The agentihooks-quota-prober user service probes every account once a minute and writes those caches. When the prober stops, the quota notice step reports how old the newest reading is.
A Claude probe runs claude -p once per stale credential and a Codex probe sends a Codex request. Both take seconds. Inside the pass they held every step that waits on the quota capacity decision, which runs before spawn. With the probes in their own service, the pass spends no time on them.
Who probes and who reads
| Caller | Call | Claude | Codex |
|---|---|---|---|
| Tick pass | HerdrRuntime.quota_capacity(...) with its default refresh=False, which calls capacity.accounts(environ, now, refresh=False) | Reads the cached observations (balancer.cached_observations) and the master snapshot; runs no claude -p | Reads the last recorded quota through CodexAccountSource(refresh=False); runs only the local login status check, never a Codex quota request |
| Quota prober | quota_refresh.probe(environ, now), which calls capacity.probe(environ, now) | Probes every stale credential and writes the Claude cache | Probes stale Codex readings and writes the Codex cache |
Any other caller of capacity.accounts | refresh defaults to True | Probes stale credentials | Probes stale readings |
capacity.probe is _claude(environ, now, True) + _codex(environ, now, True): the same rows a refreshing capacity.accounts builds, without the global aggregate route and the reservation pass. Its job is the cache write. The tick pass reads that cache on its next run.
A caller that needs a fresh reading inside the pass passes refresh=True to quota_capacity. No tick step does.
The prober service
agentihooks swarm prober runs the loop. The swarm timer install writes it as ~/.config/systemd/user/agentihooks-quota-prober.service, with Restart=always and the environment from ~/.agentihooks/.env. The service already answered quota refresh requests from the ledger page. Those requests still work the same way.
Each round of the loop:
- Probes when the next probe is due (
quota_refresh.probe_due). The first round always probes. After a probe the next one is duequota_refresh.PROBE_S(60) seconds later, measured on the monotonic clock. - Waits up to
quota_refresh.WAIT_S(30) seconds for a wake on the quota refresh channel, published when the ledger page asks for a refresh. - Answers every pending ledger page quota request (
answer_all).
A probe can only start at the top of a round, so probes run about 60 seconds apart, longer when a probe or a page answer runs past that. Each probe prints one line to the service log:
quota probe read 7 accounts
The number counts the Claude and Codex rows the probe built.
Check the prober
systemctl --user status agentihooks-quota-prober.service
journalctl --user -u agentihooks-quota-prober.service --since "5 min ago" | grep "quota probe read"
agentihooks quota
A healthy service logs a quota probe read line about once a minute. The AGE column of agentihooks quota shows how old each account’s reading is.
Restart the service after changing an AH_CC_TOKEN_<slug> or AH_CX_TOKEN_<slug> value, because it keeps the environment it started with. A re-login of the default Codex account needs no restart: the login check reads Codex’s own files on every probe.
systemctl --user restart agentihooks-quota-prober.service
Stale readings in the quota notice step
The quota notice step (quota_notice.refresh, in the placement notices group) adds one action when the newest observed_at across the accounts of the stored capacity decision is older than quota_notice.STALE_S (180) seconds:
quota readings are 999 seconds old; the quota prober has not refreshed them
The age is whole seconds from the newest reading to the pass time. One fresh account is enough to keep the line quiet. Rows with no observed_at are ignored, and a decision with no readings at all adds nothing. The step still sends the account level notices and the quota handoff warning from the readings it has. It does not probe.
The tick report’s step actions and the pass log carry the line. Three minutes covers about two missed probes.
Limits
- The pass places and warns from readings up to one probe gap old, longer when the prober is down. A Claude account that ran out inside that gap stays open until the next probe.
- The prober runs on the machine that installed the swarm timers. A
distributedworkstation retires the prober unit with the tick (Deployment modes). quota_capacitydefaults to cached readings, so code that called it expecting a probe now gets the cache. Passrefresh=Truefor a probe.
Tests
| Test | Proves |
|---|---|
tests/swarm/test_capacity.py::test_a_pass_read_runs_only_the_local_codex_login_status_check | accounts(refresh=False) runs only the Codex login check; accounts() runs claude -p, the login check and codex exec |
tests/swarm/test_capacity.py::test_the_quota_notice_step_runs_no_claude_or_codex_quota_probe | A whole quota notice step stores a capacity decision while running only the Codex login check |
tests/swarm/test_capacity.py::test_the_pass_reader_defaults_to_cached_readings | HerdrRuntime.quota_capacity defaults refresh to False |
tests/swarm/test_capacity.py::test_the_prober_writes_the_claude_cache_the_pass_reads | A probe writes the Claude cache and a later pass read reports that probe’s observed_at |
tests/swarm/test_capacity.py::test_a_probe_refreshes_claude_and_codex_at_the_given_time | capacity.probe calls the Claude and Codex readers with refresh=True at the given time |
tests/swarm/test_quota_refresh.py::test_the_prober_probes_once_a_minute_between_wakes | probe_due probes at once, then sixty seconds after each probe |
tests/swarm/test_quota_refresh.py::test_main_runs_the_quota_prober_without_a_slug_on_the_hive_command_runner | swarm prober hands the loop a probe over this process’s environment and wall clock |
tests/swarm/test_quota_refresh.py::test_the_quota_notice_names_a_reading_over_three_minutes_old | The stale line appears at 181 seconds, not at 180, and never without readings |
tests/swarm/test_quota_refresh.py::test_the_quota_notice_step_adds_the_reading_age | The quota notice step appends the stale line after its other actions |
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
quota readings are N seconds old; the quota prober has not refreshed them in the pass actions | The prober service is stopped, failing or not installed on this host | systemctl --user status agentihooks-quota-prober.service; read its journal; reinstall the swarm timers when the unit is missing |
The prober logs quota probe read but the stale line stays | Every probe failed to record a reading, for example every token expired | Run agentihooks balance and fix the accounts it reports; see Refreshing tokens |
| An account shows its old state after a token change | The prober kept its old environment | Restart the prober service |
| Spawns land on an account that ran out a minute ago | The pass read the last probe, taken before the account ran out | Expected inside one probe gap; the next probe closes the account |
| Ledger page quota refresh never answers | The prober service is down; it answers page requests too | Restart the service and check its journal |