Sydnee nightly — persistence audit — 2026-06-11 P0 findings: 0 P1 findings: 1 Risks: 2 - Area: Persistence (state survival across container restarts, DB/memory sync) - Branch: `dev` (d1db2c1 — peak-monitor log; no strategy code changed overnight) - Files scanned: `bot.py` (~18 497 lines), `core/database.py`, `core/persistence.py`, `core/risk.py`, `docs/strategy_decisions.md`, `git log --oneline -30` - **Bugs found (P0 / P1): 0 / 1 (1 new)** - **Risks noted: 2 (2 new)** Cross-check: `git log --oneline -30` shows only peak-monitor hourly logs since 2026-06-10 audit. The 7 exit-flow P1s carried from 2026-06-09/10 are confirmed unfixed; they are not persistence-domain so they appear under "Carried (other domain)" below. --- Full report (dev branch): https://github.com/kanex1/sydnee.signals/blob/dev/docs/audit_2026-06-11_persistence.md Reply FROM [email protected] to [email protected] to request fixes, e.g.: "code_task on sydnee-signals-dev: apply fix for the P0 about RVOL threshold in bot.py" Sydnee Agent will propose + you APPROVE (or plain 'approve') + auto-push to dev. --- Full audit below (first 12 KB) --- # Nightly Audit 2026-06-11 — PERSISTENCE ## Summary - Area: Persistence (state survival across container restarts, DB/memory sync) - Branch: `dev` (d1db2c1 — peak-monitor log; no strategy code changed overnight) - Files scanned: `bot.py` (~18 497 lines), `core/database.py`, `core/persistence.py`, `core/risk.py`, `docs/strategy_decisions.md`, `git log --oneline -30` - **Bugs found (P0 / P1): 0 / 1 (1 new)** - **Risks noted: 2 (2 new)** Cross-check: `git log --oneline -30` shows only peak-monitor hourly logs since 2026-06-10 audit. The 7 exit-flow P1s carried from 2026-06-09/10 are confirmed unfixed; they are not persistence-domain so they appear under "Carried (other domain)" below. --- ## Findings ### BUG [P1] (new): `risk.state.trade_count` and `risk.state.realized_pnl` not rebuilt from DB on mid-day restart — both daily risk gates reset to zero **File:** `core/risk.py:30–36` (`DailyState`); `bot.py:17782–17784` (startup); `core/risk.py:113–120` (`can_trade`) **Evidence:** `DailyState` is purely in-memory: ```python # core/risk.py:30-36 @dataclass class DailyState: date: date = field(default_factory=date.today) realized_pnl: float = 0.0 # ← initialized to 0 trade_count: int = 0 # ← initialized to 0 open_positions: int = 0 ``` On startup, only `open_positions` is seeded from DB: ```python # bot.py:17782-17784 open_count = len(self.open_trades()) if open_count > 0: self.risk.set_open_positions(open_count) # ← only open_positions restored ``` Neither `trade_count` nor `realized_pnl` is rebuilt. `can_trade()` checks both: ```python # core/risk.py:113-120 if self.state.realized_pnl <= -self.cfg.daily_loss_limit: return False, ... # ← bypassed (pnl=0 after restart) if self.state.trade_count >= self.cfg.max_daily_trades: return False, ... # ← bypassed (count=0 after restart) ``` **Impact:** 1. **Max-daily-trades bypass**: if 5 of 6 permitted trades were already fired today before a container restart, `trade_count=0` after restart allows 6 more — doubling exposure on a session that already committed full daily risk budget. 2. **Daily loss limit bypass**: if $800 of $1 500 daily limit was already lost before restart, `realized_pnl=0` after restart allows trading through a fresh $1 500 window, meaning the real intraday drawdown cap is silently doubled. Especially dangerous near RTH open when multiple signals queue quickly. Both risk gates are specifically designed as hard circuit-breakers; resetting them on every restart undermines that design. **Fix:** Add a rebuild step after `_reconcile_ibkr_positions()` in the startup sequence (near `bot.py:17784`): ```python # After risk.set_open_positions(open_count): try: _today_str = datetime.now(ET).strftime("%Y-%m-%d") _today_trades = self.db.get_trades(date_from=_today_str, limit=200) _trade_count = sum(1 for r in _today_trades if r.get("entry_time", "").startswith(_today_str)) _realized_pnl = sum(float(r["pnl"]) for r in _today_trades if r.get("pnl") is not None and r.get("exit_time", "").startswith(_today_str)) self.risk.state.trade_count = _trade_count self.risk.state.realized_pnl = _realized_pnl logger.info("Restored daily risk state: trade_count=%d realized_pnl=%.2f", _trade_count, _realized_pnl) except Exception: logger.warning("Failed to rebuild daily risk state from DB") ``` Note: trim-child rows have `trade_id` ending in `_trim*`; if the existing `insert_trade` call for trim children is considered a separate entry, filter by `not trade_id.endswith(("_trim", "_trim2", "_trim3", "_eod_trim"))` to avoid double-counting. --- ### RISK: `_update_trade_field` DB write is non-atomic — a silent DB failure permanently diverges in-memory and restart-recovered state **File:** `bot.py:1242–1254` **Evidence:** ```python def _update_trade_field(self, trade_id: str, **kwargs) -> None: with self._trades_lock: # ← in-memory update: inside lock, always succeeds for t in self._trades: if t.trade_id == trade_id: for k, v in kwargs.items(): if hasattr(t, k): setattr(t, k, v) break try: self.db.update_trade(trade_id, **kwargs) # ← DB update: outside lock, can fail except Exception: logger.warning("DB update_trade failed for %s", trade_id) # ← silently suppressed ``` If `self.db.update_trade()` raises (Postgres connection drop, pool exhaustion, serialization error), the in-memory flag is already set but the DB row is not updated. On the next container restart `_load_trades_from_db` reads the stale DB row and restores the old pre-update value, re-arming paths that had already fired: - `trimmed_1=False` in DB → TRIM1 re-fires on first bar after restart - `force_trim_pending=True` in DB (never cleared) → force-trim re-fires - `obv_be_applied=False` in DB → Rule F stop-advance re-runs - `extreme_since_trim=None` → trailing stop reverts to close-price baseline (loosens short trails, tightens long trails — documented separately) Frequency: rare under normal conditions, but transient Postgres restarts (out-of-memory OOM-kill on the container host, pg_wal full, etc.) are not uncommon in containerized paper trading. A single missed write during a trim creates ghost trim re-fires after any subsequent restart. **Mitigation (no code change required, operational):** Ensure Postgres is well-resourced and monitor `DB update_trade failed` log lines — a cluster of these messages indicates a connection-pool failure that may desync trade state. **Fix (code):** Add a post-startup "state reconcile" that re-reads all open trade rows from DB and compares the boolean flag set (trimmed_1/2/3, force_trim_pending, etc.) against in-memory values, logging any divergence. This reconcile already runs for positions via `_reconcile_ibkr_positions`; extending it to cover trade flags would surface any desync without requiring atomic DB writes. --- ### RISK: `_tv_scale_state` is purely in-memory — container restart mid-scale leaves position undersized (dev opt-in feature, no prod impact) **File:** `bot.py:4200–4229` (`_tv_scale_step`); `bot.py:17623` (init) **Evidence:** ```python # bot.py:17623 self._deferred_trades: list[tuple] = [] self._deferred_closes: list[tuple] = [] # _tv_scale_state is not initialized here — created on first webhook # bot.py:4200-4201 if not hasattr(self, "_tv_scale_state"): self._tv_scale_state = {} # ← no Redis/DB backing ``` TV_BXT_SCALE_IN enters a position over 10 slots × 60 s = up to 10.5 min. A container restart during slots 2–9 loses `slots_done`, `parent_trade_id`, `start_ms`, and `next_slot_ms`. On restart: - `_load_trades_from_db` reloads the slot-1 parent trade (already open) ✓ - Next TV webhook for the same signal creates fresh state (`slots_done=0`) - `_open_trade` detects an existing same-direction position and blocks the new entry (line ~7034 `existing = [t for t in self.open_trades() if t.symbol == sym]`) - `_tv_scale_step` pops the scale state and exits — scale is permanently abandoned with ~10% of intended size **Risk level:** Low — `TV_BXT_SCALE_IN` is a dev opt-in env var (disabled on prod). Impact is an undersized position, not a runaway or double-entry. **Fix (if promoted to prod):** Persist scale state to Redis on every slot advancement (`r.setex(f"tv_scale:{sym}", 900, json.dumps(st))`) and restore in `_load_trades_from_db` or a dedicated startup step; pop the key when scale completes/expires/aborts. --- ## Carried P1s (exit-flow domain, confirmed unfixed — see 2026-06-10 audit): These are **not** persistence bugs but are carried forward for visibility: 1. **TRIM3 1-share runner** — `_trimmed_3` flag never set; TRIM3 re-fires every bar 2. **TRIM1 1-share parent** — `_trimmed` / `_rsi_force_trim` never set/cleared 3. **TRIM2 1-share parent** — `_trimmed_2` / `_rsi_force_trim2` never cleared 4. **`profit_per_share` stale bar close** — `_in_profit` gate misroutes 5. **`_check_stops` dead code** — intrabar wick stop never checked 6. **OBV Rule F permanently disabled after in-loss skip** 7. **`TV_BXT_TRAIL_AGAINST` default mismatch** — live `1.0` vs dashboard `1.5` --- ## OK (checked, working or recently fixed): - **`risk.set_open_positions` called on startup** (`bot.py:17784`): open position count is correctly seeded, preventing immediate re-entry into a position the bot already holds. Only `trade_count` and `realized_pnl` are missing. ✓ - **`_rebuild_consec_losses` on startup** (`bot.py:827`): per-symbol streak is rebuilt from the last 20 DB trades with a Redis reset-baseline. Loss-cooldown gate survives restarts. ✓ - **OBV-trail ratchet (`obv_trail:<tid>` Redis key)** — Redis initialized before `_load_trades_from_db` (`bot.py:778–790`); ratchet restored on startup. ✓ - **`extreme_since_trim` DB column** — present in `ensure_trade_state_columns` migration (`database.py:222`); written by `_update_trade_field` on every trail advancement; restored in `_load_trades_from_db:1131`. ✓ - **`force_trim_pending` / `force_trim2_pending`** — both persisted to DB when set and restored on startup (`bot.py:1148–1151`). ✓ - **`pending_reversal`** — persisted on arm and consumed on fire (`bot.py:5226`, `5269`); restored on startup (`bot.py:1144`). ✓ - **`tick_peak_signed_vol`** — persisted per-trade to DB (`bot.py:16845`); restored in `_load_trades_from_db:1166–1167`. ✓ - **`fri_ah_close_queued`** — persisted to DB (`database.py:231`); restored on startup (`bot.py:1168–1169`). ✓ - **`_pending_manual_orders`** — backed by Redis hash (`signals.pending_manual_orders`); `_load_pending_manual_from_redis` restores live orders and drops terminal ones on startup (`bot.py:17699`). ✓ - **`ensure_trade_state_columns`** — idempotent migration runs at `__init__` (`bot.py:732–735`); covers all persisted flag columns including `trimmed_3`, `obv_be_applied`, `extreme_since_trim`. ✓ - **`_save_bars_to_redis` / `_load_bars_from_redis`** — bar snapshots persisted to Redis with 24h TTL; loaded on startup to avoid full IBKR backfill on every restart (`bot.py:1446`, `1587`). ✓ - **`DailyState.reset_if_new_day()`** — correctly resets counters at UTC midnight for next-day sessions; the P1 above is a restart-within-same-day gap, not a cross-day issue. ✓ - **`_load_pending_manual_from_redis` order-id matching** — stale orders (filled or cancelled while bot was down) are pruned from Redis; only truly open IBKR orders are restored (`bot.py:7735–7738`). ✓ - **stale comment at `bot.py:8272`** (`pass # no DB column for trim3 yet`): noted as carried risk in 2026-06-10 audit; `trimmed_3` IS already set by the `_evaluate_symbol` TRIM3 block (line 5302) before the deferred close fires, so no state loss occurs via this path for normal (>1 share) cases. ✓