Sydnee nightly — performance audit — 2026-06-14 P0 findings: 0 P1 findings: 0 Risks: 5 - Area: Performance (Saturday theme) - Branch: `dev` (ea69590) - Files scanned: `bot.py` (20 646 lines), `core/database.py`, `core/persistence.py`, `poly_ws/aggregator.py`, `docs/strategy_decisions.md`, `git log --oneline -30` - Commits since last Sat audit (2026-06-13): 4 meaningful (f977cac TAPE_STRICT_SIG2 + TAPE_TICK_EXIT; 6ea52ec + 87cfebd ENGINE/MODEL UI flattening; docs only) - Bugs found (P0 / P1): 0 / 0 - Risks noted: 5 (3 new, 2 carry from 2026-06-13) --- Full report (dev branch): https://github.com/kanex1/sydnee.signals/blob/dev/docs/audit_2026-06-14_performance.md Reply FROM [email protected] to [email protected] to request fixes, e.g.: "code_task on sydnee-signals-dev: apply fix for the P0 about RVOL threshold in bot.py" Sydnee Agent will propose + you APPROVE (or plain 'approve') + auto-push to dev. --- Full audit below (first 12 KB) --- # Nightly Audit 2026-06-14 — PERFORMANCE ## Summary - Area: Performance (Saturday theme) - Branch: `dev` (ea69590) - Files scanned: `bot.py` (20 646 lines), `core/database.py`, `core/persistence.py`, `poly_ws/aggregator.py`, `docs/strategy_decisions.md`, `git log --oneline -30` - Commits since last Sat audit (2026-06-13): 4 meaningful (f977cac TAPE_STRICT_SIG2 + TAPE_TICK_EXIT; 6ea52ec + 87cfebd ENGINE/MODEL UI flattening; docs only) - Bugs found (P0 / P1): 0 / 0 - Risks noted: 5 (3 new, 2 carry from 2026-06-13) --- ## Findings ### RISK (new): Return-on-risk (RoR) absent from all performance tracking paths **Files:** `bot.py:8493-8525` (`_compute_period_stats`), `core/database.py:1637-1695` (`compute_performance`), `core/persistence.py:19-59` (`Trade` dataclass) **Evidence:** ```python # _compute_period_stats returns: {"pnl", "trades", "win_rate", "profit_factor", "avg_win", "avg_loss", "max_drawdown", "best", "worst", "avg_daily"} # ← no return_on_risk, no avg_risk_per_trade # Trade dataclass fields: stop_price: float | None = None # only stop level stored # ← no initial_risk_dollars (shares × |entry - stop|) field ``` **Impact:** Three changes deployed 25+ days ago — 2× risk (ca4b173, 2026-05-20), reverse time-decay (5fdaad4, 2026-05-19), and MAX_POSITION_PCT 10%→15% (b3fedce, 2026-05-19) — all declare "return-on-risk (P&L ÷ total $ risk) improves" as their primary success criterion. `strategy_decisions.md` shows `Observed: TBD / Final call: TBD` for all three. Their 5-7 day validation windows closed in late May. Without RoR in any dashboard or API endpoint, passive validation is impossible; the criteria stay "pending" indefinitely. **Fix:** Compute RoR in `db.compute_performance()` via SQL: ```sql SELECT ROUND(SUM(pnl) / NULLIF(SUM( ABS(entry_price - stop_price) * shares ), 0), 3) AS return_on_risk FROM trades WHERE exit_price IS NOT NULL ... ``` Expose as `return_on_risk` in the API response and add to the Performance tab. `stop_price` is already persisted per `Trade.stop_price`. This is a read-only SQL change — no schema migration needed. --- ### RISK (new): No per-trigger breakdown in `/api/performance` — success criteria unverifiable from dashboard **Files:** `core/database.py:1637`, `bot.py:13799` **Evidence:** ```python # db.compute_performance() returns aggregate-only stats — no GROUP BY strategy_reason. # db.per_stock_performance() groups by symbol — no trigger axis. # No db.per_trigger_performance() or equivalent exists. ``` Eight pending changes in `strategy_decisions.md` use trigger-level success criteria: - "BXt WR rises from 12% to ≥25%" (2026-05-06) - "rsi_extreme WR improves week-over-week at full size" (2026-04-14) - "divergence WR rises from 30% to ≥45%" (2026-05-06) - "OBV-divergence WR > RSI-divergence WR by ≥5pp" (2026-04-26) - "bxt_reversal count drops ≥50% WoW" (Rule F, 2026-04-15) The `strategy_reason` column in `trades` carries the trigger name (e.g., `"BXt reversal"`, `"RSI extreme (oversold)"`). It exists in the DB, but no API endpoint exposes aggregated WR/avg/count by trigger. Validation is manual-SQL-only and has evidently not been performed: all 8 criteria show `Observed: TBD`. **Fix:** Add `db.per_trigger_performance(date_from, date_to)`: ```sql SELECT CASE WHEN entry_reason ILIKE '%bxt%' THEN 'bxt' WHEN entry_reason ILIKE '%rsi extreme%' THEN 'rsi_extreme' WHEN entry_reason ILIKE '%divergence%' THEN 'divergence' WHEN entry_reason ILIKE '%pullback%' THEN 'pullback' ELSE 'other' END AS trigger, COUNT(*) AS n, ROUND(100.0 * SUM(CASE WHEN pnl > 0 THEN 1 ELSE 0 END) / COUNT(*), 1) AS wr, ROUND(AVG(pnl), 2) AS avg_pnl FROM trades WHERE exit_price IS NOT NULL AND entry_time >= %s GROUP BY trigger ``` Surface via `/api/performance/by-trigger?days=7` (30s cache, matching `_perf_cache` pattern). ~20 min to add; unblocks all 8 pending validation closes. --- ### RISK (new): `TAPE_STRICT_SIG2` strict AND-gate bypassed on sidecar-absent ticks **File:** `bot.py:17211-17236` **Evidence:** ```python _strict = os.environ.get("TAPE_STRICT_SIG2", "").lower() in ("1", "true", "yes") if ask_run > 0 or bid_run > 0 or vol2 > 0: # ← sidecar tick-data present run_hit = ... flow_hit = ... if _strict: # AND-gate enforced: run_hit AND flow_hit AND opp_run<1 AND prints_5s>=70 sig2 = bool((run_hit and flow_hit and clean_tape and hot_tape) or (blocks > 0 and last_block >= _block_min * 2)) else: sig2 = bool(run_hit or flow_hit or ...) else: # ← sidecar data absent # _strict flag NOT consulted — old loose imbal30-based sig2 used: imbal30 = (abs(signed30) / vol30) if vol30 > 0 else 0.0 sig2 = bool((imbal30 >= 0.5 and d != 0 and ...) or (blocks > 0 ...)) ``` When `ask_run=bid_run=vol2=0` (sidecar quiet: no trades in the 2s window), the strict gate is silently bypassed and the pre-upgrade loose sig2 fires. The backtest that motivated `TAPE_STRICT_SIG2` was entirely on events with sidecar data present; the fallback branch was not part of the offline analysis. During brief sidecar-quiet periods (e.g., between trades at open, or during a sidecar restart), entries in the "flipper" cohort (flow_flip_within_30s = −6 bp avg, 41.8% positive) can still reach score=4 and trigger a trade despite `TAPE_STRICT_SIG2=true`. **Impact:** False confidence in "strict mode" during tick-quiet intervals. On an active RTH tape this is infrequent (v2 > 0 most of the time); in pre-market or slow midday it's more common. Doesn't affect safety (−1% stop + 60min cap still protect); affects the quality-filter property the flag is supposed to enforce. **Fix:** Add a guard in the `else` branch so strict mode requires at least the 30s imbalance to exceed the same 0.68 threshold seen in the held cohort: ```python else: imbal30 = (abs(signed30) / vol30) if vol30 > 0 else 0.0 if _strict: # No 2s data: require tighter imbal floor (held-cohort 30s avg = 0.58+) sig2 = bool(imbal30 >= 0.6 and d != 0 and ... or (blocks > 0 and last_block >= _block_min * 2)) else: sig2 = bool((imbal30 >= 0.5 and ...) or ...) ``` Or simply log `sig2=False` in strict mode when sidecar data is absent (most conservative, easiest to validate). --- ### RISK (carry from 2026-06-13): `/api/day/tick` — no response cache; Yahoo Finance 48/min call rate **File:** `bot.py:11796`, `futures_client.py:69-83` No change since 2026-06-13 filing. 5s Day-page poll × 4 Yahoo calls = 48/min; no in-process cache guard. Yahoo unofficial rate limit ~20-100/min; sustained load is fragile. Thread-pool creation overhead ~12-60 ms/min. Fix: 15s `_day_tick_cache` dict + `_day_tick_cache_ts` float (non-locking, worst case a redundant live fetch). ✓ --- ### RISK (carry from 2026-06-13): `api_day_analyze` hardcodes `claude-sonnet-4-5-20250929` **File:** `bot.py:12114` No change since 2026-06-13 filing. Current model is `claude-sonnet-4-6`. The Day page streams to the dashboard on every page open using one generation behind the current model. One-line fix. ✓ --- ## OK (checked, working or recently fixed) **`_tape_tick_exit_hit` data freshness:** `h = r.hgetall(f"signals.poly.live.{sym}")` at bot.py:18179 is fetched at the top of `_tape_eval_sym` (1Hz loop). Exit at bot.py:18266 receives the same `h`. Staleness bounded by 1s loop rate — same as the entry path. ✓ **`_tape_tick_exit_hit` zero-volume guard:** `if vol2 <= 0: return None` (bot.py:18156) prevents false exits when no trades in the 2s window. Sidecar data absent = no exit fire. ✓ **TAPE_STRICT_SIG2 AND-gate (tick-data branch):** `run_hit AND flow_hit AND opp_run<1 AND prints_5s>=70` at bot.py:17225-17228 correctly matches the held-cohort profile from the 23K-event backtest (run≥4.2 / opp≤0.5 / imbal2≥0.68 / prints_5s≥87). Threshold 70 (vs mean 87) is a deliberate calibration choice — the point was not to use the mean as a threshold. Block-only path requires 2× block_min (consistent with strategy_decisions.md). ✓ **OBV-exit and tick-exit sequence:** OBV-exit at bot.py:18252 reads from `ss` (SymbolState 5m bars); tick-exit at bot.py:18263 reads from `h` (2s Redis hash). Both gated by the same `TAPE_MIN_HOLD_SEC` (60s default). OBV runs first; if it fires, function returns before tick-exit is evaluated. No double-close. ✓ **Scalp horizon snap on stale tape:** `_snapped_{h}` is set at line 18614 regardless of whether `snap is None`. Price/flow/score land as NULL (initialized None at 18661); MFE/MAE from `ev["mfe_running"]` / `ev["mae_running"]` are non-NULL. Event finalizes correctly. `ret_{h}s` computed via `_ret(pf)` returns None on None input. Aggregates skip NULL rows. Consistent with design comment at bot.py:18504. ✓ **MFE/MAE tracking:** `mfe_running` starts 0.0 and tracks max `ret_now`; `mae_running` starts 0.0 and tracks min `ret_now`. For a long, downward moves produce negative `ret_now` → mae goes negative. For a short, upward moves produce negative `ret_now` (direction = −1). Both correct. Stale-tick guard at line 18594 (`if snap is not None and snap["price"] > 0`) prevents stale data from polluting the running trackers. ✓ **P&L formula:** `(exit_price − entry_price) × direction × shares − fees` (persistence.py:57-58). Correct for both long (+1) and short (−1). ✓ **`_compute_period_stats` max_drawdown:** sequential equity-curve walk from 0, tracking `peak − eq` max. Standard definition. Audited weekly 2026-06-14 as OK. ✓ **`profit_factor = gp / gl`:** `gp = sum(wins)`, `gl = abs(sum(losses))`. Standard definition. Guard on `gl > 0`. ✓ **`api_performance` cache:** 30s `_perf_cache` + `_perf_cache_time` (bot.py:13801-13803). Cache hits avoid the `db.compute_performance()` + `db.daily_pnl_series()` + `db.per_stock_performance()` triple query. Correct TTL for the dashboard's manual-refresh pattern. ✓ **Two prior carry risks (2026-06-13):** `_scalp_signal_log_loop` duplicate hgetall (6× Redis redundancy) and `api_scalp_signal_summary` unbounded count(*) — confirmed still present, still not auto-polled (severity unchanged, no regression). ✓