Sydnee nightly — DATA_QUALITY audit — 2026-08-12 P0 findings: 1 P1 findings: 2 Risks: 9 - Area: Data Quality (Wednesday rotation) - Branch: `dev` (7d68a01 — docs(peak-monitor): hourly log 2026-08-11 13:09 PT) - Files scanned: `bot.py` (24,173 lines), `polygon_client.py`, `core/indicators.py`, `docs/strategy_decisions.md`, `docs/audit_2026-08-05_DATA_QUALITY.md`, `git log --oneline -30` - **Bugs found (P0 / P1): 1 / 2 (all carried — no new bugs)** - **Risks noted: 9 (all carried — no new risks)** **Cross-check:** Zero code commits since the 2026-06-30 recovery commit (`c61cdb5`). All subsequent commits (48 total through today) are audit docs + peak-monitor hourly logs. All carried bugs re-verified at their exact line numbers. One status update: the P1 `_prev_rth_close` received a partial fix in the recovery commit (tz-aware branch added) that correctly handles the IBKR midnight-ET-naive path but does NOT fix the Polygon PT-naive path — the bug persists on the production codepath (30m+/daily TFs use Polygon when `USE_POLYGON_DATA=true`). API endpoints (`dev.sydnee.ai`) not reachable from sandbox — source-only analysis applied per audit instructions. --- Full report (dev branch): https://github.com/kanex1/sydnee.signals/blob/dev/docs/audit_2026-08-12_DATA_QUALITY.md Reply FROM [email protected] to [email protected] to request fixes, e.g.: "code_task on sydnee-signals-dev: apply fix for the P0 about RVOL threshold in bot.py" Sydnee Agent will propose + you APPROVE (or plain 'approve') + auto-push to dev. --- Full audit below (first 12 KB) --- # Nightly Audit 2026-08-12 — DATA QUALITY ## Summary - Area: Data Quality (Wednesday rotation) - Branch: `dev` (7d68a01 — docs(peak-monitor): hourly log 2026-08-11 13:09 PT) - Files scanned: `bot.py` (24,173 lines), `polygon_client.py`, `core/indicators.py`, `docs/strategy_decisions.md`, `docs/audit_2026-08-05_DATA_QUALITY.md`, `git log --oneline -30` - **Bugs found (P0 / P1): 1 / 2 (all carried — no new bugs)** - **Risks noted: 9 (all carried — no new risks)** **Cross-check:** Zero code commits since the 2026-06-30 recovery commit (`c61cdb5`). All subsequent commits (48 total through today) are audit docs + peak-monitor hourly logs. All carried bugs re-verified at their exact line numbers. One status update: the P1 `_prev_rth_close` received a partial fix in the recovery commit (tz-aware branch added) that correctly handles the IBKR midnight-ET-naive path but does NOT fix the Polygon PT-naive path — the bug persists on the production codepath (30m+/daily TFs use Polygon when `USE_POLYGON_DATA=true`). API endpoints (`dev.sydnee.ai`) not reachable from sandbox — source-only analysis applied per audit instructions. --- ## Findings ### BUG [P0] (carried ×7 — first filed 2026-07-08, present in every audit since): `_tf15_trend` NameError in `elif _pending:` path — exit evaluation aborts for pending-reversal symbols **File:** `bot.py:5483` **Evidence:** ```python # bot.py:5441 — immediate-fire path correctly assigns _tfmid_trend: _tfmid_trend = ss.trends.get(15, 0) # bot.py:5483 — elif _pending: path reads the never-assigned _tf15_trend: _15m_with_pos = (_tf15_trend != 0 and _tf15_trend == t.direction) # ← NameError # grep -c "_tf15_trend" bot.py → 1 (line 5483 only; no assignment anywhere in file) ``` **Impact:** `NameError` propagates and is caught by the outer `except Exception` block, aborting all subsequent exit logic (TRIM1 ~5652, TRIM2 ~5585, OBV BE-stop ~5114, HOLD_CAP ~5154) for the life of `_pending_reversal=True`. Root cause of IREN/INTC TRIM1 silent-fail incidents. **Fix:** ```python # bot.py:5483 — one rename: _15m_with_pos = (_tfmid_trend != 0 and _tfmid_trend == t.direction) ``` --- ### BUG [P1] (carried ×8 — first filed DATA_QUALITY 2026-07-01): `_prev_rth_close` returns today's Polygon daily bar as "previous RTH close" — `extension_fade` never fires since go-live **File:** `bot.py:1630–1645` **Evidence:** Polygon daily bars are stored tz-naive PT via `polygon_client.py:176`: ```python df["date"] = pd.to_datetime(df["t"], unit="ms", utc=True).dt.tz_convert(PT_TZ).dt.tz_localize(None) ``` A 2026-08-12 ET-session daily bar has `t` = epoch for 2026-08-12 00:00 UTC = 2026-08-11 17:00 PDT → stored index = `Timestamp('2026-08-11 17:00:00')` (tz-naive). The partial fix added in `c61cdb5` introduced a tz-aware branch: ```python if hasattr(ts, 'tz') and ts.tz is not None: bar_d = ts.tz_convert(ET).date() # ← NEW: correct for hypothetical tz-aware stamps else: # IBKR daily bars are naive timestamps at midnight in ET bar_d = pd.Timestamp(ts).date() # ← still fires for ALL real paths ``` Because `polygon_client.py:176` strips tz with `.dt.tz_localize(None)`, Polygon bars always arrive tz-naive → `ts.tz is not None` is always False → falls to the else branch → `pd.Timestamp('2026-08-11 17:00:00').date()` = `2026-08-11`. `today_et = datetime.now(ET).date()` = `2026-08-12`. `2026-08-11 < 2026-08-12` = True → today's Polygon bar returned as "previous RTH close". `prev_close ≈ current_price`, `move_from_prev ≈ 0`, `extended` flag never sets, `extension_fade` entry type is unreachable. *(The IBKR midnight-ET-naive path is now correctly handled by the partial fix — midnight-ET `pd.Timestamp(ts).date()` returns today's ET date, which fails the `< today_et` check and correctly falls through to yesterday's bar.)* **Impact:** `extension_fade` dead since 2026-04-19 (Polygon go-live for 30m+/daily TFs). Gap-and-fade overnight setups — the strategy's strongest mean-reversion edge — have not been identified or traded. **Fix:** ```python # bot.py:1637-1641 — replace the else branch: else: _ts = pd.Timestamp(ts) if _ts.hour >= 16: # Polygon daily bar lands at hour=17 PDT / hour=16 PST _ts = _ts + timedelta(hours=8) # recover ET calendar date (+8h covers PST and EST) bar_d = _ts.date() ``` --- ### BUG [P1] (carried ×5 — first filed DATA_QUALITY 2026-07-16): `_ab_l2_flip_check` consecutive-count buffer not cleared on stale L2 snapshot — flip exit fires on non-consecutive opposing snapshots **File:** `bot.py:20067–20088` **Evidence:** ```python def _ab_l2_flip_check(self, symbol, direction, threshold=3): l2 = self._ab_latest_l2(symbol) # max_age_ms=15000 (line 20037) if not l2: return False # ← returns False but DOES NOT clear self._ab_l2_hist[symbol] ... buf = self._ab_l2_hist.setdefault(symbol, []) if opposing: buf.append(imb) # appends on next non-stale tick — compounding prior stale count ``` `_ab_live_loop` sleeps `time.sleep(15)` at `bot.py:20249`. TTL `max_age_ms=15000` equals the loop sleep. Sub-millisecond DB write latency routinely produces `age ≥ 15001 ms` on alternate ticks → stale path fires → buf is NOT cleared → 2 opposing snapshots + 1 stale tick can satisfy `threshold=3`. **Impact:** AB executor may exit futures positions on false L2 flip signals. **Fix:** ```python # bot.py:20069-20070 — clear buf on stale: if not l2: if hasattr(self, "_ab_l2_hist"): self._ab_l2_hist[symbol] = [] return False ``` --- ### RISK (carried since 2026-08-05): `ss.last_rsi` write at `bot.py:5022` has no NaN guard — halted/constant-price stock sets `ss.last_rsi = nan`, silently blocking RSI entry and RSI-trim exits for one bar cycle **File:** `bot.py:5022` ```python ss.last_rsi = float(_rsi(close, 9).iloc[-1]) # no isnan() check ``` Constant price input → `d = s.diff()` all zeros → `ag=al=0 → 0/0 → NaN`. `ss.last_rsi = nan`. Downstream gates `nan <= 30` and `nan >= 70` both False → RSI extreme entry blocked; RSI trim exits gated on `ss.last_rsi` also silently skip. A second write site exists at `bot.py:2925` (bootstrap path, bars >= 15 but < MIN_BARS) with the same pattern, inside `try/except Exception: logger.debug(...)` — both sites unguarded. Stop-loss and ATR-trail exits still function. Risk is limited to RSI-dependent logic for the duration of the freeze. **Fix:** `ss.last_rsi = _rsi_val if not np.isnan(_rsi_val) else ss.last_rsi` (retain prior value on NaN) — match the read-site guard pattern used at `bot.py:5868,5870`. --- ### RISK (carried since 2026-08-05): `_fetch_bars` double-failure (Polygon empty + IBKR no bars) produces silent empty return — symbol stays `warming_up` indefinitely with no WARNING **File:** `bot.py:1676–1690` ```python logger.debug("Polygon returned empty for %s %dm, falling back to IBKR", ...) # DEBUG only # IBKR path — no try/except: bars = self.ib.reqHistoricalData(...) if bars: ... return df[...] return pd.DataFrame() # ← silent on combined empty; no additional log ``` **Impact:** During Polygon rate-limit or IBKR reconnect, a symbol stays in `warming_up` silently until real-time bars accumulate (~6–7h for `MIN_BARS=80` at RTH pace). An IBKR exception (error 162, disconnection) propagates to the outer handler at line 1826 → `ss.status = "no_data"` instead. No log indicates which TF failed or why. **Fix:** Raise the Polygon empty log to WARNING; add try/except around `reqHistoricalData`; add `logger.warning("_fetch_bars: both Polygon and IBKR returned empty for %s %dm", ...)` before the final `return pd.DataFrame()`. --- ### RISK (carried since 2026-08-05): No bar freshness check in `_evaluate_symbol` — stale bars from a dead feed drive live signals **File:** `bot.py:4675–4690` (`_evaluate_symbol` entry), `bot.py:4937–4955` (trend loop) `_evaluate_symbol` only checks `len(signal_df) < MIN_BARS` (count, not age). If the bar feed thread dies, `ss.bars[5]` freezes; RSI, BXt, divergence signals continue firing on hours-old data. No code of the form `time.time() - signal_df.index[-1].timestamp()` exists anywhere in the file. The WS price freshness gate (`_ws_age < 5000` at `bot.py:4267`) partially mitigates BXt confirmation, but RSI extreme and divergence setups do not require a fresh WS quote. **Fix (lightweight):** ```python # bot.py:4680 area — after MIN_BARS guard: _last_bar_age_sec = time.time() - signal_df.index[-1].timestamp() if _last_bar_age_sec > 15 * 60: logger.warning("%s bars stale: last bar %ds ago", sym, int(_last_bar_age_sec)) return ``` --- ### RISK (carried since 2026-07-16): `_ab_latest_l2` TTL equals loop sleep interval — jitter causes alternating valid/stale snapshots **File:** `bot.py:20037`, `bot.py:20249` `_ab_live_loop` sleeps 15s; `_ab_latest_l2` TTL is 15 000 ms. Sub-millisecond write latency means alternate ticks see `age > 15000 ms` → None. Root enabler of the P1 `_ab_l2_flip_check` false-fire above. **Mitigation:** Increase `max_age_ms` to 30 000 or reduce loop sleep to 10s. --- ### RISK (carried since 2026-07-15 — DORMANT, requires `USE_POLYGON_BARS=true`): `_handle_polygon_bar` bar-floor formula produces hourly buckets for tf ≥ 60 **File:** `bot.py:3076` ```python bs = bar_ts.replace(minute=(bar_ts.minute // tf) * tf, second=0, microsecond=0) # For tf=240: (minute // 240) * 240 = 0 → hourly bucket # For tf=1440: (minute // 1440) * 1440 = 0 → hourly bucket ``` `_handle_tick` (live production path, IBKR ticks) has the correct total-minute formula at `bot.py:3199–3207`. `_handle_polygon_bar` (gated by `USE_POLYGON_BARS=true`, not set in docker-compose on dev or prod) still uses the old formula. DORMANT — fix before enabling `USE_POLYGON_BARS`. --- ### RISK (carried since 2026-07-15 — DORMANT, requires `USE_POLYGON_BARS=true`): Polygon bar feed skips bars closed during startup gap **File:** `bot.py:3147–3153` On restart during RTH, `_poly_bars_last_seen[sym]` is set to the latest closed bar, then bars at that mark are skipped on the next message. Bars closed while the bot was down are silently dropped → `ss.bars[1]` has a gap at session start. DORMANT — fix when enabling. --- ### RISK (carried since 2026-07-01): VWAP session grouping treats PT-naive bar indices as UTC — session boundary at ~03:00 EDT instead of midnight ET **File:** `core/indicators.py:134` ```python idx = pd.DatetimeIndex(pd.to_datetime(close.index, utc=True)) session_groups = pd.Series(idx.date, index=close.index) ``` `pd.to_datetime(..., utc=True)` interprets PT-naive timestamps as UTC. Session boundary shifts from midnight ET to 03:00 EDT (midnight UTC = midnight PT-naive). Overnight RSI extreme entries get a 3h-shorter VWAP anchor. --- ### RISK (carried since 2026-07-01): Duplicate daily bar in `ss.bars[1440]` — Polygon PT-naive and IBKR ET-naive timestamps diverge for the same trading session **File:** `bot.py:3215` (concat without dedup) Polygon historical daily bars index at `17:00 PDT` of the prior calendar day; IBKR tick-aggregated daily bars floor to `00:00 ET` of today. Both represent the same trading session but produce different index timestamps. `pd.concat(...).iloc[-500:]` never deduplicates by session → `ss.bars[1440]` accumulates both rows, double-counting daily-bar computations. Compounds the `_prev_rth_close` P1 above. --- ### RISK (carried since 2026-07-01): `_get_rv_cached` docstring claims "60s cache" — actual Polygon client TTL is 15s **File:** `bot.py:1591–1596`, `polygon_client.py:605` ```python # bot.py:1591 — docstring: """Polygon client caches internally for 60s.""" # polygon_client.py:605 — actual: _RV_CACHE_TTL = 15 # seconds ``` Cosmetic only — no gate logic reads the docstring value. Low-priority. --- ## OK (checked, working, or recently fixed) - **Zero code commits since 2026-06-30 recovery (`c61cdb5`):** all subsequent commits are audit docs + peak-monitor hourly logs. No new code paths introduced. ✓ - **P0 `_tf15_trend` at `bot.