Sydnee nightly — data_quality audit — 2026-06-03 P0 findings: 0 P1 findings: 3 Risks: 3 - Area: Data quality (bar integrity, Polygon timestamp handling, RVOL gate fidelity) - Branch: `dev` (82c80de) - Files scanned: `bot.py` (\_prev\_rth\_close, \_classify\_setup, \_handle\_polygon\_bar, \_evaluate\_symbol, entry sizing), `polygon_client.py` (fetch\_bars, get\_relative\_volume\_5m\_detail), `docs/strategy_decisions.md`, prior data-quality audits (2026-04-22, 2026-05-06, 2026-05-07, 2026-05-20) - API health: sandbox blocked external HTTP — source-code-only audit - Bugs found (P0 / P1): 0 / 3 (2 new, 1 carried) - Risks noted: 3 (1 new, 2 carried) Cross-check: `git log --oneline -30` confirms rsi\_extreme entry flow gate removed 2026-05-21; `_handle_tick` tf≥60 fix landed `87d5e11` (2026-05-07); dashboard prev\_close fixed via `ss._polygon_prev_close` in `6493814`. None of the new bugs below were previously filed. --- Full report (dev branch): https://github.com/kanex1/sydnee.signals/blob/dev/docs/audit_2026-06-03_data_quality.md Reply FROM [email protected] to [email protected] to request fixes, e.g.: "code_task on sydnee-signals-dev: apply fix for the P0 about RVOL threshold in bot.py" Sydnee Agent will propose + you APPROVE (or plain 'approve') + auto-push to dev. --- Full audit below (first 12 KB) --- # Nightly Audit 2026-06-03 — DATA QUALITY ## Summary - Area: Data quality (bar integrity, Polygon timestamp handling, RVOL gate fidelity) - Branch: `dev` (82c80de) - Files scanned: `bot.py` (\_prev\_rth\_close, \_classify\_setup, \_handle\_polygon\_bar, \_evaluate\_symbol, entry sizing), `polygon_client.py` (fetch\_bars, get\_relative\_volume\_5m\_detail), `docs/strategy_decisions.md`, prior data-quality audits (2026-04-22, 2026-05-06, 2026-05-07, 2026-05-20) - API health: sandbox blocked external HTTP — source-code-only audit - Bugs found (P0 / P1): 0 / 3 (2 new, 1 carried) - Risks noted: 3 (1 new, 2 carried) Cross-check: `git log --oneline -30` confirms rsi\_extreme entry flow gate removed 2026-05-21; `_handle_tick` tf≥60 fix landed `87d5e11` (2026-05-07); dashboard prev\_close fixed via `ss._polygon_prev_close` in `6493814`. None of the new bugs below were previously filed. --- ## Findings ### BUG [P1] (new): `_classify_setup` uses `_prev_rth_close` — returns today's intraday close instead of yesterday's RTH close when Polygon daily bars are loaded **Files:** `bot.py:6785` (`_classify_setup`), `bot.py:1491` (`_prev_rth_close`), `bot.py:7099` (caller in live trade entry) **Evidence:** Polygon's daily-bar `t` field is midnight UTC. After `tz_convert(PT_TZ).tz_localize(None)` in `polygon_client.py:176`, a June 3 session bar is stored as `"2026-06-02 17:00:00"` (tz-naive PT). In `_prev_rth_close`: ```python # bot.py:1503–1514 — tz-naive path: bar_d = pd.Timestamp(ts).date() # "2026-06-02 17:00:00" → bar_d = June 2 if bar_d < today_et: # June 2 < June 3 → True → RETURNED return round(float(daily_bars["close"].iloc[i]), 2) ``` `today_et = datetime.now(ET).date()` = June 3 (correct ET date). June 3's in-progress Polygon daily bar appears as June 2 in PT, passes the `< today_et` check, and is returned **first** (newest→oldest iteration) before reaching June 2's actual bar. `_classify_setup` calls this at `bot.py:6785`: ```python prev_close = day_open _pc = self._prev_rth_close(ss.bars.get(1440, pd.DataFrame())) if _pc > 0: prev_close = _pc # ← today's intraday close, not yesterday's move_from_prev = abs(price - prev_close) # ≈ 0 extended = move_from_prev > 0.5 * daily_atr_dollar # always False if trigger_type == "rsi_extreme" and extended: return "extension_fade", ... # NEVER fires ``` `_classify_setup` is called in the live entry path at `bot.py:7099` on every signal fire to set stops, targets, and the trailing multiplier. **Impact:** RSI extreme entries after large gap days (stock up/down ≥ 0.5 ATR from yesterday's close) are never classified as `extension_fade`. Instead they receive `mean_revert` stops (profile multiplier, typically 1.5–2 ATR wide) instead of the tight extension\_fade stop (beyond the day extreme, floor 0.3 ATR). A short-sell RSI extreme after a +2 ATR gap-up gets a 2 ATR stop instead of a ~0.3 ATR stop — 6× the intended risk. **Partial mitigation exists:** The dashboard status path (`bot.py:17245`) was fixed in commit `6493814` to prefer `ss._polygon_prev_close` (from Polygon snapshot's `prevDay.c`). The trading path (`_classify_setup`, `_predict_setup_mode`) was not updated. **Fix:** In `_classify_setup` (bot.py:6784–6787) and `_predict_setup_mode` (bot.py:6751–6754), prefer `ss._polygon_prev_close` before falling back to `_prev_rth_close`: ```python # Replace: _pc = self._prev_rth_close(ss.bars.get(1440, pd.DataFrame())) # With: _pc = float(getattr(ss, "_polygon_prev_close", 0) or 0) if _pc <= 0: _pc = self._prev_rth_close(ss.bars.get(1440, pd.DataFrame())) ``` --- ### BUG [P1] (new): `_handle_polygon_bar` missing tf≥60 bar-boundary fix — creates hourly bars instead of 4h/daily bars when `USE_POLYGON_BARS=true` **File:** `bot.py:2926` (`_handle_polygon_bar` aggregation loop) **Evidence:** `_handle_tick` received the tf≥60 fix in commit `87d5e11` (audit 2026-05-06): ```python # _handle_tick — FIXED (bot.py:3050–3056): if tf < 60: bs = now.replace(minute=(now.minute // tf) * tf, second=0, microsecond=0) else: _total_min = now.hour * 60 + now.minute _floored = (_total_min // tf) * tf bs = now.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(minutes=_floored) ``` `_handle_polygon_bar` still uses the old broken formula: ```python # _handle_polygon_bar — UNFIXED (bot.py:2926): bs = bar_ts.replace(minute=(bar_ts.minute // tf) * tf, second=0, microsecond=0) ``` For tf=240: `bar_ts.minute // 240 = 0` (minute ∈ 0–59 < 240). `bs = bar_ts.replace(minute=0)` = start of **current hour**. - Bar at 10:30 → bs = 10:00 (wrong; should be 08:00 for the 08–12 4h block) - Bar at 11:30 → bs = 11:00 (different from 10:30's bs → **new "4h" bar every hour**) For tf=1440: `bar_ts.minute // 1440 = 0`. `bs = bar_ts.replace(minute=0)` = start of current hour → **new "daily" bar every hour**. The view TFs default to `[1440]` (`bot.py:742`). Both tf=240 and tf=1440 are in `ss.view_tfs` for symbols that include them, so both are processed in the aggregation loop at `bot.py:2923`. **Impact (when `USE_POLYGON_BARS=true`):** 1. `ss.bars[1440]` accumulates 24 rows per trading day (one per hour) instead of 1. Each "daily" bar covers only ~1 hour of data. 2. `ss.bars[240]` accumulates 6 rows per 4h block instead of 1. 3. `_prev_rth_close` becomes immune (walk-back finds a bar from today), but all daily-chart display and `_evaluate_symbol` firing on false "new daily bars" degrades signal quality. 4. `ss.trends[1440]` flips every hour as the hourly "daily" bar BxT updates — corrupting daily trend display. `USE_POLYGON_BARS` defaults to off and is not enabled on dev or prod today, so impact is dormant. The bug must be fixed before the flag is promoted. **Fix:** Apply the same `tf >= 60` guard at `bot.py:2926`: ```python # Replace: bs = bar_ts.replace(minute=(bar_ts.minute // tf) * tf, second=0, microsecond=0) # With: if tf < 60: bs = bar_ts.replace(minute=(bar_ts.minute // tf) * tf, second=0, microsecond=0) else: _total_min = bar_ts.hour * 60 + bar_ts.minute _floored = (_total_min // tf) * tf bs = bar_ts.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(minutes=_floored) ``` --- ### BUG [P1] (carried from 2026-05-07 + 2026-05-20): Swing point list temporal inversion — false-positive divergence signals **File:** `bot.py:337, 343` (`_update_swing_points`); `bot.py:372, 378` (`_update_obv_swing_points`) The `[-1] != point` last-element dedup guard does not prevent a historical swing point from being re-appended after a newer swing, flipping temporal order: ``` Call N: rsi_swing_lows = [..., pt100] Call N+1: pt103 found → [..., pt100, pt103] ← correct Call N+2: bar 100 still in range [85, 104): [-1]=pt103 ≠ pt100 → APPEND pt100 capped → [pt100, pt103, pt100] [-1]=pt100 (OLDER), [-2]=pt103 (NEWER) — REVERSED ``` `_detect_divergence` (bot.py:458–459) reads `[-2]` as "prev" and `[-1]` as "current". After reversal, backward comparisons fire false bullish/bearish divergences that persist until a swing exits the 20-bar window. Same pattern in `_update_obv_swing_points`. Three audit cycles unfixed. **Fix:** Index-track each swing so no bar can appear twice: ```python # _update_swing_points inner block — bot.py:336–339: point = (p, r) idx_point = (i, p, r) if not any(s[0] == i for s in getattr(ss, '_rsi_sl_idx', [])): ss._rsi_sl_idx = (getattr(ss, '_rsi_sl_idx', []) + [idx_point])[-3:] ss.rsi_swing_lows = [(s[1], s[2]) for s in ss._rsi_sl_idx] # Same pattern for swing highs and OBV swing trackers. ``` --- ### RISK (new): `rv is None` fail-close blocks RSI extreme off-hours when Polygon has no recent bars — `rv is None` exemption inconsistency **File:** `bot.py:6127` (RV gate fail-close); `polygon_client.py:510` (10-min freshness gate); `bot.py:6156` (RSI extreme RVOL floor exemption) During overnight quiet periods (typically 1–4 AM PT), Polygon's latest 5m bar can be >10 min old. `get_relative_volume_5m_detail` returns `None` (freshness gate at `polygon_client.py:510`). In the entry gate: ```python # bot.py:6113–6129: rv_detail = pc.get_relative_volume_5m_detail(sym) or {} rv = rv_detail.get("rv") # None ... if rv is None: self._log_activity(sym, "... RV unavailable ...", "block") continue # ← blocks ALL triggers, including rsi_extreme ``` RSI extreme has a documented exemption from the 0.8x RVOL floor (`bot.py:6156`): ```python if not _ws_floor_decided and rv < 0.8 and not _is_rsi_extreme: continue # ← rsi_extreme bypasses this floor ``` But `rv is None` fires BEFORE the floor check and has **no `_is_rsi_extreme` exemption**. CLAUDE.md specifies RSI extreme runs 24×5 except two specific time windows. During quiet overnight, RSI extreme is silently blocked and logged as "thin tape / insufficient history" — masking the real cause (freshness gate). **Impact:** RSI extreme exhaustion setups during 1–4 AM PT (lowest pre-market activity) are missed intermittently. Activity log attributes the block to "thin tape" rather than data staleness. **Fix:** Add the RSI extreme bypass to the `rv is None` block: ```python if rv is None: if _is_rsi_extreme: pass # allow — low RVOL = exhaustion setup; data gap ≠ thin tape else: self._log_activity(sym, "... RV unavailable ...", "block") continue ``` --- ### RISK (carried from 2026-04-22 + 2026-05-20): `rv_10m` uses `min_n ≥ 1` — thin overnight baseline inflates blowoff ceiling **File:** `polygon_client.py:588` ```python min_n = min(last3[0].get("sample_count", 0), last3[1].get("sample_count", 0)) if sum_avg > 0 and min_n >= 1: rv_10m = round(sum_vol / sum_avg, 2) ``` Single-bar `rv` requires `len(same_tod) >= 3`; `rv_10m` requires only `min_n >= 1`. A single overnight historical bar at an unusual HH:MM dominates and can produce `rv_10m >> 5.0x`, incorrectly blocking valid off-hours RSI extreme exhaustion setups via the blowoff ceiling at `bot.py:6175`. **Fix:** Change `min_n >= 1` to `min_n >= 3` to match the single-bar floor. --- ### RISK (carried from 2026-04-22 + 2026-05-20): Double IBKR real-time bar subscription for bootstrapped symbols **File:** `bot.py:1684` (`_bootstrap_pending`) + `bot.py:17713` (main subscription loop) `_bootstrap_pending()` at line 1684 subscribes `ss.rtb`. The main subscription loop at line 17713 re-subscribes all symbols (including bootstrapped ones) without cancelling the first handle. The orphaned handler fires a second `_handle_tick` per 5s bar — doubling aggregated volume in `ss.bars[5]`, corrupting RSI, BXt, and OBV for the session. Mitigated on dev when `USE_POLYGON_BARS=true` skips the IBKR subscription loop. Live on prod and on any dev restart without Polygon bars. --- ## OK (checked, working or recently fixed): - **Dashboard prev\_close path:** `bot.py:17245` prefers `ss._polygon_prev_close` (from Polygon snapshot `prevDay.c`) over `_prev_rth_close`. Correctly shows yesterday's actual close on the stock cards. Commit `6493814`. ✓ - **RSI extreme entry flow gate removed:** `bot.py:5529` documents removal 2026-05-21 — gate was blocking 88% of rsi\_extreme signals (backtest showed oversold+bearish-flow LONGs were the best signals). ✓ - **`_handle_tick` tf≥60 bar-boundary:** Uses `_total_min // tf * tf` formula to correctly floor to 4h/daily boundaries. Fixed `87d5e11` (audit 2026-05-06). ✓ - **15m RSI seeded at startup:** `bot.py:2787–2791` seeds `ss.last_15m_rsi` from loaded 15m bars, preventing the first post-restart entry from using the 50.0 default. Commit `82d2fbf`. ✓ - **Polygon–IBKR tz-aware concat crash:** `polygon_client.py:176` strips tz before returning; `_handle_polygon_bar` receives tz-naive timestamps compatible with IBKR aggregation. ✓ - **RVOL freshness gate:** `polygon_client.py:509–513` checks `df.index[-1]` age (in-progress bar) against 10 min threshold — fail-closes on stale Polygon data. Gate correctly prevents use of delayed tier