Sydnee nightly — data_quality audit — 2026-04-22 P0 findings: 0 P1 findings: 3 Risks: 3 - Area: Data Quality (Wednesday theme — bar data, RSI seeding, Polygon/IBKR routing) - Branch: `dev` (aece644) - Files scanned: `polygon_client.py`, `bot.py` (data routing, bar fetching, tick handler, load_history), `requirements.txt`, `config.json` - Bugs found (P0 / P1): 0 / 3 - Risks noted: 3 --- Full report (dev branch): https://github.com/kanex1/sydnee.signals/blob/dev/docs/audit_2026-04-22_data_quality.md Reply FROM [email protected] to [email protected] to request fixes, e.g.: "code_task on sydnee-signals-dev: apply fix for the P0 about RVOL threshold in bot.py" Sydnee Agent will propose + you APPROVE (or plain 'approve') + auto-push to dev. --- Full audit below (first 12 KB) --- # Nightly Audit 2026-04-22 — DATA QUALITY ## Summary - Area: Data Quality (Wednesday theme — bar data, RSI seeding, Polygon/IBKR routing) - Branch: `dev` (aece644) - Files scanned: `polygon_client.py`, `bot.py` (data routing, bar fetching, tick handler, load_history), `requirements.txt`, `config.json` - Bugs found (P0 / P1): 0 / 3 - Risks noted: 3 --- ## Findings ### BUG [P1]: `_duration_to_days` silently truncates year/month durations to 2 days — daily chart broken on dev **File:** `polygon_client.py:35-42`, `bot.py:1320-1321` **Evidence:** `_view_tf_params` returns `"1 Y"` for daily (tf=1440), `"2 Y"` for weekly (tf=10080), `"5 Y"` for monthly (tf=43200): ```python # bot.py:1320-1321 elif tf == 1440: return "1 Y", "1 day" ``` These durations are passed to `polygon_client.fetch_bars(symbol, tf, dur)` via `_fetch_bars` when `USE_POLYGON_DATA=true`. Inside `fetch_bars`, `_duration_to_days` handles only `" D"` and `" W"` suffixes and falls through to `return 2` for everything else: ```python # polygon_client.py:35-42 def _duration_to_days(dur: str) -> int: dur = dur.strip().upper() if dur.endswith(" D"): return int(dur.split()[0]) if dur.endswith(" W"): return int(dur.split()[0]) * 7 return 2 # ← "1 Y", "2 Y", "5 Y" all land here ``` Result: Polygon fetches `start = today − (2+1) = today − 3 calendar days`. On dev (USE_POLYGON_DATA=true) as of 2026-04-22 (Wed), the daily query covers 2026-04-19 → 2026-04-22, which is Sat–Wed: **2 trading days** of daily bars instead of 1 year. **Impact:** - `ss.bars[1440]` holds only ~2 candles → daily chart view shows a nearly-empty chart for all watchlist symbols. - `ss.trends[1440]` stays 0 (neutral) because `len(bar_df) < 50` check in `_load_history:2283` skips BxT computation — the daily trend on the dashboard is always "—". - Weekly and monthly charts (tf=10080, 43200 — if ever enabled) would return 0 bars (no trading sessions in a 3-day Polygon window at weekly resolution). - `_prev_rth_close` still works: it only needs one bar dated before today, which the 2-day window provides. No stop/sizing impact. **Fix (two lines in `polygon_client.py:40-41`):** ```python if dur.endswith(" Y"): return int(dur.split()[0]) * 365 if dur.endswith(" M"): return int(dur.split()[0]) * 30 ``` Insert before `return 2`. This restores ~250 daily bars for the 1-year window. --- ### BUG [P1]: Tz-aware Polygon view-TF bars + tz-naive IBKR real-time concat — `_evaluate_symbol` skipped at 30-min boundaries **File:** `polygon_client.py:100`, `bot.py:2338-2356` **Evidence:** `polygon_client.fetch_bars` returns a tz-aware Pacific-time DatetimeIndex: ```python # polygon_client.py:100 df["date"] = pd.to_datetime(df["t"], unit="ms", utc=True).dt.tz_convert(PT_TZ) ``` This data is stored in `ss.bars[30]`, `ss.bars[60]` during `_load_history` (view TFs for tf ≥ 30 use Polygon when `USE_POLYGON_DATA=true`). `_handle_tick` (called on every 5-second IBKR real-time bar) aggregates ticks for ALL timeframes, including view TFs: ```python # bot.py:2338 for tf in sorted(set(ss.all_tfs) | {1} | set(ss.view_tfs)): ... if bs > ss._bar_starts[tf] and ss._ticks[tf]: ... ss.bars[tf] = pd.concat([ss.bars[tf], new_row]).iloc[-500:] # ← line 2351 ``` The `new_row` index is built from `ss._bar_starts[tf]`, which is derived from `b.time`. In ib_insync 0.9.86 (pinned in `requirements.txt:6`), `RealTimeBar.time` is a Unix timestamp (`float`). The fallback branch fires: ```python # bot.py:2334 now = b.time if isinstance(b.time, datetime) else datetime.fromtimestamp(b.time) ``` `datetime.fromtimestamp(float)` returns a **tz-naive** local datetime. `bs` and `new_row.index` are therefore tz-naive. **pandas 3.0.2** (pinned in `requirements.txt:9`) raises `TypeError` when concatenating tz-aware and tz-naive DatetimeIndex objects: ``` TypeError: Cannot concatenate DatetimeTZDtype and DatetimeIndex ``` **Cascade:** The sorted loop processes TFs as `[1, 5, 15, 30, 60, 1440]`. At a 30-minute boundary, tf=5 (`signal_tf`) is processed successfully and `primary_done = True` is set. When the loop reaches tf=30 (30-minute boundary), the `pd.concat` at line 2351 raises TypeError. The exception exits `_handle_tick` before `_evaluate_symbol` is reached (line 2399). Additionally: - `ss._bar_starts[30]` is NOT updated (stays at the old 30m start) - `ss._ticks[30]` is NOT cleared On every subsequent 5-second tick: `bs > ss._bar_starts[30]` remains True → retry → same TypeError → `_evaluate_symbol` skipped for ALL subsequent ticks until bot restart. **Effect onset:** First 30-minute bar boundary after startup (e.g., 10:00 AM PT). After that, `_evaluate_symbol` is never called, meaning: - No new entry signals detected for any trigger type - No stop-loss, OBV, HOLD_CAP, or RSI-trim checks run on open positions - Activity log fills with "Tick handler error" exceptions **Evidence requiring on-server confirmation:** Check dev container logs for `"[{sym}] Tick handler error"` appearing at :00 and :30 marks. If the IBKR gateway delivers tz-aware `datetime` objects for `b.time`, the bug would not manifest (tz-aware+tz-aware concat is handled by pandas). **Workaround (immediate):** Strip timezone from Polygon bars at storage time. In `_load_history` and `_bootstrap_pending`, after fetching view TF bars: ```python if df.index.tz is not None: df.index = df.index.tz_localize(None) ss.bars[tf] = df ``` **Proper fix:** In `_handle_tick` before building `new_row`: ```python if ss.bars[tf].index.tz is not None: bs = bs.replace(tzinfo=None) ``` Or normalize both to UTC at fetch time. --- ### BUG [P1]: `_load_history` seeds `last_rsi` but NOT `last_15m_rsi` — first-bar over-sizing after restart **File:** `bot.py:2296-2311` **Evidence:** `_load_history` seeds `last_rsi` and `last_atr` from 5m (signal TF) bars: ```python # bot.py:2296-2308 sig_df = ss.bars.get(self.signal_tf, pd.DataFrame()) if len(sig_df) >= 15: try: ss.last_rsi = float(_rsi(sig_df["close"], 9).iloc[-1]) ss.last_atr = float(calc_atr( sig_df["high"], sig_df["low"], sig_df["close"], 14).iloc[-1]) except Exception: logger.debug("RSI/ATR bootstrap failed for %s", sym, exc_info=True) ``` There is no equivalent seed for `ss.last_15m_rsi`. It stays at the default: ```python # bot.py:520 self.last_15m_rsi: float = 50.0 ``` `last_15m_rsi` is only updated in `_evaluate_symbol` at line 3612-3615, which runs on the first 5m bar close after startup. The update window is ≤5 minutes but crosses the entry gate for size reduction: ```python # bot.py:3873 rsi15_high = (raw_sig == 1 and ss.last_15m_rsi >= 65) or (raw_sig == -1 and ss.last_15m_rsi <= 35) # bot.py:3883 self._log_activity(sym, f"BIAS REDUCE: 15m RSI(14)={ss.last_15m_rsi:.0f} → 50%", "info") ``` With `last_15m_rsi = 50.0`, no bias reduction fires regardless of actual 15m RSI. For a stock like TSLA or META with 15m RSI at 70 immediately after restart (not unusual during momentum days), the bot enters at **full size** instead of 50%. This is directly counter to the intent of the bias-reduction gate. **Impact:** With real-money go-live 2026-05-01 approaching, an over-sized entry in the first 5 minutes post-restart (e.g., after a daily cron restart at ~05:00 PT before pre-market) could result in 2× intended risk on the first signal. **15m bars ARE loaded** at startup (tf=15 is in `all_tfs = [5, 15]`), so the data is available; it's just not read. **Fix (4 lines after line 2308 in `_load_history`):** ```python bars_15m = ss.bars.get(15, pd.DataFrame()) if len(bars_15m) >= 15: try: _rsi15 = _rsi(bars_15m["close"], 14) if not _rsi15.empty and not np.isnan(float(_rsi15.iloc[-1])): ss.last_15m_rsi = round(float(_rsi15.iloc[-1]), 1) except Exception: logger.debug("15m RSI bootstrap failed for %s", sym, exc_info=True) ``` --- ### RISK: Double real-time bar subscription for startup-pending stocks **File:** `bot.py:1290-1291` (inside `_bootstrap_pending`) and `bot.py:13062-13063` (inside `start()`) **Evidence:** At startup, `_bootstrap_pending()` (line 13012) processes stocks with `ss.status == "pending"`. It subscribes to real-time bars: ```python # bot.py:1290-1291 ss.rtb = self.ib.reqRealTimeBars(ss.contract, 5, "TRADES", False) ss.rtb.updateEvent += self._make_tick_handler(sym) ``` Then `start()` unconditionally subscribes ALL symbols again at line 13061-13063: ```python for sym, ss in self.sym_states.items(): ss.rtb = self.ib.reqRealTimeBars(ss.contract, 5, "TRADES", False) ss.rtb.updateEvent += self._make_tick_handler(sym) ``` The old `ss.rtb` from `_bootstrap_pending` is overwritten but never cancelled. IBKR continues streaming to the orphaned subscription, and its `updateEvent` handler is still attached. Result: `_handle_tick` fires **twice** per 5-second bar for any stock persisted with `status="pending"` at restart. Double-tick processing doubles volume in bar aggregation (`sum(t["v"] for t in ticks)` at line 2349). This corrupts RVOL baseline and RSI computation for the session. **Narrow edge case:** Stocks in "pending" at restart include catalyst-auto-added symbols (`bot.py:7856`: `upsert_watchlist(sym, status="pending", ...)`) or any symbol added via `add_stock()` right before a container restart. With the catalyst monitor active and daily restarts, this could affect 1-2 symbols per session. **Fix:** Before the re-subscribe loop in `start()`, cancel existing `ss.rtb` if already set: ```python if ss.rtb: try: self.ib.cancelRealTimeBars(ss.rtb) except Exception: pass ss.rtb = self.ib.reqRealTimeBars(ss.contract, 5, "TRADES", False) ``` --- ### RISK: `rv_10m` blowoff ceiling uses `min_n ≥ 1` sample — blocks valid entries on thin tape **File:** `polygon_client.py:308-310` **Evidence:** The single-bar `rv` gate (BXt/div floor) requires ≥3 historical samples: ```python # polygon_client.py:268 if avg_vol > 0 and len(same_tod) >= 3: rv = round(cur_vol / avg_vol, 2) ``` But the 10-minute `rv_10m` gate (rsi_extreme blowoff ceiling) requires only `min_n ≥ 1`: ```python # polygon_client.py:308-310 min_n = min(last3[0].get("sample_count", 0), last3[1].get("sample_count", 0)) if sum_avg > 0 and min_n >= 1: rv_10m = round(sum_vol / sum_avg, 2) ``` In overnight and pre-market sessions, a single historical reference point at an unusual HH:MM (e.g., 04:17) can produce an rv_10m far above 5.0× when the current overnight volume exceeds that one anomalous historical bar. Entry gate at `bot.py:4089` uses `rv_10m` for the blowoff block: ```python if _rv_ceil > 5.0 and _is_rsi_extreme: # signal skipped ``` **Risk:** Valid off-hours rsi_extreme entries blocked by a 1-sample rv_10m that doesn't represent the true baseline. Miss rate is fail-safe (blocked, not bad entry), but off-hours rsi_extreme is already a restricted, high-edge setup — unnecessary blocking degrades capture rate. **Suggested fix:** Align rv_10m sample threshold with rv: require `min_n >= 3` for rv_10m to be computed. When `min_n < 3`, fall back to single-bar `rv` for the blowoff ceiling. --- ### RISK: Stale EOD-flat comment in `stop()` — persists from 2026-04-21 audit (unfixed) **File:** `bot.py:13597` **Evidence:** ```python def stop(self) -> None: self.running = False # Snapshot bars to Redis for fast restart self._save_bars_to_redis() # EOD flat-by-close (3:55 PM ET) handles end-of-day exits. ← line 13597 open_count = len(self.open_trades()) if open_count: logger.info("Shutdown with %d open positions — preserving for restart", open_count) ``` The EOD sweep is disabled (positions roll overnight Mon–Thu). The comment at line 13597 implies a forced close that doesn't exist. Filed as RISK in the 2026-04-21 audit; not yet fixed (read-only audits cannot fix code). Flagging again as it approaches the 2026-05-01 real-money