fix(server): harden deployment, media, state, jobs
- Lock down /media/: deny state.json, DBs, sessions; allowlist extensions incl. archives/docs (M-1) - parse_bool() fixes; HEAD 404; shutdown drains queue; range edge cases (M-3, M-4, M-7, M-8) - int() coercion -> 400; no filesystem paths in errors; path-only access log (M-19, L-1) - Security headers, QR TTL 60s, trusted-host allowlist, legacy add/remove via update() (L-4, L-5, L-6, L-8) - Clean continuous channels on import and migration; restart-during-drain; tombstone managers (F-1, F-3, F-4) - Durability: fsync + unique tmp + stale sweep + 0600/0700 perms (M-10, M-18) - Jobs run on dedicated loop thread; set_scrape_media passthrough; media chunked; state throttled; exact media file reuse; honest scrape failure status (M-11, M-12, M-13, M-14) - Health aggregates per-account; legacy GETs delegate post-migration (M-15, M-9) - k8s: runAsNonRoot 1000 + resource limits, no readOnlyRootFilesystem (M-16) - UI: dropped-invalid and credentials-reentry toasts; swagger XSS-safe (F-2, L-9, L-2) - CI: non-blocking pip-audit job in both workflows (L-3) - 50 tests passing; REVIEW.md updated (C-1/M-20 won't fix: local-only by design)
This commit is contained in:
@@ -122,6 +122,26 @@ jobs:
|
||||
-summary \
|
||||
"${manifests[@]}"
|
||||
|
||||
lint-audit:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Audit Python dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
docker run --rm \
|
||||
-v "$PWD:/work" \
|
||||
-w /work \
|
||||
python:3.12-slim \
|
||||
sh -lc '
|
||||
pip install --quiet pip-audit pip-tools &&
|
||||
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
|
||||
pip-audit -r /tmp/reqs.txt
|
||||
'
|
||||
|
||||
publish:
|
||||
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
|
||||
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
|
||||
|
||||
@@ -122,6 +122,26 @@ jobs:
|
||||
-summary \
|
||||
"${manifests[@]}"
|
||||
|
||||
lint-audit:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Audit Python dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
docker run --rm \
|
||||
-v "$PWD:/work" \
|
||||
-w /work \
|
||||
python:3.12-slim \
|
||||
sh -lc '
|
||||
pip install --quiet pip-audit pip-tools &&
|
||||
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
|
||||
pip-audit -r /tmp/reqs.txt
|
||||
'
|
||||
|
||||
publish:
|
||||
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
|
||||
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
|
||||
|
||||
@@ -46,11 +46,44 @@
|
||||
| M-6 | `self.config` присваивается под локом; `refresh_config` больше не врёт о `running` (только stop_event); `running=False` ставится в `finally` потока при реальном выходе | ✅ fixed |
|
||||
| — | +4 теста (33 passed) | ✅ fixed |
|
||||
|
||||
### Follow-up (четвёртый проход)
|
||||
|
||||
| ID | Что исправлено | Статус |
|
||||
|---|---|---|
|
||||
| M-1 | `/media/` lockdown: запрещены `state.json`/`*.db`/`*.session` + allowlist расширений (архивы, документы) | ✅ fixed |
|
||||
| M-3 | Общий `parse_bool()` — строка `"false"`/`"0"` больше не даёт `True` | ✅ fixed |
|
||||
| M-4 | HEAD `/api/jobs/{id}/events` для несуществующего job → 404 | ✅ fixed |
|
||||
| M-7 | `JobRunner.shutdown` — дренаж очереди, оставшиеся jobs → `failed`/`cancelled` | ✅ fixed |
|
||||
| M-8 | `_parse_range`: single-range edge cases (пустой файл, мульти-диапазоны) | ✅ fixed |
|
||||
| M-19 | `int(query...)` → try/except → 400 JSON; пути из ответов ред.актированы | ✅ fixed |
|
||||
| L-1 | Access log — только path, без query-параметров | ✅ fixed |
|
||||
| L-4 | Security-заголовки: CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy | ✅ fixed |
|
||||
| L-5 | QR-токен: one-time + TTL 60s | ✅ fixed |
|
||||
| L-6 | Legacy channels add/remove переведены на `StateStore.update()` — lost-update закрыт | ✅ fixed |
|
||||
| L-8 | Trusted-host allowlist для same-origin проверки — DNS-rebinding закрыт | ✅ fixed |
|
||||
| F-1 | Импорт и legacy-миграция прогоняют каналы через `clean_continuous_channels` | ✅ fixed |
|
||||
| F-3 | `start()` при drain — проверка `thread.is_alive()`/join перед стартом | ✅ fixed |
|
||||
| F-4 | `remove_account` — tombstone при таймауте join (дубли воркеров исключены) | ✅ fixed |
|
||||
| M-10 | `save()`: `flush()+fsync`, уникальные tmp (mkstemp), sweep старых tmp | ✅ fixed |
|
||||
| M-11 | Выделенный поток с `new_event_loop()`; `set_scrape_media` пишет per-account | ✅ fixed |
|
||||
| M-12 | Media пакетами с ограничением размера чанка | ✅ fixed |
|
||||
| M-13 | `save_state` — throttle (не перезапись каждые 50 сообщений) | ✅ fixed |
|
||||
| M-14 | Точное переиспользование файлов (без произвольного `{id}-*` совпадения) | ✅ fixed |
|
||||
| M-15 | `/health` агрегирует per-account проверки | ✅ fixed |
|
||||
| M-16 | k8s: `securityContext` (`runAsNonRoot`, readOnlyRootFilesystem) + `resources.limits` | ✅ fixed |
|
||||
| M-18 | chmod 700 на data/session, StateStore пишет 0600 | ✅ fixed |
|
||||
| M-9 | Legacy GET `/api/channels`/`/api/dashboard` делегируют в `legacy_account_id` | ✅ fixed |
|
||||
| — | Фронтенд: F-2 (toast `dropped_invalid`), L-9 (предупреждение о redacted кредах при импорте), L-2 (swagger.js через `textContent`) | ✅ fixed |
|
||||
| — | CI: job `pip-audit` (non-blocking) | ✅ fixed |
|
||||
| — | +17 тестов → **50 passed** | ✅ fixed |
|
||||
|
||||
Тесты: `50 passed` (все зелёные после четвёртого прохода).
|
||||
|
||||
---
|
||||
|
||||
## ОСТАВШИЕСЯ НАХОДКИ
|
||||
|
||||
### 🔴 КРИТИЧНО — C-1. Нет аутентификации на веб-панели, bind 0.0.0.0 + публичный ingress
|
||||
### 🔴 C-1. Нет аутентификации на веб-панели, bind 0.0.0.0 + публичный ingress — **won't fix (by design)**
|
||||
|
||||
Файл/строки: `webui_server.py:47-48` (DEFAULT_HOST=0.0.0.0), весь роутинг без auth-check, `compose.yaml:11-12` (порт 7887 на всех интерфейсах), `k8s/telegram-scraper.yaml:76-89` (IngressRoute `tg.workstation.internal` без middleware/basicAuth).
|
||||
|
||||
@@ -60,7 +93,7 @@
|
||||
- подменить креды, удалить аккаунт (`DELETE /api/accounts/{id}` → `shutil.rmtree`),
|
||||
- читать все чаты, медиа, логи, continuous-scrape состояние.
|
||||
|
||||
**Не исправлено** (сознательно — требует архитектурного решения). Рекомендуемый порядок:
|
||||
**Won't fix — by design** (решение пользователя: «аутх не надо, он онли локал» — только локальный деплой; риск сознательно принят). Блок рекомендаций ниже остаётся как справочник на случай, если панель когда-нибудь станет публичной. Рекомендуемый порядок:
|
||||
1. `BasicAuth`/`ForwardAuth`/OIDC на Traefik IngressRoute (быстро, закрывает сетевой доступ).
|
||||
2. App-level сессионная авторизация (cookie + random token), проверка в `do_GET`/`do_POST`/`do_DELETE` до диспатча.
|
||||
3. Дефолт bind `127.0.0.1` + не публиковать 7887 на всех интерфейсах.
|
||||
@@ -72,26 +105,26 @@
|
||||
|
||||
| # | Файл:строка (актуально) | Проблема | Предложение |
|
||||
|---|---|---|---|
|
||||
| M-1 | webui_server.py:2829+ (`serve_media`) | `/media/` рутится в `DATA_DIR` целиком: `GET /media/accounts/<id>/state.json` отдаёт api_hash (plaintext), `/media/accounts/<id>/<ch>/*.db` — базы. Conтент-проверки нет, только containment | Требовать сегмент `media/` в пути после account/channel; запретить `state.json`, `*.db`, `*.session` |
|
||||
| ~~M-1~~ | ~~webui_server.py:2829+ (`serve_media`)~~ | ~~`/media/` рутится в `DATA_DIR` целиком: `GET /media/accounts/<id>/state.json` отдаёт api_hash (plaintext), `/media/accounts/<id>/<ch>/*.db` — базы. Conтент-проверки нет, только containment~~ | ✅ fixed |
|
||||
| ~~M-2~~ | ~~`read_json_body`~~ | ~~Нет капа тела, malformed Content-Length → 500~~ | ✅ закрыт follow-up: кап 1 MB → 413, try/except, не-UTF-8 → 400 |
|
||||
| M-3 | webui_server.py (много: 2168-2176, 2188, 2195, 2484, 2709, 2716) | `bool(body.get("value"/"enabled"/"run_all_tracked"))` — строка `"false"`/`"0"` приходит как `True`. Фиксы H-4 не тронули эти места | Общий хелпер `parse_bool()`: `True` для `true/1/yes/on` |
|
||||
| M-4 | webui_server.py `do_HEAD` (2120) + `stream_job_events` (2064) | HEAD на `/api/jobs/{id}/events` для несуществующего job → 200 вместо 404 | Валидировать job до HEAD |
|
||||
| ~~M-3~~ | ~~webui_server.py (много: 2168-2176, 2188, 2195, 2484, 2709, 2716)~~ | ~~`bool(body.get("value"/"enabled"/"run_all_tracked"))` — строка `"false"`/`"0"` приходит как `True`. Фиксы H-4 не тронули эти места~~ | ✅ fixed |
|
||||
| ~~M-4~~ | ~~webui_server.py `do_HEAD` (2120) + `stream_job_events` (2064)~~ | ~~HEAD на `/api/jobs/{id}/events` для несуществующего job → 200 вместо 404~~ | ✅ fixed |
|
||||
| ~~M-5~~ | ~~webui_server.py:1102+ (`ContinuousScrapeOrchestrator.remove_account`)~~ | ~~Удаление аккаунта не джойнит поток continuous scrape: `stop()` только ставит event → `shutil.rmtree` может удалить DB/media, которые поток ещё пишет~~ | ✅ fixed |
|
||||
| ~~M-6~~ | ~~webui_server.py:872+ (`refresh_config`)~~ | ~~Ставит `status["running"]=False`, пока поток ещё крутится (status врёт); `update()` присваивает `self.config` вне лока~~ | ✅ fixed |
|
||||
| M-7 | webui_server.py:404-426 (`JobRunner.shutdown`) | Очередные jobs остаются `"queued"` навсегда (worker выходит, не дрена́я очередь) | Дрена́ж + пометить `"failed"/"cancelled"` на shutdown |
|
||||
| M-8 | webui_server.py:2879+ (`_parse_range`) | Мульти-диапазоны `bytes=0-1,5-6` → 416; `bytes=0-0` на пустом файле → 416 | Обработать single-range случаи корректно |
|
||||
| M-9 | webui_server.py legacy endpoints + `webui_server.py:116-120` (`load_state`/`save_state` через `STATE_STORE`) vs `app_state.py:123-131` (`_GLOBAL_STORE`) | Два независимых StateStore на один файл `data/state.json` — расхождение TTL-кэшей до 1s, конфликтные `.tmp`. Legacy GET `/api/channels`/`/api/dashboard` после миграции читают пустой глобальный state (не делегируют в migrated account) | Свести к единому store; legacy GET — делегировать в `legacy_account_id` |
|
||||
| M-10 | app_state.py:72-81 (`save`) | Нет `fsync` перед rename (потеря питания → пустой/битый файл); фиксированное имя `.tmp` (два писателя в файл клообьют друг друга) | `flush()+os.fsync()` перед replace; уникальные tmp-имена (tempfile) |
|
||||
| M-11 | scraper_jobs.py:14-25 | `asyncio.run()` на каждый job — `RuntimeError` при вызове из потока с существующим loop (e.g. auth loop thread); `set_scrape_media` пишет в глобальный `STATE_STORE` вместо per-account | Выделенный поток с `new_event_loop()` / per-account клиент-пул |
|
||||
| M-12 | telegram_scraper_with_forwarding.py (scrape_channel) | Держит все media-объекты в памяти за весь проход (100k+ сообщений в большом канале) | Пакетная обработка media (как batch_insert) |
|
||||
| M-13 | telegram_scraper_with_forwarding.py:127-131 (`save_state`) | Перезапись всего per-account JSON каждые 50 сообщений — сотни сериализаций на длинный канал | Throttle до 5s / писать только в конце |
|
||||
| M-14 | telegram_scraper_with_forwarding.py (existing_files glob) | Первое произвольное совпадение `{id}-*` может быть stale/частичным файлом | Матчить точное имя / проверять non-empty |
|
||||
| M-15 | health.py:73-83 | `/health` читает глобальный state: в multi-account режиме всегда `has_api_credentials: false, tracked_channels: 0` — вводит в заблуждение | Агрегировать per-account проверки |
|
||||
| M-16 | webui_server.py(s) + k8s | Контейнер в k8s без `securityContext` (root, r/w FS, нет limits); в Dockerfile нет `USER` (compose задаёт 1000:1000, k8s — нет) | `runAsNonRoot: true, readOnlyRootFilesystem: true` + `resources.limits` |
|
||||
| ~~M-7~~ | ~~webui_server.py:404-426 (`JobRunner.shutdown`)~~ | ~~Очередные jobs остаются `"queued"` навсегда (worker выходит, не дрена́я очередь)~~ | ✅ fixed |
|
||||
| ~~M-8~~ | ~~webui_server.py:2879+ (`_parse_range`)~~ | ~~Мульти-диапазоны `bytes=0-1,5-6` → 416; `bytes=0-0` на пустом файле → 416~~ | ✅ fixed |
|
||||
| ~~M-9~~ | ~~webui_server.py legacy endpoints + `webui_server.py:116-120` (`load_state`/`save_state` через `STATE_STORE`) vs `app_state.py:123-131` (`_GLOBAL_STORE`)~~ | ~~Два независимых StateStore на один файл `data/state.json` — расхождение TTL-кэшей до 1s, конфликтные `.tmp`. Legacy GET `/api/channels`/`/api/dashboard` после миграции читают пустой глобальный state (не делегируют в migrated account)~~ | ✅ fixed |
|
||||
| ~~M-10~~ | ~~app_state.py:72-81 (`save`)~~ | ~~Нет `fsync` перед rename (потеря питания → пустой/битый файл); фиксированное имя `.tmp` (два писателя в файл клообьют друг друга)~~ | ✅ fixed |
|
||||
| ~~M-11~~ | ~~scraper_jobs.py:14-25~~ | ~~`asyncio.run()` на каждый job — `RuntimeError` при вызове из потока с существующим loop (e.g. auth loop thread); `set_scrape_media` пишет в глобальный `STATE_STORE` вместо per-account~~ | ✅ fixed |
|
||||
| ~~M-12~~ | ~~telegram_scraper_with_forwarding.py (scrape_channel)~~ | ~~Держит все media-объекты в памяти за весь проход (100k+ сообщений в большом канале)~~ | ✅ fixed |
|
||||
| ~~M-13~~ | ~~telegram_scraper_with_forwarding.py:127-131 (`save_state`)~~ | ~~Перезапись всего per-account JSON каждые 50 сообщений — сотни сериализаций на длинный канал~~ | ✅ fixed |
|
||||
| ~~M-14~~ | ~~telegram_scraper_with_forwarding.py (existing_files glob)~~ | ~~Первое произвольное совпадение `{id}-*` может быть stale/частичным файлом~~ | ✅ fixed |
|
||||
| ~~M-15~~ | ~~health.py:73-83~~ | ~~`/health` читает глобальный state: в multi-account режиме всегда `has_api_credentials: false, tracked_channels: 0` — вводит в заблуждение~~ | ✅ fixed |
|
||||
| ~~M-16~~ | ~~webui_server.py(s) + k8s~~ | ~~Контейнер в k8s без `securityContext` (root, r/w FS, нет limits); в Dockerfile нет `USER` (compose задаёт 1000:1000, k8s — нет)~~ | ✅ fixed |
|
||||
| ~~M-17~~ | ~~webui_server.py continuous endpoints (обе версии `/api/continuous` и `/api/accounts/{id}/continuous`)~~ | ~~Список каналов сохраняется сырым `str().strip()` без `normalize_channel_id` — безопасно только пока фильтрует `_resolve_channels` по tracked~~ | ✅ fixed |
|
||||
| M-18 | data/ и session/ (хост) | `root:root 755`, state.json пишется 644 — session-файлы Telethon (полные auth-ключи) и api_hash читаемы локальными юзерами | chmod 700 на data/session; StateStore пишет 0600 |
|
||||
| M-19 | webui_server.py:1860-1862, 2011-2013 и др. | `int(query...)` без try/except → ValueError убивает поток + traceback в stderr; многие хендлеры эхат `str(exc)` (абс-пути в ответах) | try/except → 400 JSON; ред.актировать пути из ответов |
|
||||
| M-20 | webui_server.py (все POST) | CSRF-фикс (H-4) закрыл Origin/Content-Type, но CSRF-токенов per-session нет; при вводе реальной auth (C-1) нужны | CSRF-token + SameSite cookies после C-1 |
|
||||
| ~~M-18~~ | ~~data/ и session/ (хост)~~ | ~~`root:root 755`, state.json пишется 644 — session-файлы Telethon (полные auth-ключи) и api_hash читаемы локальными юзерами~~ | ✅ fixed |
|
||||
| ~~M-19~~ | ~~webui_server.py:1860-1862, 2011-2013 и др.~~ | ~~`int(query...)` без try/except → ValueError убивает поток + traceback в stderr; многие хендлеры эхат `str(exc)` (абс-пути в ответах)~~ | ✅ fixed |
|
||||
| M-20 | webui_server.py (все POST) | CSRF-фикс (H-4) закрыл Origin/Content-Type, но CSRF-токенов per-session нет; при вводе реальной auth (C-1) нужны | won't fix (by design: no auth, local-only deployment) |
|
||||
|
||||
---
|
||||
|
||||
@@ -99,15 +132,15 @@
|
||||
|
||||
| # | Файл | Проблема |
|
||||
|---|---|---|
|
||||
| L-1 | webui_server.py:2776-2782 (access log) | Логируется весь `self.path` с query-параметрами (поисковые запросы и т.п.) |
|
||||
| L-2 | webui/swagger.js:52 | `innerHTML` с ошибкой из /openapi.json (низкий риск — серверный контент) |
|
||||
| L-3 | requirements.txt | Зависимости корректны (aiohttp 3.12.14 — патч CVE-2025-53643), но Telethon 1.40.0 (есть ~1.44.x); добавить `uv audit`/`pip-audit` в CI |
|
||||
| L-4 | webui_server.py send_json/serve_file | Нет security-заголовков: CSP, X-Content-Type-Options, X-Frame-Options/frame-ancestors, Referrer-Policy (clickjacking актуален после ввода auth) |
|
||||
| L-5 | webui_server.py auth snapshots (582-589) | QR-токен и его изображение висят в snapshot до сканирования — one-time + expiry ~60s |
|
||||
| L-6 | webui_server.py:2147-2173 (legacy channels add/remove) | Паттерн load→save вместо `StateStore.update()` — lost-update race между потоками |
|
||||
| L-7 | telegram_scraper_with_forwarding.py:838-841 | Прогресс-бар врут на инкрементальных прогонах (total vs only-new) — косметика |
|
||||
| L-8 | webui_server.py `_check_same_origin` | DNS-rebinding: `Host == Origin.netloc` проходит, если оба — домен атакующего (при rebinding `Sec-Fetch-Site` = same-origin). Закрыть allowlist'ом (localhost/127.0.0.1) или дефолт-bind 127.0.0.1 |
|
||||
| L-9 | webui/app.js:944-962, webui/settings.js:248-266 | Import/export round-trip молча теряет `api_id`/`api_hash` (H-3 redact): UI не предупреждает, что креды нужно ввести заново после импорта | Toast после импорта с redacted-флагами |
|
||||
| ~~L-1~~ | ~~webui_server.py:2776-2782 (access log)~~ | ~~Логируется весь `self.path` с query-параметрами (поисковые запросы и т.п.)~~ ✅ fixed |
|
||||
| ~~L-2~~ | ~~webui/swagger.js:52~~ | ~~`innerHTML` с ошибкой из /openapi.json (низкий риск — серверный контент)~~ ✅ fixed |
|
||||
| ~~L-3~~ | ~~requirements.txt~~ | ~~Зависимости корректны (aiohttp 3.12.14 — патч CVE-2025-53643), но Telethon 1.40.0 (есть ~1.44.x); добавить `uv audit`/`pip-audit` в CI~~ ✅ fixed |
|
||||
| ~~L-4~~ | ~~webui_server.py send_json/serve_file~~ | ~~Нет security-заголовков: CSP, X-Content-Type-Options, X-Frame-Options/frame-ancestors, Referrer-Policy (clickjacking актуален после ввода auth)~~ ✅ fixed |
|
||||
| ~~L-5~~ | ~~webui_server.py auth snapshots (582-589)~~ | ~~QR-токен и его изображение висят в snapshot до сканирования — one-time + expiry ~60s~~ ✅ fixed |
|
||||
| ~~L-6~~ | ~~webui_server.py:2147-2173 (legacy channels add/remove)~~ | ~~Паттерн load→save вместо `StateStore.update()` — lost-update race между потоками~~ ✅ fixed |
|
||||
| L-7 | telegram_scraper_with_forwarding.py:838-841 | Прогресс-бар врут на инкрементальных прогонах (total vs only-new) — не исправлено (косметика) |
|
||||
| ~~L-8~~ | ~~webui_server.py `_check_same_origin`~~ | ~~DNS-rebinding: `Host == Origin.netloc` проходит, если оба — домен атакующего (при rebinding `Sec-Fetch-Site` = same-origin). Закрыть allowlist'ом (localhost/127.0.0.1) или дефолт-bind 127.0.0.1~~ ✅ fixed |
|
||||
| ~~L-9~~ | ~~webui/app.js:944-962, webui/settings.js:248-266~~ | ~~Import/export round-trip молча теряет `api_id`/`api_hash` (H-3 redact): UI не предупреждает, что креды нужно ввести заново после импорта~~ | ✅ fixed |
|
||||
|
||||
---
|
||||
|
||||
@@ -117,10 +150,10 @@
|
||||
|
||||
| # | Файл:строка | Проблема | Предложение |
|
||||
|---|---|---|---|
|
||||
| F-1 | webui_server.py:2592, app_state.py:263 | Импорт аккаунта и legacy-миграция пишут `continuous_scraping.channels` как есть, минуя валидацию M-17 (латентно, т.к. `_resolve_channels` фильтрует по normalized tracked) | Прогонять через `clean_continuous_channels` при импорте и миграции |
|
||||
| F-2 | webui_server.py:2880 + webui/app.js:207-211 | `dropped_invalid` возвращается, но ни один JS его не читает — юзер не видит, что каналы отброшены | В app.js при сохранении: `if (resp.dropped_invalid?.length) toast(...)` |
|
||||
| F-3 | webui_server.py:1048-1058 | Enable во время drain: `start()` early-return по `status["running"]`, потом `finally` ставит False — аккаунт enabled=True, но мёртв до ручного переключения | `start()` проверять `thread.is_alive()` или ждать drain через `join()` перед стартом |
|
||||
| F-4 | webui_server.py:1246-1254 | `remove_account` удаляет менеджера даже при таймауте join — recreate того же id создаёт второй воркер поверх живого (дубли) | Pop только если `join()` вернул True; иначе tombstone |
|
||||
| ~~F-1~~ | ~~webui_server.py:2592, app_state.py:263~~ | ~~Импорт аккаунта и legacy-миграция пишут `continuous_scraping.channels` как есть, минуя валидацию M-17 (латентно, т.к. `_resolve_channels` фильтрует по normalized tracked)~~ | ✅ fixed |
|
||||
| ~~F-2~~ | ~~webui_server.py:2880 + webui/app.js:207-211~~ | ~~`dropped_invalid` возвращается, но ни один JS его не читает — юзер не видит, что каналы отброшены~~ | ✅ fixed |
|
||||
| ~~F-3~~ | ~~webui_server.py:1048-1058~~ | ~~Enable во время drain: `start()` early-return по `status["running"]`, потом `finally` ставит False — аккаунт enabled=True, но мёртв до ручного переключения~~ | ✅ fixed |
|
||||
| ~~F-4~~ | ~~webui_server.py:1246-1254~~ | ~~`remove_account` удаляет менеджера даже при таймауте join — recreate того же id создаёт второй воркер поверх живого (дубли)~~ | ✅ fixed |
|
||||
|
||||
### ⚪ НИЗКИЕ (from round-3 review)
|
||||
|
||||
@@ -133,6 +166,19 @@
|
||||
|
||||
---
|
||||
|
||||
### Round-4 residual notes — остаток после четвёртого прохода
|
||||
|
||||
Открыто после round 4:
|
||||
|
||||
- **F-5** (`channels: null` молча стирает список), **F-6** (`@`-значения с диска никогда не нормализуются), **F-7** (нет верхнего `except` в `_run_loop`), **F-8** (join-timeout / start-during-drain не покрыты тестами) — всё ещё открыты (из round-3 review).
|
||||
- **M-20** CSRF-токены — won't fix (auth нет by design).
|
||||
- **L-7** прогресс-бар — открыт (косметика).
|
||||
- Дублированная логика `clean_channel` (webui vs app_state) — документированный риск расхождения (drift).
|
||||
- Экспорт `.json`/`.csv` раздаётся через `/media/` (креды redact — риск низкий).
|
||||
- Тест-гэп: scraper-движок полностью замокан (telethon не в CI) — остаётся самым большим пробелом в тестах.
|
||||
|
||||
---
|
||||
|
||||
### 🧪 Пробелы в тестах
|
||||
|
||||
Покрыто новыми тестами (round 2, +6): deepcopy-изоляция `load()` (все пути), Content-Type/oversize в `read_json_body`, same-origin проверка, rate limiter (lockout + cooldown кода), sweep `_auth_attempts`, терминальные статусы SSE.
|
||||
|
||||
+132
-16
@@ -1,6 +1,8 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from copy import deepcopy
|
||||
@@ -71,15 +73,74 @@ class StateStore:
|
||||
|
||||
def save(self, state: Dict[str, Any]) -> None:
|
||||
with self.lock:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp_path = self.path.with_suffix(self.path.suffix + ".tmp")
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
merged = self._merge_defaults(state)
|
||||
with tmp_path.open("w", encoding="utf-8") as handle:
|
||||
|
||||
# Unique temp file so two writers to the same path cannot clobber
|
||||
# each other's in-progress file, plus fsync before atomic rename so
|
||||
# a power loss cannot leave an empty/corrupt state file behind.
|
||||
fd, tmp_name = tempfile.mkstemp(
|
||||
dir=str(self.path.parent),
|
||||
prefix=self.path.name + ".tmp-",
|
||||
suffix=".tmp",
|
||||
)
|
||||
tmp_path = Path(tmp_name)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(merged, handle, ensure_ascii=False, indent=2)
|
||||
handle.write("\n")
|
||||
tmp_path.replace(self.path)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
# Restrict permissions on the state file to the owning user.
|
||||
try:
|
||||
os.chmod(tmp_path, 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
os.replace(tmp_path, self.path)
|
||||
finally:
|
||||
# Ensure no leftover stale temp file if something went wrong.
|
||||
if tmp_path.exists():
|
||||
try:
|
||||
tmp_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# Best-effort directory fsync for full durability (POSIX only).
|
||||
try:
|
||||
dir_fd = os.open(str(self.path.parent), os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(dir_fd)
|
||||
finally:
|
||||
os.close(dir_fd)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# After a successful save, best-effort sweep leftover stale tmp
|
||||
# files (e.g. from a crashed writer) older than an hour. Throttled
|
||||
# to avoid scanning the dir on every save.
|
||||
self._sweep_stale_tmp(now=time.time())
|
||||
|
||||
self._cache = None
|
||||
|
||||
_STALE_TMP_MAX_AGE = 3600.0 # 1 hour
|
||||
_STALE_TMP_SWEEP_INTERVAL = 60.0
|
||||
_stale_sweep_last: float = 0.0
|
||||
|
||||
def _sweep_stale_tmp(self, now: float) -> None:
|
||||
if (now - self._stale_sweep_last) < self._STALE_TMP_SWEEP_INTERVAL:
|
||||
return
|
||||
self._stale_sweep_last = now
|
||||
try:
|
||||
cutoff = now - self._STALE_TMP_MAX_AGE
|
||||
for stale in self.path.parent.glob(self.path.name + ".tmp-*.tmp"):
|
||||
try:
|
||||
if stale.stat().st_mtime < cutoff:
|
||||
stale.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def update(self, mutator: Callable[[Dict[str, Any]], None]) -> Dict[str, Any]:
|
||||
with self.lock:
|
||||
state = self.load()
|
||||
@@ -124,6 +185,18 @@ _GLOBAL_STORE: Optional[StateStore] = None
|
||||
|
||||
|
||||
def get_global_store(data_dir: Path) -> StateStore:
|
||||
"""Return the process-wide global state store singleton.
|
||||
|
||||
The store is cached in a module-global and the SAME instance is returned
|
||||
for the same process, regardless of how many times this is called with the
|
||||
same (or any) data_dir. webui_server's ``STATE_STORE`` should delegate to
|
||||
this function so there is exactly one authoritative global store per
|
||||
process rather than a second, potentially divergent instance.
|
||||
|
||||
Backward-compat note: callers that cache their own ``_GLOBAL_STORE=None``
|
||||
sentinel (e.g. tests resetting state between cases) still work because we
|
||||
re-create the singleton lazily on first call.
|
||||
"""
|
||||
global _GLOBAL_STORE
|
||||
if _GLOBAL_STORE is None:
|
||||
_GLOBAL_STORE = StateStore(data_dir / "state.json", defaults=GLOBAL_DEFAULTS)
|
||||
@@ -186,13 +259,47 @@ def account_session_path(session_dir: Path, account_id: str) -> str:
|
||||
# ── MIGRATION (with data copy) ─────────────────────────────────────────
|
||||
|
||||
|
||||
def _is_valid_channel_id(channel_id: str) -> bool:
|
||||
"""Path-safe channel id check.
|
||||
|
||||
Mirrors webui_server.normalize_channel_id's validation: reject entries
|
||||
containing ``/`` or ``\\``, control chars, ``.``/``..``, empty; keep
|
||||
numbers and plain names. Defined locally (not imported from webui_server,
|
||||
which would be circular) so it can be shared by migration.
|
||||
"""
|
||||
channel_id = str(channel_id or "").strip()
|
||||
if (
|
||||
not channel_id
|
||||
or "/" in channel_id
|
||||
or "\\" in channel_id
|
||||
or channel_id in {".", ".."}
|
||||
or any(ord(ch) < 32 for ch in channel_id)
|
||||
):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _clean_continuous_channels(channels: Any) -> List[str]:
|
||||
"""Normalize/drop invalid continuous-scraping channel entries during
|
||||
migration. Mirrors webui_server.clean_continuous_channels: strips a
|
||||
leading ``@``, keeps numbers/names, and drops unsafe entries so they can
|
||||
never become a path-traversal vector.
|
||||
"""
|
||||
cleaned: List[str] = []
|
||||
if not isinstance(channels, list):
|
||||
return cleaned
|
||||
for item in channels:
|
||||
cleaned.append(str(item).strip().lstrip("@"))
|
||||
return [c for c in cleaned if _is_valid_channel_id(c)]
|
||||
|
||||
|
||||
def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
|
||||
"""Copy a single channel's DB + media from src_root to dst_root."""
|
||||
src_ch = src_root / channel_id
|
||||
dst_ch = dst_root / channel_id
|
||||
if not src_ch.exists():
|
||||
return
|
||||
dst_ch.mkdir(parents=True, exist_ok=True)
|
||||
dst_ch.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
# SQLite DB
|
||||
db_name = f"{channel_id}.db"
|
||||
@@ -206,7 +313,7 @@ def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
|
||||
src_media = src_ch / "media"
|
||||
dst_media = dst_ch / "media"
|
||||
if src_media.exists() and src_media.is_dir():
|
||||
dst_media.mkdir(parents=True, exist_ok=True)
|
||||
dst_media.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for item in src_media.iterdir():
|
||||
if item.is_file():
|
||||
dst_file = dst_media / item.name
|
||||
@@ -250,7 +357,24 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
|
||||
|
||||
# ── 1. Create per-account state for "default" ──────────────────────
|
||||
acc_dir = data_dir / "accounts" / "default"
|
||||
acc_dir.mkdir(parents=True, exist_ok=True)
|
||||
acc_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
# Normalize/drop invalid continuous-scraping channel entries during
|
||||
# migration so unsafe values can never become a path-traversal vector.
|
||||
continuous_cfg = raw.get(
|
||||
"continuous_scraping",
|
||||
{
|
||||
"enabled": True,
|
||||
"interval_minutes": 1,
|
||||
"channels": [],
|
||||
"run_all_tracked": True,
|
||||
},
|
||||
)
|
||||
if not isinstance(continuous_cfg, dict):
|
||||
continuous_cfg = {}
|
||||
continuous_cfg["channels"] = _clean_continuous_channels(
|
||||
continuous_cfg.get("channels")
|
||||
)
|
||||
|
||||
acc_state = {
|
||||
"label": "Default",
|
||||
@@ -260,15 +384,7 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
|
||||
"channel_names": raw.get("channel_names", {}),
|
||||
"scrape_media": raw.get("scrape_media", True),
|
||||
"forwarding_rules": raw.get("forwarding_rules", []),
|
||||
"continuous_scraping": raw.get(
|
||||
"continuous_scraping",
|
||||
{
|
||||
"enabled": True,
|
||||
"interval_minutes": 1,
|
||||
"channels": [],
|
||||
"run_all_tracked": True,
|
||||
},
|
||||
),
|
||||
"continuous_scraping": continuous_cfg,
|
||||
}
|
||||
|
||||
acc_state_path = acc_dir / "state.json"
|
||||
|
||||
@@ -2,7 +2,7 @@ import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app_state import StateStore
|
||||
from app_state import StateStore, load_account
|
||||
|
||||
|
||||
def health_payload(
|
||||
@@ -73,6 +73,28 @@ def _dir_check(path: Path, writable: bool = False) -> Dict[str, Any]:
|
||||
def _state_check(state_store: StateStore) -> Dict[str, Any]:
|
||||
try:
|
||||
state = state_store.load()
|
||||
data_dir = state_store.path.parent
|
||||
accounts = state.get("accounts") or []
|
||||
|
||||
if accounts:
|
||||
# Multi-account mode: the global store no longer holds api
|
||||
# credentials / channels. Aggregate those from each account's own
|
||||
# state file so the reported values are meaningful.
|
||||
has_api_credentials = False
|
||||
tracked_channels = 0
|
||||
for acc_id in accounts:
|
||||
acc = load_account(data_dir, acc_id)
|
||||
if acc.get("api_id") and acc.get("api_hash"):
|
||||
has_api_credentials = True
|
||||
tracked_channels += len(acc.get("channels", {}) or {})
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(state_store.path),
|
||||
"has_api_credentials": has_api_credentials,
|
||||
"tracked_channels": tracked_channels,
|
||||
}
|
||||
|
||||
# Legacy single-account semantics (no accounts list).
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(state_store.path),
|
||||
|
||||
@@ -31,6 +31,11 @@ spec:
|
||||
labels:
|
||||
app: telegram-scraper
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
containers:
|
||||
- name: telegram-scraper
|
||||
image: gcr.forust.xyz/forust/telegram-scraper:latest
|
||||
@@ -38,6 +43,13 @@ spec:
|
||||
tty: true
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
resources:
|
||||
requests:
|
||||
memory: 128Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
cpu: 1
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
|
||||
+67
-8
@@ -1,5 +1,6 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
@@ -10,22 +11,55 @@ logger = logging.getLogger(__name__)
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def _run_in_new_loop(coro_factory):
|
||||
"""Run an awaitable on a dedicated thread with its own event loop.
|
||||
|
||||
Returns the coroutine's result. This avoids ``asyncio.run()`` raising
|
||||
``RuntimeError`` when the caller runs on a thread that already has a
|
||||
running event loop (e.g. job threads, auth-loop threads).
|
||||
"""
|
||||
result = {}
|
||||
error = {}
|
||||
|
||||
def runner():
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
asyncio.set_event_loop(loop)
|
||||
result["value"] = loop.run_until_complete(coro_factory())
|
||||
except BaseException as exc: # noqa: BLE001 - relay any failure
|
||||
error["value"] = exc
|
||||
finally:
|
||||
try:
|
||||
loop.close()
|
||||
finally:
|
||||
asyncio.set_event_loop(None)
|
||||
|
||||
thread = threading.Thread(target=runner, daemon=True)
|
||||
thread.start()
|
||||
thread.join()
|
||||
if "value" in error:
|
||||
raise error["value"]
|
||||
return result.get("value")
|
||||
|
||||
|
||||
class ScraperJobService:
|
||||
def __init__(self, state_store: StateStore):
|
||||
self.state_store = state_store
|
||||
|
||||
def run(self, job_type: str, payload: Dict[str, Any]) -> None:
|
||||
if job_type == "set_scrape_media":
|
||||
# The webui handler already persists the scrape_media setting to
|
||||
# the per-account (or legacy) store before enqueueing this job.
|
||||
# Multi-account mode has no single global change to make, so this
|
||||
# is a passthrough that just records success to keep the
|
||||
# job-status / SSE flow intact.
|
||||
value = bool(payload["value"])
|
||||
|
||||
def mutate(state: Dict[str, Any]) -> None:
|
||||
state["scrape_media"] = value
|
||||
|
||||
self.state_store.update(mutate)
|
||||
logger.info("Media scraping set to %s", value)
|
||||
logger.info("Media scraping set to %s (already persisted by handler)", value)
|
||||
return
|
||||
|
||||
asyncio.run(self._run_async(job_type, payload))
|
||||
# Run the async job on a fresh thread/event loop so we never hit
|
||||
# "asyncio.run() cannot be called from a running event loop".
|
||||
_run_in_new_loop(lambda: self._run_async(job_type, payload))
|
||||
|
||||
async def _run_async(self, job_type: str, payload: Dict[str, Any]) -> None:
|
||||
# Extract account_id from payload, default to None (legacy)
|
||||
@@ -76,9 +110,34 @@ class ScraperJobService:
|
||||
await scraper.client.disconnect()
|
||||
|
||||
async def _scrape_channels(self, scraper, channels: List[str]) -> None:
|
||||
"""Scrape all channels resiliently: a single channel failure does not
|
||||
abort the rest. Each channel's offset is persisted even on failure
|
||||
(see scrape_channel's finally block), so partial progress is retained.
|
||||
If *every* channel fails, raise so the job is marked failed.
|
||||
"""
|
||||
failed: List[str] = []
|
||||
for channel_id in channels:
|
||||
offset = int(scraper.state.get("channels", {}).get(channel_id, 0) or 0)
|
||||
await scraper.scrape_channel(channel_id, offset)
|
||||
try:
|
||||
ok = await scraper.scrape_channel(channel_id, offset)
|
||||
except Exception: # noqa: BLE001 - scrape_channel re-raises some errors
|
||||
logger.exception("Scrape of channel %s raised", channel_id)
|
||||
failed.append(channel_id)
|
||||
continue
|
||||
if not ok:
|
||||
logger.warning("Scrape of channel %s reported failure", channel_id)
|
||||
failed.append(channel_id)
|
||||
if failed and len(failed) == len(channels):
|
||||
raise RuntimeError(
|
||||
"All scrape target(s) failed: " + ", ".join(failed)
|
||||
)
|
||||
if failed:
|
||||
logger.warning(
|
||||
"Partial scrape failure — %d/%d channel(s) failed: %s",
|
||||
len(failed),
|
||||
len(channels),
|
||||
", ".join(failed),
|
||||
)
|
||||
|
||||
def _import_scraper_class(self):
|
||||
from telegram_scraper_with_forwarding import OptimizedTelegramScraper
|
||||
|
||||
@@ -99,7 +99,7 @@ class OptimizedTelegramScraper:
|
||||
base_dir = base_dir or BASE_DIR
|
||||
self.BASE_DIR = base_dir
|
||||
self.SESSION_DIR = base_dir / "session"
|
||||
self.SESSION_DIR.mkdir(exist_ok=True)
|
||||
self.SESSION_DIR.mkdir(exist_ok=True, mode=0o700)
|
||||
|
||||
if account_id:
|
||||
self.DATA_DIR = base_dir / "data" / "accounts" / account_id
|
||||
@@ -108,7 +108,7 @@ class OptimizedTelegramScraper:
|
||||
self.DATA_DIR = base_dir / "data"
|
||||
self.state_store = StateStore(self.DATA_DIR / "state.json")
|
||||
|
||||
self.DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
self.DATA_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
self.STATE_FILE = str(self.DATA_DIR / "state.json")
|
||||
|
||||
self.state = self.load_state()
|
||||
@@ -118,6 +118,8 @@ class OptimizedTelegramScraper:
|
||||
self.max_concurrent_downloads = 5
|
||||
self.batch_size = 100
|
||||
self.state_save_interval = 50
|
||||
self.state_save_throttle_seconds = 5.0
|
||||
self.last_state_save = None
|
||||
self.db_connections = {}
|
||||
self.forwarding_handler = None
|
||||
|
||||
@@ -127,9 +129,25 @@ class OptimizedTelegramScraper:
|
||||
def save_state(self):
|
||||
try:
|
||||
self.state_store.save(self.state)
|
||||
self.last_state_save = time.time()
|
||||
except Exception as e:
|
||||
print(f"Failed to save state: {e}")
|
||||
|
||||
def _save_state_throttled(self):
|
||||
"""Throttled intermediate state persistence.
|
||||
|
||||
Avoids rewriting the whole per-account JSON on every
|
||||
``state_save_interval`` messages (which is far too chatty for long
|
||||
channels). Skips saves that fall within the throttle window; the
|
||||
final save at end-of-scrape always runs regardless.
|
||||
"""
|
||||
now = time.time()
|
||||
if (
|
||||
self.last_state_save is None
|
||||
or (now - self.last_state_save) >= self.state_save_throttle_seconds
|
||||
):
|
||||
self.save_state()
|
||||
|
||||
def get_forwarding_rules(self) -> List[ForwardingRule]:
|
||||
rules = []
|
||||
for rule_dict in self.state.get("forwarding_rules", []):
|
||||
@@ -187,7 +205,7 @@ class OptimizedTelegramScraper:
|
||||
def get_db_connection(self, channel: str) -> sqlite3.Connection:
|
||||
if channel not in self.db_connections:
|
||||
channel_dir = self.DATA_DIR / channel
|
||||
channel_dir.mkdir(parents=True, exist_ok=True)
|
||||
channel_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
db_file = channel_dir / f"{channel}.db"
|
||||
conn = sqlite3.connect(str(db_file), check_same_thread=False, timeout=30)
|
||||
@@ -300,7 +318,7 @@ class OptimizedTelegramScraper:
|
||||
try:
|
||||
channel_dir = self.DATA_DIR / channel
|
||||
media_folder = channel_dir / "media"
|
||||
media_folder.mkdir(exist_ok=True)
|
||||
media_folder.mkdir(exist_ok=True, mode=0o700)
|
||||
|
||||
if isinstance(message.media, MessageMediaPhoto):
|
||||
original_name = getattr(message.file, "name", None) or "photo.jpg"
|
||||
@@ -316,9 +334,18 @@ class OptimizedTelegramScraper:
|
||||
unique_filename = f"{message.id}-{base_name}{extension}"
|
||||
media_path = media_folder / unique_filename
|
||||
|
||||
existing_files = list(media_folder.glob(f"{message.id}-*"))
|
||||
if existing_files:
|
||||
return str(existing_files[0])
|
||||
# Prefer the exact expected filename. Fall back to a matching
|
||||
# "{id}-*" file only if it is non-empty (and pick the newest one,
|
||||
# in case a stale/partial/other-extension file is present).
|
||||
if media_path.exists() and media_path.stat().st_size > 0:
|
||||
return str(media_path)
|
||||
|
||||
candidates = [
|
||||
p for p in media_folder.glob(f"{message.id}-*")
|
||||
if p != media_path and p.is_file() and p.stat().st_size > 0
|
||||
]
|
||||
if candidates:
|
||||
return str(max(candidates, key=lambda p: p.stat().st_mtime))
|
||||
|
||||
for attempt in range(3):
|
||||
try:
|
||||
@@ -859,7 +886,14 @@ class OptimizedTelegramScraper:
|
||||
else:
|
||||
return await self.client.get_entity(channel)
|
||||
|
||||
async def scrape_channel(self, channel: str, offset_id: int):
|
||||
async def scrape_channel(self, channel: str, offset_id: int) -> bool:
|
||||
"""Scrape a single channel. Returns True on success, False on failure.
|
||||
|
||||
Offset progress is persisted in a ``finally`` block so partial
|
||||
progress is never lost even when an error occurs mid-scrape.
|
||||
"""
|
||||
last_message_id = offset_id
|
||||
success = False
|
||||
try:
|
||||
if not self.client.is_connected():
|
||||
await self.client.connect()
|
||||
@@ -872,15 +906,70 @@ class OptimizedTelegramScraper:
|
||||
|
||||
if total_messages == 0:
|
||||
print(f"No messages found in channel {channel}")
|
||||
return
|
||||
return True
|
||||
|
||||
print(f"Found {total_messages} messages in channel {channel}")
|
||||
|
||||
message_batch = []
|
||||
media_tasks = []
|
||||
processed_messages = 0
|
||||
last_message_id = offset_id
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
|
||||
media_flush_chunk = 50
|
||||
|
||||
# Media progress counters tracked across chunked flushes so the
|
||||
# progress bar stays coherent even though downloads happen in
|
||||
# bounded chunks during the pass instead of all at the end.
|
||||
total_media = 0
|
||||
completed_media = 0
|
||||
successful_downloads = 0
|
||||
|
||||
async def flush_media_batch():
|
||||
"""Download the accumulated media messages in small batches.
|
||||
|
||||
Keeps memory bounded (we never hold references to every
|
||||
media-capable message for the whole channel), and updates the
|
||||
shared media progress counters via ``nonlocal``.
|
||||
"""
|
||||
nonlocal total_media, completed_media, successful_downloads
|
||||
if not media_tasks:
|
||||
return
|
||||
batch = list(media_tasks)
|
||||
media_tasks.clear()
|
||||
total_media += len(batch)
|
||||
|
||||
async def download_single_media(message):
|
||||
async with semaphore:
|
||||
return await self.download_media(channel, message)
|
||||
|
||||
sub_batch = 10
|
||||
for i in range(0, len(batch), sub_batch):
|
||||
sub = batch[i : i + sub_batch]
|
||||
tasks = [
|
||||
asyncio.create_task(download_single_media(msg)) for msg in sub
|
||||
]
|
||||
for j, task in enumerate(tasks):
|
||||
try:
|
||||
media_path = await task
|
||||
if media_path:
|
||||
await self.update_media_path(
|
||||
channel, sub[j].id, media_path
|
||||
)
|
||||
successful_downloads += 1
|
||||
except Exception:
|
||||
pass
|
||||
completed_media += 1
|
||||
if total_media:
|
||||
mprogress = (completed_media / total_media) * 100
|
||||
bar_length = 30
|
||||
mfilled = int(
|
||||
bar_length * completed_media // total_media
|
||||
)
|
||||
mbar = "█" * mfilled + "░" * (bar_length - mfilled)
|
||||
sys.stdout.write(
|
||||
f"\r📥 Media: [{mbar}] {mprogress:.1f}% "
|
||||
f"({completed_media}/{total_media})"
|
||||
)
|
||||
sys.stdout.flush()
|
||||
|
||||
async for message in self.client.iter_messages(
|
||||
entity, offset_id=offset_id, reverse=True
|
||||
@@ -932,6 +1021,11 @@ class OptimizedTelegramScraper:
|
||||
and not isinstance(message.media, MessageMediaWebPage)
|
||||
):
|
||||
media_tasks.append(message)
|
||||
# Flush the pending media list as soon as it reaches the
|
||||
# bounded chunk so we never hold thousands of message
|
||||
# references in memory for the whole channel.
|
||||
if len(media_tasks) >= media_flush_chunk:
|
||||
await flush_media_batch()
|
||||
|
||||
last_message_id = message.id
|
||||
processed_messages += 1
|
||||
@@ -939,10 +1033,15 @@ class OptimizedTelegramScraper:
|
||||
if len(message_batch) >= self.batch_size:
|
||||
self.batch_insert_messages(channel, message_batch)
|
||||
message_batch.clear()
|
||||
# After each insert batch, also flush any accumulated
|
||||
# media (bounded) rather than deferring everything to
|
||||
# the end of the full pass.
|
||||
if media_tasks:
|
||||
await flush_media_batch()
|
||||
|
||||
if processed_messages % self.state_save_interval == 0:
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
self._save_state_throttled()
|
||||
|
||||
progress = (processed_messages / total_messages) * 100
|
||||
bar_length = 30
|
||||
@@ -963,56 +1062,35 @@ class OptimizedTelegramScraper:
|
||||
self.batch_insert_messages(channel, message_batch)
|
||||
|
||||
if media_tasks:
|
||||
total_media = len(media_tasks)
|
||||
completed_media = 0
|
||||
successful_downloads = 0
|
||||
print(f"\n📥 Downloading {total_media} media files...")
|
||||
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
|
||||
|
||||
async def download_single_media(message):
|
||||
async with semaphore:
|
||||
return await self.download_media(channel, message)
|
||||
|
||||
batch_size = 10
|
||||
for i in range(0, len(media_tasks), batch_size):
|
||||
batch = media_tasks[i : i + batch_size]
|
||||
tasks = [
|
||||
asyncio.create_task(download_single_media(msg)) for msg in batch
|
||||
]
|
||||
|
||||
for j, task in enumerate(tasks):
|
||||
try:
|
||||
media_path = await task
|
||||
if media_path:
|
||||
await self.update_media_path(
|
||||
channel, batch[j].id, media_path
|
||||
)
|
||||
successful_downloads += 1
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
completed_media += 1
|
||||
progress = (completed_media / total_media) * 100
|
||||
bar_length = 30
|
||||
filled_length = int(bar_length * completed_media // total_media)
|
||||
bar = "█" * filled_length + "░" * (bar_length - filled_length)
|
||||
|
||||
sys.stdout.write(
|
||||
f"\r📥 Media: [{bar}] {progress:.1f}% ({completed_media}/{total_media})"
|
||||
)
|
||||
sys.stdout.flush()
|
||||
await flush_media_batch()
|
||||
|
||||
if total_media:
|
||||
print(
|
||||
f"\n✅ Media download complete! ({successful_downloads}/{total_media} successful)"
|
||||
)
|
||||
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
print(f"Completed scraping channel {channel}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error with channel {channel}: {e}")
|
||||
# Final state save moved to ``finally`` below so the offset
|
||||
# persists even when an exception aborts the scrape mid-way.
|
||||
success = True
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error with channel %s", channel)
|
||||
|
||||
finally:
|
||||
# Persist the last-known offset even on partial failure so a
|
||||
# re-scrape resumes from the furthest point reached, not from the
|
||||
# start. save_state() is itself best-effort (logs internally),
|
||||
# so a save failure here must not mask the scrape's own result.
|
||||
try:
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
except Exception:
|
||||
logger.exception("Failed to save state for channel %s", channel)
|
||||
|
||||
return success
|
||||
|
||||
async def rescrape_media(self, channel: str):
|
||||
conn = self.get_db_connection(channel)
|
||||
|
||||
@@ -907,6 +907,343 @@ class TestSecurityHardening:
|
||||
assert "done" in ws.TERMINAL_JOB_STATUSES
|
||||
|
||||
|
||||
class TestParseBool:
|
||||
"""M-3: boolean coercion must never turn 'false'/'0' into True."""
|
||||
|
||||
def _ws(self):
|
||||
import webui_server as ws_module
|
||||
return ws_module
|
||||
|
||||
def test_true_variants(self):
|
||||
pb = self._ws().parse_bool
|
||||
for value in (True, "true", "TRUE", "1", "yes", "on"):
|
||||
assert pb(value) is True, f"{value!r} should be True"
|
||||
|
||||
def test_false_variants(self):
|
||||
pb = self._ws().parse_bool
|
||||
for value in (False, "false", "False", "0", "no", "off"):
|
||||
assert pb(value) is False, f"{value!r} should be False"
|
||||
|
||||
def test_unknown_defaults_to_default(self):
|
||||
pb = self._ws().parse_bool
|
||||
assert pb("garbage") is False
|
||||
assert pb(123) is False
|
||||
assert pb([]) is False
|
||||
assert pb(None) is False
|
||||
assert pb("garbage", default=True) is True
|
||||
|
||||
|
||||
class TestTrustedHost:
|
||||
"""L-8: trusted-host allowlist for DNS-rebinding defence."""
|
||||
|
||||
def _ws(self):
|
||||
import webui_server as ws_module
|
||||
return ws_module
|
||||
|
||||
def test_accepted_hosts(self):
|
||||
ih = self._ws()._is_trusted_host
|
||||
for host in (
|
||||
"localhost", "localhost:8080",
|
||||
"127.0.0.1", "127.0.0.1:8080",
|
||||
"10.0.0.5", "192.168.1.50", "172.16.0.1", "172.31.255.255",
|
||||
):
|
||||
assert ih(host) is True, f"{host!r} should be trusted"
|
||||
|
||||
def test_rejected_hosts(self):
|
||||
ih = self._ws()._is_trusted_host
|
||||
for host in ("evil.example", "example.com", "", " "):
|
||||
assert ih(host) is False, f"{host!r} should NOT be trusted"
|
||||
|
||||
def test_check_same_origin_rejects_untrusted_host_with_origin(self):
|
||||
# DNS-rebinding: attacker sets both Host and Origin to their domain.
|
||||
h = _make_ws_handler(
|
||||
headers={"Host": "evil.example", "Origin": "http://evil.example"}
|
||||
)
|
||||
assert h._check_same_origin() is False
|
||||
h.send_error_json.assert_called_once()
|
||||
|
||||
# But a trusted LAN host with a matching same-origin Origin is allowed.
|
||||
h2 = _make_ws_handler(
|
||||
headers={
|
||||
"Host": "192.168.1.5:8080",
|
||||
"Origin": "http://192.168.1.5:8080",
|
||||
"Sec-Fetch-Site": "same-origin",
|
||||
}
|
||||
)
|
||||
assert h2._check_same_origin() is True
|
||||
|
||||
# Requests WITHOUT an Origin remain allowed (curl / LAN tools).
|
||||
h3 = _make_ws_handler(headers={"Host": "evil.example"})
|
||||
assert h3._check_same_origin() is True
|
||||
|
||||
|
||||
class TestMediaServingLockdown:
|
||||
"""M-1: /media/ must never serve state.json / *.db / *.session."""
|
||||
|
||||
def _make_media_handler(self, relative, data_dir):
|
||||
import webui_server as ws_module
|
||||
self._ws_orig_data = ws_module.DATA_DIR
|
||||
ws_module.DATA_DIR = data_dir
|
||||
handler = object.__new__(ws_module.TelegramScraperRequestHandler)
|
||||
handler.headers = {}
|
||||
handler.rfile = io.BytesIO()
|
||||
handler.wfile = io.BytesIO()
|
||||
handler.path = "/media/" + relative
|
||||
handler.command = "GET"
|
||||
handler.client_address = ("127.0.0.1", 4321)
|
||||
handler.server = MagicMock()
|
||||
handler.send_error_json = MagicMock()
|
||||
return handler
|
||||
|
||||
def _restore_data_dir(self):
|
||||
import webui_server as ws_module
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
def test_rejects_state_json(self):
|
||||
import webui_server as ws_module
|
||||
state_dir = TEST_DATA / "accounts" / "acc1"
|
||||
state_dir.mkdir(parents=True, exist_ok=True)
|
||||
(state_dir / "state.json").write_text('{"api_hash":"secret"}')
|
||||
h = self._make_media_handler("accounts/acc1/state.json", TEST_DATA)
|
||||
self._ws_orig_data = h # no-op placeholder; restored below
|
||||
try:
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
h.serve_media("accounts/acc1/state.json")
|
||||
h.send_error_json.assert_called_once()
|
||||
status = h.send_error_json.call_args[0][0]
|
||||
assert int(status) == 403
|
||||
finally:
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
def test_rejects_db_files(self):
|
||||
import webui_server as ws_module
|
||||
db_file = TEST_DATA / "ch" / "ch.db"
|
||||
db_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
db_file.write_text("sqlite")
|
||||
h = self._make_media_handler("ch/ch.db", TEST_DATA)
|
||||
try:
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
h.serve_media("ch/ch.db")
|
||||
h.send_error_json.assert_called_once()
|
||||
status = h.send_error_json.call_args[0][0]
|
||||
assert int(status) == 403
|
||||
finally:
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
def test_rejects_session_file(self):
|
||||
import webui_server as ws_module
|
||||
h = self._make_media_handler("dummy.session", TEST_DATA)
|
||||
try:
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
h.serve_media("dummy.session")
|
||||
h.send_error_json.assert_called_once()
|
||||
status = h.send_error_json.call_args[0][0]
|
||||
assert int(status) == 403
|
||||
finally:
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
def test_rejects_arbitrary_extension(self):
|
||||
import webui_server as ws_module
|
||||
# A non-media extension (e.g. config) must not be served either.
|
||||
f = TEST_DATA / "config.yaml"
|
||||
f.write_text("x")
|
||||
h = self._make_media_handler("config.yaml", TEST_DATA)
|
||||
try:
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
h.serve_media("config.yaml")
|
||||
h.send_error_json.assert_called_once()
|
||||
status = h.send_error_json.call_args[0][0]
|
||||
assert int(status) == 403
|
||||
finally:
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
def test_serves_normal_media_extension(self):
|
||||
import webui_server as ws_module
|
||||
# A normal media file should reach serve_file (not be denied). We
|
||||
# verify serve_file is reached by checking the FORBIDDEN path is NOT
|
||||
# taken (no send_error_json) and that a media file resolves.
|
||||
media_dir = TEST_DATA / "accounts" / "acc1" / "ch" / "media"
|
||||
media_dir.mkdir(parents=True, exist_ok=True)
|
||||
img = media_dir / "1-photo.jpg"
|
||||
img.write_bytes(b"\xff\xd8\xff\xe0")
|
||||
h = self._make_media_handler("accounts/acc1/ch/media/1-photo.jpg", TEST_DATA)
|
||||
# Stub serve_file so we can assert it is invoked for a media file.
|
||||
h.serve_file = MagicMock()
|
||||
try:
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
h.serve_media("accounts/acc1/ch/media/1-photo.jpg")
|
||||
h.send_error_json.assert_not_called()
|
||||
h.serve_file.assert_called_once()
|
||||
finally:
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
|
||||
|
||||
class TestRangeEdgeCases:
|
||||
"""M-8: _parse_range must never throw; empty-file Range ignored."""
|
||||
|
||||
def _ws(self):
|
||||
import webui_server as ws_module
|
||||
return ws_module
|
||||
|
||||
def test_empty_file_range_ignored(self):
|
||||
ws = self._ws()
|
||||
# A size-0 file with bytes=0-0 should not 416 — serve_file treats empty
|
||||
# files as a full 200. We verify _parse_range is bypassed for size 0 by
|
||||
# checking the serve_file logic path (header treated as no-range).
|
||||
# Simulate a handler.
|
||||
h = object.__new__(ws.TelegramScraperRequestHandler)
|
||||
h.headers = {"Range": "bytes=0-0"}
|
||||
h.send_error_json = MagicMock()
|
||||
h.send_response = MagicMock()
|
||||
h.send_header = MagicMock()
|
||||
h.end_headers = MagicMock()
|
||||
h._write_file_range = MagicMock()
|
||||
h.wfile = io.BytesIO()
|
||||
tmp = TEST_DATA / "range-empty.bin"
|
||||
tmp.write_bytes(b"")
|
||||
# Empty file: Range must be ignored, so send_response is called with OK.
|
||||
h.serve_file(tmp, "application/octet-stream")
|
||||
calls = [c.args[0] for c in h.send_response.call_args_list]
|
||||
assert 200 in calls, f"expected a 200 for empty file, got {calls}"
|
||||
|
||||
def test_multi_range_returns_416_not_crash(self):
|
||||
ws = self._ws()
|
||||
h = object.__new__(ws.TelegramScraperRequestHandler)
|
||||
h.headers = {"Range": "bytes=0-1,5-6"}
|
||||
h.send_error_json = MagicMock()
|
||||
tmp = TEST_DATA / "range-multi.bin"
|
||||
tmp.write_bytes(b"0123456789")
|
||||
h.serve_file(tmp, "application/octet-stream")
|
||||
h.send_error_json.assert_called_once()
|
||||
status = h.send_error_json.call_args[0][0]
|
||||
assert int(status) == 416
|
||||
|
||||
def test_valid_single_range_still_works(self):
|
||||
ws = self._ws()
|
||||
h = object.__new__(ws.TelegramScraperRequestHandler)
|
||||
h.headers = {"Range": "bytes=0-3"}
|
||||
h.send_error_json = MagicMock()
|
||||
h.send_response = MagicMock()
|
||||
h.send_header = MagicMock()
|
||||
h.end_headers = MagicMock()
|
||||
h._write_file_range = MagicMock()
|
||||
h.wfile = io.BytesIO()
|
||||
tmp = TEST_DATA / "range-valid.bin"
|
||||
tmp.write_bytes(b"0123456789")
|
||||
h.serve_file(tmp, "application/octet-stream")
|
||||
calls = [c.args[0] for c in h.send_response.call_args_list]
|
||||
assert 206 in calls, f"expected 206 for valid range, got {calls}"
|
||||
h.send_error_json.assert_not_called()
|
||||
|
||||
|
||||
class TestJobShutdownDrains:
|
||||
"""M-7: JobRunner.shutdown must drain queued jobs to failed."""
|
||||
|
||||
def test_shutdown_marks_queued_jobs_failed(self):
|
||||
import webui_server as ws_module
|
||||
runner = ws_module.JobRunner()
|
||||
# Stop the real worker thread immediately so queued jobs remain queued.
|
||||
runner._shutdown_flag = True
|
||||
# Create jobs directly into the queue (bypass create_job dedup).
|
||||
j1 = ws_module.Job("job-1", "scrape_all", "A", {})
|
||||
j2 = ws_module.Job("job-2", "scrape_all", "B", {})
|
||||
with runner.lock:
|
||||
runner.jobs[j1.job_id] = j1
|
||||
runner.jobs[j2.job_id] = j2
|
||||
runner.job_order = [j1.job_id, j2.job_id]
|
||||
runner.queue.put(j1)
|
||||
runner.queue.put(j2)
|
||||
runner.shutdown(timeout=0)
|
||||
assert j1.status == "failed"
|
||||
assert j2.status == "failed"
|
||||
assert "Server shutting down" in (j1.error or "")
|
||||
# The queue must now be empty.
|
||||
assert runner.queue.empty()
|
||||
|
||||
|
||||
class TestContinuousRestartDuringDrain:
|
||||
"""F-3 + F-4: manager restart / remove-account tombstone safety."""
|
||||
|
||||
def _setup_ws_data_dir(self):
|
||||
import webui_server as ws_module
|
||||
self._ws_orig_data = ws_module.DATA_DIR
|
||||
self._ws_orig_session = ws_module.SESSION_DIR
|
||||
ws_module.DATA_DIR = TEST_DATA
|
||||
ws_module.SESSION_DIR = TEST_SESSION
|
||||
ws_module.START_CONTINUOUS = False
|
||||
|
||||
def _restore_ws_data_dir(self):
|
||||
import webui_server as ws_module
|
||||
ws_module.DATA_DIR = self._ws_orig_data
|
||||
ws_module.SESSION_DIR = self._ws_orig_session
|
||||
|
||||
def test_start_during_drain_waits_and_restarts(self):
|
||||
"""F-3: start() while the old thread is draining must wait for it to
|
||||
exit, then spawn a fresh thread (not leave a dead-but-enabled state)."""
|
||||
import webui_server as ws_module
|
||||
PerAccountContinuousScrapeManager = ws_module.PerAccountContinuousScrapeManager
|
||||
self._setup_ws_data_dir()
|
||||
try:
|
||||
aid = make_account_id()
|
||||
create_account(TEST_DATA, aid, continuous_scraping={
|
||||
"enabled": True, "interval_minutes": 60, "channels": [], "run_all_tracked": True,
|
||||
})
|
||||
mgr = PerAccountContinuousScrapeManager(aid)
|
||||
mgr.refresh_config = lambda: None
|
||||
mgr.start()
|
||||
t1 = mgr.thread
|
||||
assert t1 is not None and t1.is_alive()
|
||||
|
||||
# Request a stop (drain) and call start() again while the old
|
||||
# thread is still alive (its finally has not flipped running yet).
|
||||
mgr.stop()
|
||||
mgr.start() # must wait for t1 to drain, then start a fresh one
|
||||
t2 = mgr.thread
|
||||
assert t2 is not None and t2.is_alive()
|
||||
assert t2 is not t1 or t2 is t1 # either reused thread object or a new one
|
||||
# Exactly one worker may be alive at a time: old must be dead.
|
||||
assert not t1.is_alive() or t2 is t1
|
||||
|
||||
# Shut it down cleanly.
|
||||
mgr.stop()
|
||||
mgr.join(timeout=5.0)
|
||||
finally:
|
||||
self._restore_ws_data_dir()
|
||||
|
||||
def test_remove_account_leaves_tombstone_when_join_times_out(self):
|
||||
"""F-4: remove_account must NOT pop the manager when the join times
|
||||
out; it keeps a tombstone so a re-add reuses it instead of spawning a
|
||||
duplicate worker."""
|
||||
import webui_server as ws_module
|
||||
_, ContinuousScrapeOrchestrator, _ = (
|
||||
ws_module.PerAccountContinuousScrapeManager,
|
||||
ws_module.ContinuousScrapeOrchestrator,
|
||||
ws_module,
|
||||
)
|
||||
self._setup_ws_data_dir()
|
||||
try:
|
||||
orch = ContinuousScrapeOrchestrator()
|
||||
aid = make_account_id()
|
||||
create_account(TEST_DATA, aid, continuous_scraping={
|
||||
"enabled": False, "interval_minutes": 1, "channels": [], "run_all_tracked": True,
|
||||
})
|
||||
mgr = orch._get_or_create(aid)
|
||||
# Override join to return False (simulate a still-running thread).
|
||||
original_join = mgr.join
|
||||
mgr.join = lambda timeout=None: False
|
||||
orch.remove_account(aid)
|
||||
assert aid in orch.managers, "manager should be kept as tombstone"
|
||||
assert orch.managers[aid]._removing is True
|
||||
mgr.join = original_join
|
||||
# Re-add the account: _get_or_create clears the tombstone and reuses
|
||||
# the same manager object (no duplicate worker created).
|
||||
mgr2 = orch._get_or_create(aid)
|
||||
assert mgr2 is mgr, "re-add must reuse the tombstone manager"
|
||||
assert mgr2._removing is False
|
||||
finally:
|
||||
self._restore_ws_data_dir()
|
||||
|
||||
|
||||
# ── Cleanup all temp data ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
+8
-1
@@ -204,10 +204,16 @@ function renderAccountPanel(accountId) {
|
||||
if (!runAllTracked && channels.length === 0 && enabled) {
|
||||
if (!confirmAction('Continuous scraping enabled with no selected channels. Save anyway?')) return;
|
||||
}
|
||||
await api(`/api/accounts/${accountId}/continuous`, {
|
||||
const resp = await api(`/api/accounts/${accountId}/continuous`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ enabled, interval_minutes: intervalMinutes, run_all_tracked: runAllTracked, channels }),
|
||||
});
|
||||
if (resp.dropped_invalid && resp.dropped_invalid.length) {
|
||||
const n = resp.dropped_invalid.length;
|
||||
const shown = resp.dropped_invalid.slice(0, 3).join(', ');
|
||||
const extra = n > 3 ? '…' : '';
|
||||
showToast(`${n} invalid channel(s) skipped: ${shown}${extra}`, 'warn');
|
||||
}
|
||||
await refreshAccount(accountId);
|
||||
});
|
||||
|
||||
@@ -956,6 +962,7 @@ async function main() {
|
||||
await loadAccounts();
|
||||
switchAccount(accountId);
|
||||
showToast(`Imported ${accountId}.`, 'success');
|
||||
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
|
||||
} catch (err) {
|
||||
showToast(`Failed to import account: ${err.message}`, 'error');
|
||||
}
|
||||
|
||||
@@ -260,6 +260,7 @@ async function importAccount(event) {
|
||||
await loadAccounts();
|
||||
await loadAccount(accountId);
|
||||
showToast(`Imported ${accountId}.`, 'success');
|
||||
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
|
||||
} catch (err) {
|
||||
showToast(`Failed to import account: ${err.message}`, 'error');
|
||||
}
|
||||
|
||||
+6
-1
@@ -49,5 +49,10 @@ loadSpec()
|
||||
.then(renderSpec)
|
||||
.catch((error) => {
|
||||
console.error(error);
|
||||
document.getElementById('api-docs').innerHTML = `<section class="panel">${error.message}</section>`;
|
||||
const docsEl = document.getElementById('api-docs');
|
||||
docsEl.textContent = '';
|
||||
const section = document.createElement('section');
|
||||
section.className = 'panel';
|
||||
section.textContent = error.message;
|
||||
docsEl.appendChild(section);
|
||||
});
|
||||
|
||||
+323
-53
@@ -91,6 +91,74 @@ AUTH_ATTEMPTS_MAX_ENTRIES = 10_000
|
||||
_auth_attempts: Dict[Tuple[str, str], Dict[str, Any]] = {}
|
||||
_auth_attempts_lock = threading.Lock()
|
||||
|
||||
# ── M-3: boolean coercion helper ─────────────────────────────────────────
|
||||
_TRUE_VALUES = {"true", "1", "yes", "on"}
|
||||
_FALSE_VALUES = {"false", "0", "no", "off"}
|
||||
|
||||
|
||||
def parse_bool(value: Any, default: bool = False) -> bool:
|
||||
"""Coerce *value* to bool safely. Returns *default* for unrecognised input.
|
||||
|
||||
Recognises True/False, ``"true"``/``"false"``, ``"1"``/``"0"``,
|
||||
``"yes"``/``"no"``, ``"on"``/``"off"`` (case-insensitive). Strings
|
||||
outside these sets return *default* instead of silently being truthy.
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value is None:
|
||||
return default
|
||||
s = str(value).strip().lower()
|
||||
if s in _TRUE_VALUES:
|
||||
return True
|
||||
if s in _FALSE_VALUES:
|
||||
return False
|
||||
return default
|
||||
|
||||
|
||||
# ── L-8: trusted-host check ──────────────────────────────────────────────
|
||||
import ipaddress # noqa: E402
|
||||
|
||||
|
||||
def _is_trusted_host(host: str) -> bool:
|
||||
"""Return True if *host* (the ``Host`` header value) is a loopback /
|
||||
private address that this local-only deployment should trust."""
|
||||
hostname = host.split("@")[-1].split(":")[0] # strip auth / port
|
||||
if not hostname:
|
||||
return False
|
||||
if hostname in {"localhost", "127.0.0.1", "::1"}:
|
||||
return True
|
||||
try:
|
||||
addr = ipaddress.ip_address(hostname)
|
||||
return addr.is_loopback or addr.is_private or addr.is_link_local
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
# ── M-1: sensitive file / media extension allowlists ──────────────────────
|
||||
_SENSITIVE_FILE_SUFFIXES = frozenset({
|
||||
".db", ".session", ".db-wal", ".db-shm", ".db-journal",
|
||||
})
|
||||
_SENSITIVE_FILE_NAMES = frozenset({"state.json"})
|
||||
|
||||
# How long a generated QR login token/image is considered valid before it is
|
||||
# dropped from the auth snapshot (seconds). L-5.
|
||||
QR_TTL_SECONDS = 60
|
||||
|
||||
_MEDIA_FILE_EXTENSIONS = frozenset({
|
||||
# images
|
||||
".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".svg", ".ico",
|
||||
# video
|
||||
".mp4", ".webm", ".mov", ".m4v", ".avi",
|
||||
# audio
|
||||
".mp3", ".ogg", ".wav", ".m4a", ".flac",
|
||||
# documents / generic downloaded binary
|
||||
".pdf", ".doc", ".docx", ".xls", ".xlsx", ".ppt", ".pptx",
|
||||
".txt", ".csv", ".json", ".xml", ".bin",
|
||||
# archives / other Telegram document types
|
||||
".zip", ".rar", ".7z", ".apk", ".epub", ".tar", ".gz", ".bz2", ".xz",
|
||||
".odt", ".ods", ".odp",
|
||||
})
|
||||
|
||||
|
||||
def _sweep_auth_attempts(now: float) -> None:
|
||||
"""Evict expired lockout/cooldown entries when the dict grows too large."""
|
||||
@@ -589,6 +657,23 @@ class JobRunner:
|
||||
len(still_running),
|
||||
timeout,
|
||||
)
|
||||
# M-7: The worker exits once _shutdown_flag is set, leaving any yet-to-be
|
||||
# processed queued jobs stranded as "queued". Drain the queue and mark
|
||||
# each leftover as failed so their event streams / polling terminate.
|
||||
drained = 0
|
||||
while True:
|
||||
try:
|
||||
job = self.queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
self.queue.task_done()
|
||||
with self.lock:
|
||||
job.status = "failed"
|
||||
job.error = "Server shutting down; job was cancelled"
|
||||
job.finished_at = utc_now_iso()
|
||||
drained += 1
|
||||
if drained:
|
||||
logger.info("Failed %d queued job(s) during shutdown.", drained)
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._shutdown_flag:
|
||||
@@ -653,6 +738,7 @@ class TelegramAuthManager:
|
||||
"details": "",
|
||||
"qr_url": None,
|
||||
"qr_image": None,
|
||||
"qr_created_at": None,
|
||||
"phone": None,
|
||||
"phone_code_hash": None,
|
||||
"qr_login": None,
|
||||
@@ -713,6 +799,16 @@ class TelegramAuthManager:
|
||||
with self.lock:
|
||||
data = self._get_auth_data(account_id)
|
||||
snapshot = dict(data)
|
||||
# L-5: expire a QR login token that has not been scanned within
|
||||
# QR_TTL_SECONDS so it cannot live forever in the auth snapshot.
|
||||
qr_created = data.get("qr_created_at")
|
||||
if qr_created and (time.time() - float(qr_created)) > QR_TTL_SECONDS:
|
||||
if data.get("qr_url") or data.get("qr_image"):
|
||||
data["qr_url"] = None
|
||||
data["qr_image"] = None
|
||||
data["qr_created_at"] = None
|
||||
snapshot = dict(data)
|
||||
snapshot["qr_expired"] = True
|
||||
snapshot.pop("qr_login", None)
|
||||
snapshot.pop("qr_wait_task", None)
|
||||
snapshot.pop("phone_code_hash", None)
|
||||
@@ -768,6 +864,7 @@ class TelegramAuthManager:
|
||||
details="Scan the QR code in Telegram: Settings -> Devices -> Scan QR.",
|
||||
qr_url=qr_url,
|
||||
qr_image=self._make_qr_image(qr_url),
|
||||
qr_created_at=time.time(),
|
||||
)
|
||||
with self.lock:
|
||||
data = self._get_auth_data(account_id)
|
||||
@@ -792,6 +889,7 @@ class TelegramAuthManager:
|
||||
details="Telegram session is authorized.",
|
||||
qr_url=None,
|
||||
qr_image=None,
|
||||
qr_created_at=None,
|
||||
phone=None,
|
||||
)
|
||||
except SessionPasswordNeededError:
|
||||
@@ -809,6 +907,7 @@ class TelegramAuthManager:
|
||||
details=f"QR login failed: {exc}",
|
||||
qr_url=None,
|
||||
qr_image=None,
|
||||
qr_created_at=None,
|
||||
)
|
||||
|
||||
def start_qr_login(self, account_id: str) -> Dict[str, Any]:
|
||||
@@ -859,6 +958,7 @@ class TelegramAuthManager:
|
||||
details="Telegram session is authorized.",
|
||||
qr_url=None,
|
||||
qr_image=None,
|
||||
qr_created_at=None,
|
||||
)
|
||||
except SessionPasswordNeededError:
|
||||
self._set_state(
|
||||
@@ -882,6 +982,7 @@ class TelegramAuthManager:
|
||||
details="Telegram session is authorized.",
|
||||
qr_url=None,
|
||||
qr_image=None,
|
||||
qr_created_at=None,
|
||||
)
|
||||
return self.auth_state(account_id)
|
||||
|
||||
@@ -937,6 +1038,10 @@ class PerAccountContinuousScrapeManager:
|
||||
self.lock = threading.RLock()
|
||||
self.thread: Optional[threading.Thread] = None
|
||||
self.stop_event = threading.Event()
|
||||
# F-4: set on remove_account when its thread could not be joined in
|
||||
# time; the manager is kept as a tombstone so it is reused (after the
|
||||
# drain) instead of spawning a duplicate worker on account re-add.
|
||||
self._removing = False
|
||||
self.config: Dict[str, Any] = self._load_config()
|
||||
self.status: Dict[str, Any] = {
|
||||
"running": False,
|
||||
@@ -1053,9 +1158,34 @@ class PerAccountContinuousScrapeManager:
|
||||
|
||||
def start(self) -> None:
|
||||
with self.lock:
|
||||
if self.status["running"]:
|
||||
thread = self.thread
|
||||
if thread is not None and thread.is_alive():
|
||||
if self.stop_event.is_set():
|
||||
# F-3: a stop was requested and the old thread is still
|
||||
# draining (its finally has not yet flipped running=False).
|
||||
# Wait for it to exit before starting a fresh one so the
|
||||
# account does not end up enabled=True with a dead thread.
|
||||
self._log(
|
||||
"Previous scrape thread is stopping; waiting before restart.",
|
||||
"warn",
|
||||
)
|
||||
else:
|
||||
# A live thread is genuinely running — do NOT spawn a
|
||||
# duplicate.
|
||||
self._log("Continuous scraping is already running.", "warn")
|
||||
return
|
||||
if thread is not None and thread.is_alive():
|
||||
# Join outside the lock (bounded) so the draining thread can mark
|
||||
# itself finished. If it does not exit in time, refuse to start a
|
||||
# duplicate worker rather than risk overlapping writes.
|
||||
thread.join(timeout=5.0)
|
||||
if thread.is_alive():
|
||||
self._log(
|
||||
"Previous scrape thread still stopping; cannot restart yet.",
|
||||
"error",
|
||||
)
|
||||
return
|
||||
with self.lock:
|
||||
self.stop_event.clear()
|
||||
self.status["running"] = True
|
||||
self.status["last_started_at"] = utc_now_iso()
|
||||
@@ -1180,9 +1310,18 @@ class ContinuousScrapeOrchestrator:
|
||||
|
||||
def _get_or_create(self, account_id: str) -> PerAccountContinuousScrapeManager:
|
||||
with self.lock:
|
||||
if account_id not in self.managers:
|
||||
self.managers[account_id] = PerAccountContinuousScrapeManager(account_id)
|
||||
return self.managers[account_id]
|
||||
mgr = self.managers.get(account_id)
|
||||
if mgr is None:
|
||||
mgr = PerAccountContinuousScrapeManager(account_id)
|
||||
self.managers[account_id] = mgr
|
||||
elif mgr._removing:
|
||||
# F-4: the manager was left as a tombstone because its worker
|
||||
# could not be joined during remove_account. The account is
|
||||
# being re-added / accessed, so clear the tombstone and reuse
|
||||
# the manager (start() waits for the old thread to drain before
|
||||
# spawning a fresh one, avoiding duplicate workers).
|
||||
mgr._removing = False
|
||||
return mgr
|
||||
|
||||
def start_account(self, account_id: str) -> None:
|
||||
mgr = self._get_or_create(account_id)
|
||||
@@ -1240,18 +1379,24 @@ class ContinuousScrapeOrchestrator:
|
||||
mgr.stop()
|
||||
# Wait for the scrape thread to actually stop before the caller
|
||||
# deletes the account directory / session files, so rmtree does not
|
||||
# race with a writer mid-iteration. If the thread is still running
|
||||
# (e.g. mid-scrape) after the timeout we proceed best-effort and
|
||||
# log a warning.
|
||||
if not mgr.join(timeout=REMOVE_ACCOUNT_JOIN_TIMEOUT):
|
||||
# race with a writer mid-iteration.
|
||||
if mgr.join(timeout=REMOVE_ACCOUNT_JOIN_TIMEOUT):
|
||||
# Thread finished: safe to drop the manager.
|
||||
with self.lock:
|
||||
self.managers.pop(account_id, None)
|
||||
else:
|
||||
# F-4: the thread is still draining. Do NOT pop the manager —
|
||||
# leave it as a tombstone (marked removing) so a later re-add of
|
||||
# the same account id reuses it / waits for the drain instead of
|
||||
# spawning a duplicate worker that writes the same dirs.
|
||||
logger.warning(
|
||||
"Continuous scrape thread for account %r still running after "
|
||||
"%.1fs; removing account data best-effort",
|
||||
"%.1fs; keeping manager as tombstone",
|
||||
account_id,
|
||||
REMOVE_ACCOUNT_JOIN_TIMEOUT,
|
||||
)
|
||||
with self.lock:
|
||||
self.managers.pop(account_id, None)
|
||||
mgr._removing = True
|
||||
|
||||
def stop_all(self) -> None:
|
||||
with self.lock:
|
||||
@@ -1336,18 +1481,31 @@ def auth_status() -> Dict[str, Any]:
|
||||
return auth_status_for(account_id=None)
|
||||
|
||||
|
||||
def dashboard_payload(job_runner: JobRunner) -> Dict[str, Any]:
|
||||
def dashboard_payload(job_runner: JobRunner, account_id: Optional[str] = None) -> Dict[str, Any]:
|
||||
if account_id:
|
||||
acc_state = load_account(DATA_DIR, account_id)
|
||||
state = {
|
||||
"scrape_media": bool(acc_state.get("scrape_media", True)),
|
||||
"channel_count": len(acc_state.get("channels", {})),
|
||||
"forwarding_rules": acc_state.get("forwarding_rules", []),
|
||||
}
|
||||
channels = list_channels_snapshot(account_id)
|
||||
auth = auth_status_for(account_id)
|
||||
jobs = job_runner.recent_jobs(account_id=account_id)
|
||||
else:
|
||||
state = load_state()
|
||||
channels = list_channels_snapshot()
|
||||
auth = auth_status()
|
||||
jobs = job_runner.recent_jobs()
|
||||
return {
|
||||
"state": {
|
||||
"scrape_media": bool(state.get("scrape_media", True)),
|
||||
"channel_count": len(state.get("channels", {})),
|
||||
"forwarding_rules": state.get("forwarding_rules", []),
|
||||
},
|
||||
"auth": auth_status(),
|
||||
"auth": auth,
|
||||
"channels": channels,
|
||||
"jobs": job_runner.recent_jobs(),
|
||||
"jobs": jobs,
|
||||
}
|
||||
|
||||
|
||||
@@ -2050,7 +2208,9 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
if path == "/openapi.json":
|
||||
return self.send_json(openapi_payload())
|
||||
if path == "/api/dashboard":
|
||||
return self.send_json(dashboard_payload(self.app.job_runner))
|
||||
return self.send_json(
|
||||
dashboard_payload(self.app.job_runner, self.app.legacy_account_id)
|
||||
)
|
||||
if path == "/api/auth":
|
||||
if self.app.legacy_account_id:
|
||||
return self.send_json(self.app.auth_manager.auth_state(self.app.legacy_account_id))
|
||||
@@ -2094,27 +2254,34 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
return self.send_error_json(HTTPStatus.NOT_FOUND, "Job not found")
|
||||
return self.send_json(job)
|
||||
if path == "/api/channels":
|
||||
return self.send_json(list_channels_snapshot())
|
||||
return self.send_json(list_channels_snapshot(self.app.legacy_account_id))
|
||||
if path.startswith("/api/channels/") and path.endswith("/messages"):
|
||||
parts = path.split("/")
|
||||
try:
|
||||
channel_id = normalize_channel_id(urllib.parse.unquote(parts[3]))
|
||||
except ValueError as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
try:
|
||||
limit = max(1, min(int(query.get("limit", ["120"])[0]), 300))
|
||||
except (TypeError, ValueError):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "Invalid 'limit' query parameter")
|
||||
before = query.get("before")
|
||||
try:
|
||||
before_message_id = int(before[0]) if before else None
|
||||
except (TypeError, ValueError):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "Invalid 'before' query parameter")
|
||||
search = (query.get("search") or query.get("q") or [""])[0].strip()
|
||||
account_id = self.app.legacy_account_id
|
||||
payload = {
|
||||
"channel_id": channel_id,
|
||||
"messages": load_messages(
|
||||
None, channel_id, limit=limit, before_message_id=before_message_id,
|
||||
account_id, channel_id, limit=limit, before_message_id=before_message_id,
|
||||
search=search or None
|
||||
),
|
||||
"channel": next(
|
||||
(
|
||||
item
|
||||
for item in list_channels_snapshot()
|
||||
for item in list_channels_snapshot(account_id)
|
||||
if item["channel_id"] == channel_id
|
||||
),
|
||||
None,
|
||||
@@ -2258,9 +2425,15 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
def _handle_get_account_channel_messages(
|
||||
self, account_id: str, channel_id: str, query: Dict[str, List[str]]
|
||||
) -> None:
|
||||
try:
|
||||
limit = max(1, min(int(query.get("limit", ["120"])[0]), 300))
|
||||
except (TypeError, ValueError):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "Invalid 'limit' query parameter")
|
||||
before = query.get("before")
|
||||
try:
|
||||
before_message_id = int(before[0]) if before else None
|
||||
except (TypeError, ValueError):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "Invalid 'before' query parameter")
|
||||
search = (query.get("search") or query.get("q") or [""])[0].strip()
|
||||
payload = {
|
||||
"channel_id": channel_id,
|
||||
@@ -2316,6 +2489,12 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
or (path.startswith("/api/channels/") and path.endswith("/messages"))
|
||||
or path.startswith("/api/accounts/")
|
||||
):
|
||||
# M-4: for the job-events route, mirror GET's behavior and verify
|
||||
# the job actually exists before responding with 200.
|
||||
if path.startswith("/api/jobs/") and path.endswith("/events"):
|
||||
job_id = path.split("/")[-2] if path.endswith("/events") else None
|
||||
if job_id and not self.app.job_runner.get_job(job_id):
|
||||
return self.send_error_json(HTTPStatus.NOT_FOUND, "Job not found")
|
||||
self.send_response(HTTPStatus.OK)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.end_headers()
|
||||
@@ -2339,22 +2518,23 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
|
||||
# ── Legacy endpoints ────────────────────────────────────────────
|
||||
if path == "/api/settings/media":
|
||||
value = bool(body.get("value"))
|
||||
value = parse_bool(body.get("value"))
|
||||
if self.app.legacy_account_id:
|
||||
return self._handle_account_settings_media(self.app.legacy_account_id, value)
|
||||
job = self.app.job_runner.create_job(
|
||||
"set_scrape_media",
|
||||
"Update media scraping setting",
|
||||
{"value": value},
|
||||
)
|
||||
return self.send_json(job.to_dict(), status=HTTPStatus.ACCEPTED)
|
||||
# No legacy account: persist to the (global) legacy store directly.
|
||||
# The ScraperJobService set_scrape_media job is a passthrough, so
|
||||
# enqueueing it alone would silently drop the setting.
|
||||
def _media_mutate(global_state: Dict[str, Any]) -> None:
|
||||
global_state["scrape_media"] = value
|
||||
STATE_STORE.update(_media_mutate)
|
||||
return self.send_json({"ok": True, "scrape_media": value})
|
||||
|
||||
if path == "/api/continuous":
|
||||
try:
|
||||
enabled = bool(body.get("enabled"))
|
||||
enabled = parse_bool(body.get("enabled"))
|
||||
interval_minutes = int(body.get("interval_minutes", 1))
|
||||
channels, dropped = clean_continuous_channels(body.get("channels", []))
|
||||
run_all_tracked = bool(body.get("run_all_tracked", True))
|
||||
run_all_tracked = parse_bool(body.get("run_all_tracked", True), default=True)
|
||||
if self.app.legacy_account_id:
|
||||
payload = self.app.continuous_orchestrator.update_for(
|
||||
account_id=self.app.legacy_account_id,
|
||||
@@ -2366,7 +2546,9 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
else:
|
||||
payload = {"config": {}, "status": {"running": False, "logs": [], "log_entries": []}}
|
||||
except Exception as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(
|
||||
exc, fallback="Failed to update continuous scraping"
|
||||
)
|
||||
if isinstance(payload, dict):
|
||||
payload["dropped_invalid"] = dropped
|
||||
return self.send_json(payload)
|
||||
@@ -2383,7 +2565,9 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
save_state(state)
|
||||
payload = self._legacy_auth_snapshot()
|
||||
except Exception as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(
|
||||
exc, fallback="Failed to save API credentials"
|
||||
)
|
||||
return self.send_json(payload)
|
||||
|
||||
if path == "/api/auth/qr/start":
|
||||
@@ -2405,12 +2589,13 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
if self.app.legacy_account_id:
|
||||
return self._handle_account_channel_add(self.app.legacy_account_id, channel_id, body.get("name"))
|
||||
state = load_state()
|
||||
if channel_id not in state["channels"]:
|
||||
state["channels"][channel_id] = 0
|
||||
# L-6: use the atomic mutator to avoid lost-update races.
|
||||
def _add_mutate(state: Dict[str, Any]) -> None:
|
||||
if channel_id not in state.get("channels", {}):
|
||||
state.setdefault("channels", {})[channel_id] = 0
|
||||
if body.get("name"):
|
||||
state.setdefault("channel_names", {})[channel_id] = str(body["name"]).strip()
|
||||
save_state(state)
|
||||
STATE_STORE.update(_add_mutate)
|
||||
return self.send_json({"ok": True, "channel_id": channel_id})
|
||||
|
||||
if path == "/api/channels/remove":
|
||||
@@ -2420,11 +2605,16 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
if self.app.legacy_account_id:
|
||||
return self._handle_account_channel_remove(self.app.legacy_account_id, channel_id)
|
||||
state = load_state()
|
||||
existed = channel_id in state.get("channels", {})
|
||||
state.get("channels", {}).pop(channel_id, None)
|
||||
save_state(state)
|
||||
return self.send_json({"ok": existed, "channel_id": channel_id})
|
||||
# L-6: use the atomic mutator to avoid lost-update races.
|
||||
existed = [False]
|
||||
def _remove_mutate(state: Dict[str, Any]) -> None:
|
||||
chans = state.get("channels", {})
|
||||
if channel_id in chans:
|
||||
existed[0] = True
|
||||
del chans[channel_id]
|
||||
state.get("channel_names", {}).pop(channel_id, None)
|
||||
STATE_STORE.update(_remove_mutate)
|
||||
return self.send_json({"ok": existed[0], "channel_id": channel_id})
|
||||
|
||||
if path == "/api/jobs/scrape":
|
||||
channel_id = body.get("channel_id")
|
||||
@@ -2581,19 +2771,30 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
if not isinstance(state, dict):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "state object is required")
|
||||
|
||||
# F-1: normalise the imported continuous channels and drop invalid
|
||||
# entries (path traversal / control chars) before persisting.
|
||||
cs = state.get("continuous_scraping")
|
||||
cs = cs if isinstance(cs, dict) else {}
|
||||
cs_channels, _dropped = clean_continuous_channels(cs.get("channels", []))
|
||||
|
||||
try:
|
||||
interval_minutes = int(cs.get("interval_minutes", 1) or 1)
|
||||
except (TypeError, ValueError):
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, "interval_minutes must be a valid integer")
|
||||
|
||||
imported_state = {
|
||||
"label": str(body.get("label") or state.get("label") or account_id).strip(),
|
||||
"api_id": state.get("api_id"),
|
||||
"api_hash": state.get("api_hash"),
|
||||
"channels": self._clean_imported_channels(state.get("channels")),
|
||||
"channel_names": self._clean_imported_channel_names(state.get("channel_names")),
|
||||
"scrape_media": bool(state.get("scrape_media", True)),
|
||||
"scrape_media": parse_bool(state.get("scrape_media", True), default=True),
|
||||
"forwarding_rules": state.get("forwarding_rules") if isinstance(state.get("forwarding_rules"), list) else [],
|
||||
"continuous_scraping": state.get("continuous_scraping") if isinstance(state.get("continuous_scraping"), dict) else {
|
||||
"enabled": False,
|
||||
"interval_minutes": 1,
|
||||
"channels": [],
|
||||
"run_all_tracked": True,
|
||||
"continuous_scraping": {
|
||||
"enabled": parse_bool(cs.get("enabled", False)),
|
||||
"interval_minutes": interval_minutes,
|
||||
"channels": cs_channels,
|
||||
"run_all_tracked": parse_bool(cs.get("run_all_tracked", True), default=True),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -2645,7 +2846,7 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
if sub == ["jobs", "refresh-dialogs"]:
|
||||
return self._handle_account_job_refresh_dialogs(account_id)
|
||||
if sub == ["settings", "media"]:
|
||||
return self._handle_account_settings_media(account_id, bool(body.get("value")))
|
||||
return self._handle_account_settings_media(account_id, parse_bool(body.get("value")))
|
||||
if sub == ["continuous"]:
|
||||
return self._handle_account_continuous(account_id, body)
|
||||
return self.send_error_json(HTTPStatus.NOT_FOUND, "Not found")
|
||||
@@ -2664,14 +2865,14 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
try:
|
||||
payload = self.app.auth_manager.save_credentials(account_id, int(api_id), api_hash)
|
||||
except Exception as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to save API credentials")
|
||||
return self.send_json(payload)
|
||||
|
||||
def _handle_post_account_auth_qr_start(self, account_id: str) -> None:
|
||||
try:
|
||||
payload = self.app.auth_manager.start_qr_login(account_id)
|
||||
except Exception as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to start QR login")
|
||||
return self.send_json(payload)
|
||||
|
||||
def _handle_post_account_auth_phone_request(self, account_id: str, body: Dict[str, Any]) -> None:
|
||||
@@ -2693,7 +2894,7 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
payload = self.app.auth_manager.request_phone_code(account_id, phone)
|
||||
except Exception as exc:
|
||||
_record_auth_failure(ip, account_id)
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to request phone code")
|
||||
_record_auth_code_request(ip, account_id)
|
||||
return self.send_json(payload)
|
||||
|
||||
@@ -2711,7 +2912,7 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
payload = self.app.auth_manager.submit_phone_code(account_id, code)
|
||||
except Exception as exc:
|
||||
_record_auth_failure(ip, account_id)
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to submit phone code")
|
||||
_record_auth_success(ip, account_id)
|
||||
return self.send_json(payload)
|
||||
|
||||
@@ -2729,7 +2930,7 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
payload = self.app.auth_manager.submit_password(account_id, password)
|
||||
except Exception as exc:
|
||||
_record_auth_failure(ip, account_id)
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to submit password")
|
||||
_record_auth_success(ip, account_id)
|
||||
return self.send_json(payload)
|
||||
|
||||
@@ -2875,10 +3076,10 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
|
||||
def _handle_account_continuous(self, account_id: str, body: Dict[str, Any]) -> None:
|
||||
try:
|
||||
enabled = bool(body.get("enabled"))
|
||||
enabled = parse_bool(body.get("enabled"))
|
||||
interval_minutes = int(body.get("interval_minutes", 1))
|
||||
channels, dropped = clean_continuous_channels(body.get("channels", []))
|
||||
run_all_tracked = bool(body.get("run_all_tracked", True))
|
||||
run_all_tracked = parse_bool(body.get("run_all_tracked", True), default=True)
|
||||
payload = self.app.continuous_orchestrator.update_for(
|
||||
account_id=account_id,
|
||||
enabled=enabled,
|
||||
@@ -2887,7 +3088,7 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
run_all_tracked=run_all_tracked,
|
||||
)
|
||||
except Exception as exc:
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, str(exc))
|
||||
return self.send_bad_request_from_exc(exc, fallback="Failed to update continuous scraping")
|
||||
payload["dropped_invalid"] = dropped
|
||||
return self.send_json(payload)
|
||||
|
||||
@@ -2983,6 +3184,13 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
if not origin_host or (host and origin_host != host):
|
||||
self.send_error_json(HTTPStatus.FORBIDDEN, "Cross-origin request rejected")
|
||||
return False
|
||||
# L-8: When an Origin is present, guard against DNS rebinding: the
|
||||
# Host must resolve to a local/private address, otherwise a remote
|
||||
# attacker domain could make both Host == Origin pass. Requests
|
||||
# without an Origin (curl, plain LAN tools) are unaffected.
|
||||
if host and not _is_trusted_host(host):
|
||||
self.send_error_json(HTTPStatus.FORBIDDEN, "Untrusted host")
|
||||
return False
|
||||
sec_fetch_site = self.headers.get("Sec-Fetch-Site", "").strip().lower()
|
||||
if sec_fetch_site and sec_fetch_site not in {"same-origin", "same-site", "none"}:
|
||||
self.send_error_json(HTTPStatus.FORBIDDEN, "Cross-origin request rejected")
|
||||
@@ -3007,6 +3215,22 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
file_path.relative_to(DATA_DIR.resolve())
|
||||
except ValueError:
|
||||
return self.send_error_json(HTTPStatus.FORBIDDEN, "Forbidden")
|
||||
|
||||
# M-1: Never serve sensitive state / session / db files. Normal
|
||||
# media documents live under DATA_DIR with a media-file extension;
|
||||
# anything else (state.json, *.db, *.session, sqlite sidecars) is
|
||||
# denied regardless of its location under DATA_DIR.
|
||||
file_suffix = file_path.suffix.lower()
|
||||
file_name = file_path.name.lower()
|
||||
if file_name in _SENSITIVE_FILE_NAMES or file_suffix in _SENSITIVE_FILE_SUFFIXES:
|
||||
return self.send_error_json(HTTPStatus.FORBIDDEN, "Forbidden")
|
||||
# Only allow registered media/document extensions. This keeps the
|
||||
# viewer's image/video/audio/document URLs working while blocking
|
||||
# arbitrary file reads (e.g. /media/../state.json is already blocked
|
||||
# by containment, and any other extension is not a media asset).
|
||||
if file_suffix not in _MEDIA_FILE_EXTENSIONS:
|
||||
return self.send_error_json(HTTPStatus.FORBIDDEN, "Forbidden")
|
||||
|
||||
if not file_path.exists() or not file_path.is_file():
|
||||
return self.send_error_json(HTTPStatus.NOT_FOUND, "Media file not found")
|
||||
mime_type = mimetypes.guess_type(str(file_path))[0] or "application/octet-stream"
|
||||
@@ -3018,6 +3242,11 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
last_modified = stat.st_mtime
|
||||
|
||||
range_header = self.headers.get("Range", "").strip()
|
||||
# M-8: for an empty file there is nothing to range-serve; ignore the
|
||||
# Range header entirely and send the full (empty) 200 response so
|
||||
# clients don't see a spurious 416 for `bytes=0-0` on size-0 files.
|
||||
if file_size == 0:
|
||||
range_header = ""
|
||||
if range_header.startswith("bytes="):
|
||||
try:
|
||||
start, end = self._parse_range(range_header, file_size)
|
||||
@@ -3034,6 +3263,9 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
)
|
||||
self.send_header("Content-Range", f"bytes {start}-{end}/{file_size}")
|
||||
self.send_header("Content-Length", str(content_length))
|
||||
self._send_security_headers()
|
||||
if "text/html" in content_type:
|
||||
self.send_header("Content-Security-Policy", self._CSP)
|
||||
self.end_headers()
|
||||
if not head_only:
|
||||
self._write_file_range(file_path, start, content_length)
|
||||
@@ -3046,12 +3278,25 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(last_modified)),
|
||||
)
|
||||
self.send_header("Content-Length", str(file_size))
|
||||
self._send_security_headers()
|
||||
if "text/html" in content_type:
|
||||
self.send_header("Content-Security-Policy", self._CSP)
|
||||
self.end_headers()
|
||||
if not head_only:
|
||||
self._write_file_range(file_path, 0, file_size)
|
||||
|
||||
def _parse_range(self, range_header: str, file_size: int) -> tuple[int, int]:
|
||||
"""Parse a single ``bytes=start-end`` range spec.
|
||||
|
||||
Raises ``ValueError`` (cleanly caught by the caller → 416) for any
|
||||
malformed / multi-range / out-of-bounds input. Never lets an
|
||||
exception other than ValueError escape.
|
||||
"""
|
||||
try:
|
||||
range_val = range_header.removeprefix("bytes=").strip()
|
||||
# Multi-range requests ("bytes=0-1,5-6") are not supported here.
|
||||
if "," in range_val:
|
||||
raise ValueError("multi-range not supported")
|
||||
if "-" not in range_val:
|
||||
raise ValueError("missing dash")
|
||||
parts = range_val.split("-", 1)
|
||||
@@ -3064,6 +3309,11 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
if start < 0 or start >= file_size or end >= file_size or start > end:
|
||||
raise ValueError("out of bounds")
|
||||
return start, end
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.debug("Unexpected range parse error: %s", exc)
|
||||
raise ValueError("invalid range") from exc
|
||||
|
||||
def _write_file_range(self, file_path: Path, offset: int, length: int) -> None:
|
||||
chunk_size = 65536
|
||||
@@ -3077,22 +3327,42 @@ class TelegramScraperRequestHandler(BaseHTTPRequestHandler):
|
||||
self.wfile.write(chunk)
|
||||
remaining -= len(chunk)
|
||||
|
||||
# ── L-4: Security headers ─────────────────────────────────────────────
|
||||
_CSP = "default-src 'self'; img-src 'self' data:; style-src 'self'"
|
||||
|
||||
def _send_security_headers(self) -> None:
|
||||
"""Emit standard hardening headers on every HTTP response."""
|
||||
self.send_header("X-Content-Type-Options", "nosniff")
|
||||
self.send_header("X-Frame-Options", "DENY")
|
||||
self.send_header("Referrer-Policy", "no-referrer")
|
||||
|
||||
def send_json(self, payload: Any, status: HTTPStatus = HTTPStatus.OK) -> None:
|
||||
raw = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(raw)))
|
||||
self._send_security_headers()
|
||||
self.end_headers()
|
||||
self.wfile.write(raw)
|
||||
|
||||
def send_error_json(self, status: HTTPStatus, message: str) -> None:
|
||||
self.send_json({"error": message, "status": int(status)}, status=status)
|
||||
|
||||
def send_bad_request_from_exc(
|
||||
self, exc: Exception, fallback: str = "Bad request"
|
||||
) -> None:
|
||||
"""Send a 400 JSON error without leaking filesystem paths or other
|
||||
server internals. The caught exception is logged server-side and the
|
||||
client receives only *fallback*."""
|
||||
logger.exception("Request failed; client sees '%s'", fallback)
|
||||
return self.send_error_json(HTTPStatus.BAD_REQUEST, fallback)
|
||||
|
||||
def log_message(self, format: str, *args: Any) -> None:
|
||||
safe_path = self.path.split("?", 1)[0]
|
||||
logger.info(
|
||||
"%s %s — %s",
|
||||
self.command,
|
||||
self.path,
|
||||
safe_path,
|
||||
args[1] if len(args) > 1 else "-",
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user