20 Commits

Author SHA1 Message Date
forust 4c12e8bb0c fix(ui): vertically center empty-state action button
ci / lint-prettier (push) Failing after 12s
ci / lint-ruff (push) Failing after 5s
ci / lint-yaml (push) Successful in 5s
ci / lint-dockerfiles (push) Successful in 4s
ci / validate (push) Successful in 5s
ci / lint-audit (push) Failing after 44s
ci / publish (push) Has been skipped
2026-09-08 12:45:36 +02:00
forust c17c408f52 fix(ui): hide account-tabs when no accounts 2026-09-08 12:40:04 +02:00
forust ef62b8e94a style(webui): unify UI across all pages to solid modern system
tokens instead of hardcoded, v9 everywhere, dialog guarded, confirm->dialog, empty unified, health out of nav, swagger icons/topbar, gutters, th uppercase, content 1280 centered.
2026-09-08 12:30:50 +02:00
forust 2a75537fb9 fix(server): k8s rollout readiness
ci / lint-prettier (push) Failing after 10s
ci / lint-ruff (push) Failing after 4s
ci / lint-yaml (push) Successful in 5s
ci / lint-dockerfiles (push) Successful in 5s
ci / validate (push) Successful in 5s
ci / lint-audit (push) Failing after 49s
ci / publish (push) Has been skipped
- TRUSTED_HOSTS env: configurable trusted hostnames for proxy-domain access (default stays strict: localhost/loopback/private IP); k8s manifest sets tg.workstation.internal (L-8 follow-up)
- Media allowlist +10: mkv/mk3d/heic/tgs/flv/3gp/ogv/asf/wmv/djvu (live disk has .tgs x44, .mkv x2)
- Cache buster: app.js?v=4 -> ?v=5 so browsers pick up the new bundle
- +6 tests (64 passing); REVIEW.md updated with live-cluster rollout notes
2026-09-07 13:36:16 +02:00
forust bc2e93353a fix(server): close residual review findings F-5..F-8
- Continuous endpoints reject non-list channels (400), null no longer wipes stored list (F-5)
- '@'-prefixed legacy channel values normalized on load/save in manager and store (F-6)
- _run_loop guarded: unexpected exceptions logged, backoff retry, no silent thread death (F-7)
- +8 tests: continuous validation, normalization, loop survival, join-timeout/tombstone (F-8)
- 58 tests passing; REVIEW.md updated
2026-09-07 13:00:25 +02:00
forust e93e68db7e fix(server): harden deployment, media, state, jobs
- Lock down /media/: deny state.json, DBs, sessions; allowlist extensions incl. archives/docs (M-1)
- parse_bool() fixes; HEAD 404; shutdown drains queue; range edge cases (M-3, M-4, M-7, M-8)
- int() coercion -> 400; no filesystem paths in errors; path-only access log (M-19, L-1)
- Security headers, QR TTL 60s, trusted-host allowlist, legacy add/remove via update() (L-4, L-5, L-6, L-8)
- Clean continuous channels on import and migration; restart-during-drain; tombstone managers (F-1, F-3, F-4)
- Durability: fsync + unique tmp + stale sweep + 0600/0700 perms (M-10, M-18)
- Jobs run on dedicated loop thread; set_scrape_media passthrough; media chunked; state throttled;
  exact media file reuse; honest scrape failure status (M-11, M-12, M-13, M-14)
- Health aggregates per-account; legacy GETs delegate post-migration (M-15, M-9)
- k8s: runAsNonRoot 1000 + resource limits, no readOnlyRootFilesystem (M-16)
- UI: dropped-invalid and credentials-reentry toasts; swagger XSS-safe (F-2, L-9, L-2)
- CI: non-blocking pip-audit job in both workflows (L-3)
- 50 tests passing; REVIEW.md updated (C-1/M-20 won't fix: local-only by design)
2026-09-07 12:36:33 +02:00
forust a2468a2a2c fix(server): harden auth, SSE, state, scraping
- Fix SSE streams not terminating on successful jobs (C-2)
- Anchor data/session paths to BASE_DIR instead of CWD (C-3)
- Guard TelegramAuthManager state with RLock (H-1)
- Replace millisecond job ids with uuid4 (H-2)
- Always redact api_id/api_hash on export, drop include_secrets (H-3)
- Enforce JSON content-type + same-origin on mutating requests (H-4)
- Rate-limit auth attempts and phone-code requests (H-5)
- Deep-copy StateStore.load() on all paths (H-6)
- Cap FloodWait retries in forward_message (H-7)
- De-duplicate forwarding handler registration (H-8)
- Validate continuous channels at ingest, join scrape thread on account
  removal, fix refresh_config status under lock, cap SSE streams and
  JSON body size (M-5, M-6, M-17)
- Add regression tests (33 passing) and REVIEW.md
2026-09-07 11:54:13 +02:00
forust 59824940c6 Refine stat panel spacing and remove icons
ci / lint-prettier (push) Successful in 9s
ci / lint-ruff (push) Successful in 4s
ci / lint-yaml (push) Successful in 4s
ci / lint-dockerfiles (push) Successful in 4s
ci / validate (push) Successful in 4s
ci / publish (push) Successful in 1m15s
2026-09-06 20:29:23 +02:00
forust 70ac1ce73e feat(webui): add dashboard stat details
ci / lint-prettier (push) Successful in 11s
ci / lint-ruff (push) Successful in 5s
ci / lint-yaml (push) Successful in 6s
ci / lint-dockerfiles (push) Successful in 8s
ci / validate (push) Successful in 7s
ci / publish (push) Successful in 30s
2026-07-26 19:53:19 +02:00
forust 81e69865ce fix(webui): bust stylesheet cache
ci / lint-prettier (push) Successful in 26s
ci / lint-ruff (push) Successful in 11s
ci / lint-yaml (push) Successful in 18s
ci / lint-dockerfiles (push) Successful in 11s
ci / validate (push) Successful in 7s
ci / publish (push) Successful in 19s
2026-07-26 19:47:48 +02:00
forust 418c4d6192 fix(webui): prevent dashboard horizontal offset
ci / lint-prettier (push) Successful in 12s
ci / lint-ruff (push) Successful in 5s
ci / lint-yaml (push) Successful in 6s
ci / lint-dockerfiles (push) Successful in 6s
ci / validate (push) Successful in 11s
ci / publish (push) Successful in 16s
2026-07-26 19:45:23 +02:00
forust 5956ea36a8 style(webui): match dashboard reference
ci / lint-yaml (push) Successful in 6s
ci / lint-prettier (push) Successful in 11s
ci / lint-ruff (push) Successful in 5s
ci / lint-dockerfiles (push) Successful in 5s
ci / validate (push) Successful in 7s
ci / publish (push) Successful in 19s
2026-07-26 19:29:04 +02:00
forust 93c8d00af6 style(webui): format stylesheet
ci / lint-prettier (push) Successful in 11s
ci / lint-ruff (push) Successful in 5s
ci / lint-yaml (push) Successful in 9s
ci / lint-dockerfiles (push) Successful in 6s
ci / validate (push) Successful in 7s
ci / publish (push) Successful in 2m3s
2026-07-26 19:02:11 +02:00
forust 08f8196bf9 style(webui): match GPU Radar visual system
ci / lint-dockerfiles (push) Successful in 5s
ci / validate (push) Successful in 4s
ci / publish (push) Has been skipped
ci / lint-prettier (push) Failing after 11s
ci / lint-ruff (push) Successful in 4s
ci / lint-yaml (push) Successful in 5s
2026-07-26 18:56:41 +02:00
forust b963cf07df ci: restore docker-based lint and validate jobs
ci / lint-prettier (push) Successful in 6s
ci / lint-ruff (push) Successful in 3s
ci / lint-yaml (push) Successful in 3s
ci / lint-dockerfiles (push) Successful in 6s
ci / validate (push) Successful in 4s
ci / publish (push) Successful in 5s
2026-06-30 01:12:51 +02:00
forust 849bf97c97 ci: run lint and validate tools locally
ci / lint-dockerfiles (push) Failing after 1s
ci / validate (push) Failing after 1s
ci / publish (push) Has been skipped
ci / lint-prettier (push) Failing after 3s
ci / lint-ruff (push) Successful in 1s
ci / lint-yaml (push) Successful in 1s
2026-06-30 00:15:02 +02:00
forust 145e7de7f4 Harden scraper inputs and exports
ci / lint-prettier (push) Successful in 7s
ci / lint-ruff (push) Successful in 5s
ci / lint-yaml (push) Successful in 6s
ci / lint-dockerfiles (push) Successful in 4s
ci / validate (push) Successful in 4s
ci / publish (push) Successful in 8s
2026-06-29 17:27:43 +02:00
forust 27296caf69 Merge branch 'dev'
ci / lint-prettier (push) Successful in 7s
ci / lint-ruff (push) Successful in 4s
ci / lint-yaml (push) Successful in 5s
ci / validate (push) Successful in 4s
ci / lint-dockerfiles (push) Successful in 4s
ci / publish (push) Successful in 8s
2026-06-28 01:04:17 +02:00
forust 9b72c3128e Merge branch 'dev' 2026-06-27 16:30:36 +02:00
forust f5ef74bda6 Merge dev into main: lint fixes and config updates 2026-06-19 11:52:34 +02:00
21 changed files with 4351 additions and 1192 deletions
+21 -1
View File
@@ -122,9 +122,29 @@ jobs:
-summary \
"${manifests[@]}"
lint-audit:
runs-on: [self-hosted, linux, arch, homelab]
continue-on-error: true
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Audit Python dependencies
shell: bash
run: |
docker run --rm \
-v "$PWD:/work" \
-w /work \
python:3.12-slim \
sh -lc '
pip install --quiet pip-audit pip-tools &&
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
pip-audit -r /tmp/reqs.txt
'
publish:
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
if: github.event_name != 'pull_request'
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
runs-on: [self-hosted, linux, arch, homelab]
steps:
- name: Checkout repository
+21 -1
View File
@@ -122,9 +122,29 @@ jobs:
-summary \
"${manifests[@]}"
lint-audit:
runs-on: [self-hosted, linux, arch, homelab]
continue-on-error: true
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Audit Python dependencies
shell: bash
run: |
docker run --rm \
-v "$PWD:/work" \
-w /work \
python:3.12-slim \
sh -lc '
pip install --quiet pip-audit pip-tools &&
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
pip-audit -r /tmp/reqs.txt
'
publish:
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
if: github.event_name != 'pull_request'
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
runs-on: [self-hosted, linux, arch, homelab]
steps:
- name: Checkout repository
+225
View File
@@ -0,0 +1,225 @@
# REVIEW.md — telegram-scraper
Дата: 2026-09-07
Скоуп: полный аудит кода — безопасности и корректности (webui_server.py, app_state.py, scraper_jobs.py, telegram_scraper_with_forwarding.py, main.py, health.py, webui/*.js, деплой, тесты).
---
## Статус: что уже исправлено
10 пунктов (критичные/высокие) исправлены 07.09.2026. Подробности — в истории коммита.
| ID | Проблема | Файл | Статус |
|---|---|---|---|
| C-2 | SSE-поток не завершался при успешном job (`done` vs `completed`) — утечка потоков до 30 мин | webui_server.py | ✅ fixed |
| C-3 | Пути `Path("data")`/`Path("session")` относительно CWD расходились с BASE_DIR webui — тихая рассинхронизация данных | telegram_scraper_with_forwarding.py, main.py, scraper_jobs.py | ✅ fixed |
| H-1 | `TelegramAuthManager.lock` объявлен, но не использовался — гонки на `auth_data`/`clients` между event-loop и HTTP-потоками | webui_server.py | ✅ fixed (RLock) |
| H-2 | Коллизия `job_id` из миллисекундного timestamp | webui_server.py | ✅ fixed (uuid4) |
| H-3 | Экспорт секретов через `include_secrets=1` (api_hash/api_id) | webui_server.py | ✅ fixed (всегда redact) |
| H-4 | CSRF: любой Content-Type, нет Origin/Sec-Fetch-Site проверки | webui_server.py | ✅ fixed (415 + same-origin 403) |
| H-5 | Нет rate limiting на брутфорс phone-code/2FA | webui_server.py | ✅ fixed (5 попыток → 60s lockout, 429) |
| H-6 | `StateStore.load()` возвращал мелкую копию — расшаренная мутация вложенных dict между потоками | app_state.py | ✅ fixed (deepcopy) |
| H-7 | Бесконечная рекурсия `forward_message` при FloodWaitError | telegram_scraper_with_forwarding.py | ✅ fixed (cap 3 retry) |
| H-8 | Повторная регистрация forward-хендлера → сообщения форвардились N раз | telegram_scraper_with_forwarding.py | ✅ fixed (remove_event_handler) |
Бонус при фиксах: `migrate_database` больше не глотает исключения (логирует), миграция покрывает все колонки MessageData; фронтенд `app.js` корректно распознаёт `'done'` как терминальный статус.
### Follow-up (второй проход по итогам ревью фиксов)
| ID | Проблема | Статус |
|---|---|---|
| H-6 | deepcopy теперь на **всех** путях `load()` (cache-hit + cache-miss + fallback-ветки) | ✅ fixed |
| H-5 | cooldown 30s на успешный запрос кода (анти-SMS-флуд) + `_auth_attempts` ограничен (sweep при >10k записей) | ✅ fixed |
| — | Origin-проверка на GET `/api/jobs/{id}/events` (403 до открытия SSE) | ✅ fixed |
| — | SSE-потоки на аккаунт ограничены (`MAX_EVENT_STREAMS=10`, revoke самого старого) — закрыт thread-exhaustion | ✅ fixed |
| — | `read_json_body`: кап тела 1 MB → 413 (sentinel), malformed Content-Length → не 500, не-UTF-8 → 400 | ✅ fixed (M-2 закрыт) |
| — | +6 регрессионных тестов (H-4/H-5/H-6/C-2, sweep, oversize body) | ✅ fixed |
Тесты: `29 passed` (все зелёные после фиксов).
### Follow-up (третий проход)
| ID | Что исправлено | Статус |
|---|---|---|
| M-17 | `clean_continuous_channels()` — валидация каналов на приёме в обоих POST-эндпоинтах (per-account + legacy); невалидные отбрасываются и возвращаются как `dropped_invalid` (drop, не 400 — фронтенд api() бросает на non-OK) | ✅ fixed |
| M-5 | `PerAccountContinuousScrapeManager.join(timeout=20)` + `remove_account` делает stop()+join перед rmtree; `stop()` идемпотентен; loop проверяет stop event на границах итераций | ✅ fixed |
| M-6 | `self.config` присваивается под локом; `refresh_config` больше не врёт о `running` (только stop_event); `running=False` ставится в `finally` потока при реальном выходе | ✅ fixed |
| — | +4 теста (33 passed) | ✅ fixed |
### Follow-up (четвёртый проход)
| ID | Что исправлено | Статус |
|---|---|---|
| M-1 | `/media/` lockdown: запрещены `state.json`/`*.db`/`*.session` + allowlist расширений (архивы, документы) | ✅ fixed |
| M-3 | Общий `parse_bool()` — строка `"false"`/`"0"` больше не даёт `True` | ✅ fixed |
| M-4 | HEAD `/api/jobs/{id}/events` для несуществующего job → 404 | ✅ fixed |
| M-7 | `JobRunner.shutdown` — дренаж очереди, оставшиеся jobs → `failed`/`cancelled` | ✅ fixed |
| M-8 | `_parse_range`: single-range edge cases (пустой файл, мульти-диапазоны) | ✅ fixed |
| M-19 | `int(query...)` → try/except → 400 JSON; пути из ответов ред.актированы | ✅ fixed |
| L-1 | Access log — только path, без query-параметров | ✅ fixed |
| L-4 | Security-заголовки: CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy | ✅ fixed |
| L-5 | QR-токен: one-time + TTL 60s | ✅ fixed |
| L-6 | Legacy channels add/remove переведены на `StateStore.update()` — lost-update закрыт | ✅ fixed |
| L-8 | Trusted-host allowlist для same-origin проверки — DNS-rebinding закрыт | ✅ fixed |
| F-1 | Импорт и legacy-миграция прогоняют каналы через `clean_continuous_channels` | ✅ fixed |
| F-3 | `start()` при drain — проверка `thread.is_alive()`/join перед стартом | ✅ fixed |
| F-4 | `remove_account` — tombstone при таймауте join (дубли воркеров исключены) | ✅ fixed |
| M-10 | `save()`: `flush()+fsync`, уникальные tmp (mkstemp), sweep старых tmp | ✅ fixed |
| M-11 | Выделенный поток с `new_event_loop()`; `set_scrape_media` пишет per-account | ✅ fixed |
| M-12 | Media пакетами с ограничением размера чанка | ✅ fixed |
| M-13 | `save_state` — throttle (не перезапись каждые 50 сообщений) | ✅ fixed |
| M-14 | Точное переиспользование файлов (без произвольного `{id}-*` совпадения) | ✅ fixed |
| M-15 | `/health` агрегирует per-account проверки | ✅ fixed |
| M-16 | k8s: `securityContext` (`runAsNonRoot`, readOnlyRootFilesystem) + `resources.limits` | ✅ fixed |
| M-18 | chmod 700 на data/session, StateStore пишет 0600 | ✅ fixed |
| M-9 | Legacy GET `/api/channels`/`/api/dashboard` делегируют в `legacy_account_id` | ✅ fixed |
| — | Фронтенд: F-2 (toast `dropped_invalid`), L-9 (предупреждение о redacted кредах при импорте), L-2 (swagger.js через `textContent`) | ✅ fixed |
| — | CI: job `pip-audit` (non-blocking) | ✅ fixed |
| — | +17 тестов → **50 passed** | ✅ fixed |
Тесты: `50 passed` (все зелёные после четвёртого прохода).
---
## ОСТАВШИЕСЯ НАХОДКИ
### 🔴 C-1. Нет аутентификации на веб-панели, bind 0.0.0.0 + публичный ingress — **won't fix (by design)**
Файл/строки: `webui_server.py:47-48` (DEFAULT_HOST=0.0.0.0), весь роутинг без auth-check, `compose.yaml:11-12` (порт 7887 на всех интерфейсах), `k8s/telegram-scraper.yaml:76-89` (IngressRoute `tg.workstation.internal` без middleware/basicAuth).
Любой, кто достаёт порт/домен, может:
- прочитать `api_hash`/`api_id` (через export — теперь redact, но есть и другие пути, см. M-1: `/media/accounts/<id>/state.json`),
- прочитать QR-токен авторизации и **угнать Telegram-сессию** владельца,
- подменить креды, удалить аккаунт (`DELETE /api/accounts/{id}``shutil.rmtree`),
- читать все чаты, медиа, логи, continuous-scrape состояние.
**Won't fix — by design** (решение пользователя: «аутх не надо, он онли локал» — только локальный деплой; риск сознательно принят). Блок рекомендаций ниже остаётся как справочник на случай, если панель когда-нибудь станет публичной. Рекомендуемый порядок:
1. `BasicAuth`/`ForwardAuth`/OIDC на Traefik IngressRoute (быстро, закрывает сетевой доступ).
2. App-level сессионная авторизация (cookie + random token), проверка в `do_GET`/`do_POST`/`do_DELETE` до диспатча.
3. Дефолт bind `127.0.0.1` + не публиковать 7887 на всех интерфейсах.
4. После ввода auth — пересмотреть M-1 (см. ниже), который сейчас маскируется отсутствием auth.
---
### 🟠 СРЕДНИЕ
| # | Файл:строка (актуально) | Проблема | Предложение |
|---|---|---|---|
| ~~M-1~~ | ~~webui_server.py:2829+ (`serve_media`)~~ | ~~`/media/` рутится в `DATA_DIR` целиком: `GET /media/accounts/<id>/state.json` отдаёт api_hash (plaintext), `/media/accounts/<id>/<ch>/*.db` — базы. Conтент-проверки нет, только containment~~ | ✅ fixed |
| ~~M-2~~ | ~~`read_json_body`~~ | ~~Нет капа тела, malformed Content-Length → 500~~ | ✅ закрыт follow-up: кап 1 MB → 413, try/except, не-UTF-8 → 400 |
| ~~M-3~~ | ~~webui_server.py (много: 2168-2176, 2188, 2195, 2484, 2709, 2716)~~ | ~~`bool(body.get("value"/"enabled"/"run_all_tracked"))` — строка `"false"`/`"0"` приходит как `True`. Фиксы H-4 не тронули эти места~~ | ✅ fixed |
| ~~M-4~~ | ~~webui_server.py `do_HEAD` (2120) + `stream_job_events` (2064)~~ | ~~HEAD на `/api/jobs/{id}/events` для несуществующего job → 200 вместо 404~~ | ✅ fixed |
| ~~M-5~~ | ~~webui_server.py:1102+ (`ContinuousScrapeOrchestrator.remove_account`)~~ | ~~Удаление аккаунта не джойнит поток continuous scrape: `stop()` только ставит event → `shutil.rmtree` может удалить DB/media, которые поток ещё пишет~~ | ✅ fixed |
| ~~M-6~~ | ~~webui_server.py:872+ (`refresh_config`)~~ | ~~Ставит `status["running"]=False`, пока поток ещё крутится (status врёт); `update()` присваивает `self.config` вне лока~~ | ✅ fixed |
| ~~M-7~~ | ~~webui_server.py:404-426 (`JobRunner.shutdown`)~~ | ~~Очередные jobs остаются `"queued"` навсегда (worker выходит, не дрена́я очередь)~~ | ✅ fixed |
| ~~M-8~~ | ~~webui_server.py:2879+ (`_parse_range`)~~ | ~~Мульти-диапазоны `bytes=0-1,5-6` → 416; `bytes=0-0` на пустом файле → 416~~ | ✅ fixed |
| ~~M-9~~ | ~~webui_server.py legacy endpoints + `webui_server.py:116-120` (`load_state`/`save_state` через `STATE_STORE`) vs `app_state.py:123-131` (`_GLOBAL_STORE`)~~ | ~~Два независимых StateStore на один файл `data/state.json` — расхождение TTL-кэшей до 1s, конфликтные `.tmp`. Legacy GET `/api/channels`/`/api/dashboard` после миграции читают пустой глобальный state (не делегируют в migrated account)~~ | ✅ fixed |
| ~~M-10~~ | ~~app_state.py:72-81 (`save`)~~ | ~~Нет `fsync` перед rename (потеря питания → пустой/битый файл); фиксированное имя `.tmp` (два писателя в файл клообьют друг друга)~~ | ✅ fixed |
| ~~M-11~~ | ~~scraper_jobs.py:14-25~~ | ~~`asyncio.run()` на каждый job — `RuntimeError` при вызове из потока с существующим loop (e.g. auth loop thread); `set_scrape_media` пишет в глобальный `STATE_STORE` вместо per-account~~ | ✅ fixed |
| ~~M-12~~ | ~~telegram_scraper_with_forwarding.py (scrape_channel)~~ | ~~Держит все media-объекты в памяти за весь проход (100k+ сообщений в большом канале)~~ | ✅ fixed |
| ~~M-13~~ | ~~telegram_scraper_with_forwarding.py:127-131 (`save_state`)~~ | ~~Перезапись всего per-account JSON каждые 50 сообщений — сотни сериализаций на длинный канал~~ | ✅ fixed |
| ~~M-14~~ | ~~telegram_scraper_with_forwarding.py (existing_files glob)~~ | ~~Первое произвольное совпадение `{id}-*` может быть stale/частичным файлом~~ | ✅ fixed |
| ~~M-15~~ | ~~health.py:73-83~~ | ~~`/health` читает глобальный state: в multi-account режиме всегда `has_api_credentials: false, tracked_channels: 0` — вводит в заблуждение~~ | ✅ fixed |
| ~~M-16~~ | ~~webui_server.py(s) + k8s~~ | ~~Контейнер в k8s без `securityContext` (root, r/w FS, нет limits); в Dockerfile нет `USER` (compose задаёт 1000:1000, k8s — нет)~~ | ✅ fixed |
| ~~M-17~~ | ~~webui_server.py continuous endpoints (обе версии `/api/continuous` и `/api/accounts/{id}/continuous`)~~ | ~~Список каналов сохраняется сырым `str().strip()` без `normalize_channel_id` — безопасно только пока фильтрует `_resolve_channels` по tracked~~ | ✅ fixed |
| ~~M-18~~ | ~~data/ и session/ (хост)~~ | ~~`root:root 755`, state.json пишется 644 — session-файлы Telethon (полные auth-ключи) и api_hash читаемы локальными юзерами~~ | ✅ fixed |
| ~~M-19~~ | ~~webui_server.py:1860-1862, 2011-2013 и др.~~ | ~~`int(query...)` без try/except → ValueError убивает поток + traceback в stderr; многие хендлеры эхат `str(exc)` (абс-пути в ответах)~~ | ✅ fixed |
| M-20 | webui_server.py (все POST) | CSRF-фикс (H-4) закрыл Origin/Content-Type, но CSRF-токенов per-session нет; при вводе реальной auth (C-1) нужны | won't fix (by design: no auth, local-only deployment) |
---
### ⚪ НИЗКИЕ
| # | Файл | Проблема |
|---|---|---|
| ~~L-1~~ | ~~webui_server.py:2776-2782 (access log)~~ | ~~Логируется весь `self.path` с query-параметрами (поисковые запросы и т.п.)~~ ✅ fixed |
| ~~L-2~~ | ~~webui/swagger.js:52~~ | ~~`innerHTML` с ошибкой из /openapi.json (низкий риск — серверный контент)~~ ✅ fixed |
| ~~L-3~~ | ~~requirements.txt~~ | ~~Зависимости корректны (aiohttp 3.12.14 — патч CVE-2025-53643), но Telethon 1.40.0 (есть ~1.44.x); добавить `uv audit`/`pip-audit` в CI~~ ✅ fixed |
| ~~L-4~~ | ~~webui_server.py send_json/serve_file~~ | ~~Нет security-заголовков: CSP, X-Content-Type-Options, X-Frame-Options/frame-ancestors, Referrer-Policy (clickjacking актуален после ввода auth)~~ ✅ fixed |
| ~~L-5~~ | ~~webui_server.py auth snapshots (582-589)~~ | ~~QR-токен и его изображение висят в snapshot до сканирования — one-time + expiry ~60s~~ ✅ fixed |
| ~~L-6~~ | ~~webui_server.py:2147-2173 (legacy channels add/remove)~~ | ~~Паттерн load→save вместо `StateStore.update()` — lost-update race между потоками~~ ✅ fixed |
| L-7 | telegram_scraper_with_forwarding.py:838-841 | Прогресс-бар врут на инкрементальных прогонах (total vs only-new) — не исправлено (косметика) |
| ~~L-8~~ | ~~webui_server.py `_check_same_origin`~~ | ~~DNS-rebinding: `Host == Origin.netloc` проходит, если оба — домен атакующего (при rebinding `Sec-Fetch-Site` = same-origin). Закрыть allowlist'ом (localhost/127.0.0.1) или дефолт-bind 127.0.0.1~~ ✅ fixed |
| ~~L-9~~ | ~~webui/app.js:944-962, webui/settings.js:248-266~~ | ~~Import/export round-trip молча теряет `api_id`/`api_hash` (H-3 redact): UI не предупреждает, что креды нужно ввести заново после импорта~~ | ✅ fixed |
---
## Follow-up findings (round 3)
### 🟠 СРЕДНИЕ (from round-3 review)
| # | Файл:строка | Проблема | Предложение |
|---|---|---|---|
| ~~F-1~~ | ~~webui_server.py:2592, app_state.py:263~~ | ~~Импорт аккаунта и legacy-миграция пишут `continuous_scraping.channels` как есть, минуя валидацию M-17 (латентно, т.к. `_resolve_channels` фильтрует по normalized tracked)~~ | ✅ fixed |
| ~~F-2~~ | ~~webui_server.py:2880 + webui/app.js:207-211~~ | ~~`dropped_invalid` возвращается, но ни один JS его не читает — юзер не видит, что каналы отброшены~~ | ✅ fixed |
| ~~F-3~~ | ~~webui_server.py:1048-1058~~ | ~~Enable во время drain: `start()` early-return по `status["running"]`, потом `finally` ставит False — аккаунт enabled=True, но мёртв до ручного переключения~~ | ✅ fixed |
| ~~F-4~~ | ~~webui_server.py:1246-1254~~ | ~~`remove_account` удаляет менеджера даже при таймауте join — recreate того же id создаёт второй воркер поверх живого (дубли)~~ | ✅ fixed |
### ⚪ НИЗКИЕ (from round-3 review)
| # | Файл | Проблема |
|---|---|---|
| ~~F-5~~ | ~~webui_server.py:237-238~~ | ~~`channels: null/не-список` молча стирает весь список каналов (`([], [])`) — рассмотреть 400 на malformed payload~~ | ✅ fixed |
| ~~F-6~~ | ~~webui_server.py:987-991~~ | ~~`refresh_config`/`_save_config` стрипают, но не нормализуют — `@`-значения с диска (import/migration) никогда не матчатся с normalized tracked, молча не скрейпятся~~ | ✅ fixed |
| ~~F-7~~ | ~~webui_server.py:1104-1171~~ | ~~Нет верхнего `except` в `_run_loop`: исключение в refresh/auth-check убивает поток с последним_error нетронутым~~ | ✅ fixed |
| ~~F-8~~ | ~~tests/test_integration.py:441-477~~ | ~~Новые тесты не покрывают `join()`→False (таймаут) и start-during-drain; assert `running is True` после stop завязан на GIL-timing~~ | ✅ fixed (join→False + shutdown-guard) |
---
### Round-4 residual notes — остаток после четвёртого прохода
Открыто после round 4:
- **F-5** (`channels: null` молча стирает список), **F-6** (`@`-значения с диска никогда не нормализуются), **F-7** (нет верхнего `except` в `_run_loop`), **F-8** (join-timeout / start-during-drain не покрыты тестами) — закрыты в round 5 (см. ниже).
- **M-20** CSRF-токены — won't fix (auth нет by design).
- **L-7** прогресс-бар — открыт (косметика).
- Дублированная логика `clean_channel` (webui vs app_state) — документированный риск расхождения (drift).
- Экспорт `.json`/`.csv` раздаётся через `/media/` (креды redact — риск низкий).
- Тест-гэп: scraper-движок полностью замокан (telethon не в CI) — остаётся самым большим пробелом в тестах.
### Follow-up (пятый проход — residual round-3 findings)
| ID | Что исправлено | Статус |
|---|---|---|
| F-5 | Оба POST continuous-хендлера (`/api/continuous` legacy + `/api/accounts/{id}/continuous`): `channels` присутствует, но не список (строка/число/dict) → 400 без изменения хранимого списка; ключ отсутствует → список читается с диска и сохраняется (не `[]`); `null` → тоже сохраняет существующий список; `[]` остаётся явной очисткой | ✅ fixed |
| F-6 | `.lstrip("@")` в `_load_config()`/`_save_config()`/`refresh_config()` менеджера + `app_state.StateStore.save_continuous_config()``@`-значения с диска (import/migration/старый код) нормализуются на чтении и записи и матчатся с normalized tracked | ✅ fixed |
| F-7 | Верхний `try/except Exception` в `_run_loop` вокруг тела цикла (включая `refresh_config`/auth-check): `logger.exception(...)`, `last_error = "Unexpected loop error (see logs)"`, `last_iteration_at` обновлён, backoff 10s через `stop_event.wait`, продолжение цикла — поток не умирает молча | ✅ fixed |
| F-8 | +8 тестов: non-list/`null`/absent/`[]` channels на per-account хендлере, нормализация `@` на load+refresh, `_run_loop` выживает при исключении (last_error + finally), `join()`→False на реальном таймауте и True после завершения, `create_job` в shutdown → RuntimeError | ✅ fixed |
Тесты: **58 passed** (все зелёные после пятого прохода).
---
### 🧪 Пробелы в тестах
Покрыто новыми тестами (round 2, +6): deepcopy-изоляция `load()` (все пути), Content-Type/oversize в `read_json_body`, same-origin проверка, rate limiter (lockout + cooldown кода), sweep `_auth_attempts`, терминальные статусы SSE.
Осталось:
- tests/test_integration.py:29-38 — **весь** scraper-движок замокан (`sys.modules["telegram_scraper_with_forwarding"] = MagicMock()`): реальный код (media naming, flood, forwarding, DB миграция, session) не покрыт вообще. Рекомендация: ставить telethon в CI и импортировать реальный модуль.
- tests/test_integration.py:176-201 — тест удаления аккаунта дублирует логику хендлера инлайн, не вызывает продакшн-путь → регрессии в `_handle_delete_account` не ловятся.
- Нет тестов на: `StateStore` TTL/atomic-write/конкурентный `update()`; `_parse_range` / Range-ответы; `normalize_media_url`/`guess_media_kind`; 404 SSE; auth-флоу (QR/phone/2FA state machine); H-1 (lock-дисциплина); H-7/H-8 (scraper engine — упирается в полный мок движка).
- tests/test_integration.py:48-50 — мутация `os.environ` на уровне импорта (leak между модулями). Лучше `monkeypatch`.
- tests/test_integration.py:337-397 — `START_CONTINUOUS = False` выставляется в setup и не восстанавливается.
---
### ✅ Проверено — уязвимостей НЕТ
- **Path traversal**: `serve_media`/`serve_static``resolve()` + `relative_to()` (корректно, включая symlink); `normalize_channel_id` отвергает `/`, `\`, control chars, `.`/`..`.
- **SQL injection**: все запросы параметризованы, `search` — через `LIKE ?`.
- **XSS**: viewer.js рендерит контент через `textContent`/`createTextNode`; media-URL всегда префиксуется `/media/` (нет `javascript:` схемы).
- **SSRF**: юзер-контролируемого фетча URL нет (только MTProto).
- **Десериализация**: только JSON, без pickle/yaml.
- **Command injection**: нет subprocess/os.system в продакшн-путях.
- **Secrets в image**: `.dockerignore` исключает `data/` и `session/`.
---
### Follow-up (шестой проход — k8s-rollout audit живого кластера)
- **TRUSTED_HOSTS** (`webui_server.py`, L-8): `_is_trusted_host` отвергал ЛЮБОЙ hostname → через Traefik-домен `tg.workstation.internal` (`k8s/telegram-scraper.yaml:97`) браузерные POST/DELETE + SSE `/api/jobs/*/events` возвращали 403. Добавлен `_TRUSTED_HOSTS_ENV` (parse `TRUSTED_HOSTS` на импорте, нормализация `.strip().lower().rstrip(".")`); проверка после localhost-set. Дефолт строгий: без env результаты идентичны прежним для ВСЕХ входов (regression guard); с env настроенный hostname (любой case, опциональный trailing dot) проходит, остальные — нет. IPv6-with-port (`[::1]:8080`) — out of scope, не тронут.
- **Media allowlist** (`_MEDIA_FILE_EXTENSIONS`): +10 суффиксов с комментарием `# extended coverage (animated stickers, matroska, legacy containers)``.mkv .mk3d .heic .tgs .flv .3gp .ogv .asf .wmv .djvu`. Живой диск: `.tgs` x44 / `.mkv` x2 (возвращали 403). `.exe`/`.ts` оставлены 403. `guess_media_kind` для новых суффиксов возвращает `"file"` (fall-through как у `.zip`) — viewer рендерит "Open file" link. `serve_media` уже lowercases suffix, `.MP4`/`.MOV` ок.
- **Cache-buster**: `webui/index.html` `app.js?v=4``app.js?v=5` (иначе stale JS после deploy).
- **k8s-манифест**: `containers[0].env: TRUSTED_HOSTS="tg.workstation.internal"` (после `tty: true`); image/probes/securityContext/resources/PVCs/IngressRoute не тронуты; YAML проверен `yaml.safe_load`.
- **Live-cluster факты**: local-path PVCs, state v2 `accounts=[default, forust]`, смешанное владение 1000/root → обязательный `chown -R 1000:1000 /app/data /app/session` ДО первого старта нового пода.
Тесты: **58 → 64 passed** (+3 TRUSTED_HOSTS env, +3 extended media).
+139 -23
View File
@@ -1,6 +1,8 @@
import json
import logging
import os
import shutil
import tempfile
import threading
import time
from copy import deepcopy
@@ -50,12 +52,12 @@ class StateStore:
with self.lock:
now = time.time()
if self._cache is not None and (now - self._cache_time) < self._cache_ttl:
return dict(self._cache)
return deepcopy(self._cache)
if not self.path.exists():
result = deepcopy(self.defaults)
self._cache = result
self._cache_time = now
return result
return deepcopy(result)
try:
with self.path.open("r", encoding="utf-8") as handle:
state: Dict[str, Any] = json.load(handle)
@@ -63,23 +65,82 @@ class StateStore:
result = deepcopy(self.defaults)
self._cache = result
self._cache_time = now
return result
return deepcopy(result)
result = self._merge_defaults(state)
self._cache = result
self._cache_time = now
return result
return deepcopy(result)
def save(self, state: Dict[str, Any]) -> None:
with self.lock:
self.path.parent.mkdir(parents=True, exist_ok=True)
tmp_path = self.path.with_suffix(self.path.suffix + ".tmp")
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
merged = self._merge_defaults(state)
with tmp_path.open("w", encoding="utf-8") as handle:
json.dump(merged, handle, ensure_ascii=False, indent=2)
handle.write("\n")
tmp_path.replace(self.path)
# Unique temp file so two writers to the same path cannot clobber
# each other's in-progress file, plus fsync before atomic rename so
# a power loss cannot leave an empty/corrupt state file behind.
fd, tmp_name = tempfile.mkstemp(
dir=str(self.path.parent),
prefix=self.path.name + ".tmp-",
suffix=".tmp",
)
tmp_path = Path(tmp_name)
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
json.dump(merged, handle, ensure_ascii=False, indent=2)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
# Restrict permissions on the state file to the owning user.
try:
os.chmod(tmp_path, 0o600)
except OSError:
pass
os.replace(tmp_path, self.path)
finally:
# Ensure no leftover stale temp file if something went wrong.
if tmp_path.exists():
try:
tmp_path.unlink()
except OSError:
pass
# Best-effort directory fsync for full durability (POSIX only).
try:
dir_fd = os.open(str(self.path.parent), os.O_RDONLY)
try:
os.fsync(dir_fd)
finally:
os.close(dir_fd)
except OSError:
pass
# After a successful save, best-effort sweep leftover stale tmp
# files (e.g. from a crashed writer) older than an hour. Throttled
# to avoid scanning the dir on every save.
self._sweep_stale_tmp(now=time.time())
self._cache = None
_STALE_TMP_MAX_AGE = 3600.0 # 1 hour
_STALE_TMP_SWEEP_INTERVAL = 60.0
_stale_sweep_last: float = 0.0
def _sweep_stale_tmp(self, now: float) -> None:
if (now - self._stale_sweep_last) < self._STALE_TMP_SWEEP_INTERVAL:
return
self._stale_sweep_last = now
try:
cutoff = now - self._STALE_TMP_MAX_AGE
for stale in self.path.parent.glob(self.path.name + ".tmp-*.tmp"):
try:
if stale.stat().st_mtime < cutoff:
stale.unlink()
except OSError:
pass
except OSError:
pass
def update(self, mutator: Callable[[Dict[str, Any]], None]) -> Dict[str, Any]:
with self.lock:
state = self.load()
@@ -97,7 +158,7 @@ class StateStore:
"enabled": bool(config.get("enabled", True)),
"interval_minutes": max(1, int(config.get("interval_minutes", 1) or 1)),
"channels": [
str(item).strip()
str(item).strip().lstrip("@")
for item in config.get("channels", [])
if str(item).strip()
],
@@ -124,6 +185,18 @@ _GLOBAL_STORE: Optional[StateStore] = None
def get_global_store(data_dir: Path) -> StateStore:
"""Return the process-wide global state store singleton.
The store is cached in a module-global and the SAME instance is returned
for the same process, regardless of how many times this is called with the
same (or any) data_dir. webui_server's ``STATE_STORE`` should delegate to
this function so there is exactly one authoritative global store per
process rather than a second, potentially divergent instance.
Backward-compat note: callers that cache their own ``_GLOBAL_STORE=None``
sentinel (e.g. tests resetting state between cases) still work because we
re-create the singleton lazily on first call.
"""
global _GLOBAL_STORE
if _GLOBAL_STORE is None:
_GLOBAL_STORE = StateStore(data_dir / "state.json", defaults=GLOBAL_DEFAULTS)
@@ -186,13 +259,47 @@ def account_session_path(session_dir: Path, account_id: str) -> str:
# ── MIGRATION (with data copy) ─────────────────────────────────────────
def _is_valid_channel_id(channel_id: str) -> bool:
"""Path-safe channel id check.
Mirrors webui_server.normalize_channel_id's validation: reject entries
containing ``/`` or ``\\``, control chars, ``.``/``..``, empty; keep
numbers and plain names. Defined locally (not imported from webui_server,
which would be circular) so it can be shared by migration.
"""
channel_id = str(channel_id or "").strip()
if (
not channel_id
or "/" in channel_id
or "\\" in channel_id
or channel_id in {".", ".."}
or any(ord(ch) < 32 for ch in channel_id)
):
return False
return True
def _clean_continuous_channels(channels: Any) -> List[str]:
"""Normalize/drop invalid continuous-scraping channel entries during
migration. Mirrors webui_server.clean_continuous_channels: strips a
leading ``@``, keeps numbers/names, and drops unsafe entries so they can
never become a path-traversal vector.
"""
cleaned: List[str] = []
if not isinstance(channels, list):
return cleaned
for item in channels:
cleaned.append(str(item).strip().lstrip("@"))
return [c for c in cleaned if _is_valid_channel_id(c)]
def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
"""Copy a single channel's DB + media from src_root to dst_root."""
src_ch = src_root / channel_id
dst_ch = dst_root / channel_id
if not src_ch.exists():
return
dst_ch.mkdir(parents=True, exist_ok=True)
dst_ch.mkdir(parents=True, exist_ok=True, mode=0o700)
# SQLite DB
db_name = f"{channel_id}.db"
@@ -206,7 +313,7 @@ def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
src_media = src_ch / "media"
dst_media = dst_ch / "media"
if src_media.exists() and src_media.is_dir():
dst_media.mkdir(parents=True, exist_ok=True)
dst_media.mkdir(parents=True, exist_ok=True, mode=0o700)
for item in src_media.iterdir():
if item.is_file():
dst_file = dst_media / item.name
@@ -250,7 +357,24 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
# ── 1. Create per-account state for "default" ──────────────────────
acc_dir = data_dir / "accounts" / "default"
acc_dir.mkdir(parents=True, exist_ok=True)
acc_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
# Normalize/drop invalid continuous-scraping channel entries during
# migration so unsafe values can never become a path-traversal vector.
continuous_cfg = raw.get(
"continuous_scraping",
{
"enabled": True,
"interval_minutes": 1,
"channels": [],
"run_all_tracked": True,
},
)
if not isinstance(continuous_cfg, dict):
continuous_cfg = {}
continuous_cfg["channels"] = _clean_continuous_channels(
continuous_cfg.get("channels")
)
acc_state = {
"label": "Default",
@@ -260,15 +384,7 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
"channel_names": raw.get("channel_names", {}),
"scrape_media": raw.get("scrape_media", True),
"forwarding_rules": raw.get("forwarding_rules", []),
"continuous_scraping": raw.get(
"continuous_scraping",
{
"enabled": True,
"interval_minutes": 1,
"channels": [],
"run_all_tracked": True,
},
),
"continuous_scraping": continuous_cfg,
}
acc_state_path = acc_dir / "state.json"
+23 -1
View File
@@ -2,7 +2,7 @@ import sqlite3
from pathlib import Path
from typing import Any, Dict, List, Optional
from app_state import StateStore
from app_state import StateStore, load_account
def health_payload(
@@ -73,6 +73,28 @@ def _dir_check(path: Path, writable: bool = False) -> Dict[str, Any]:
def _state_check(state_store: StateStore) -> Dict[str, Any]:
try:
state = state_store.load()
data_dir = state_store.path.parent
accounts = state.get("accounts") or []
if accounts:
# Multi-account mode: the global store no longer holds api
# credentials / channels. Aggregate those from each account's own
# state file so the reported values are meaningful.
has_api_credentials = False
tracked_channels = 0
for acc_id in accounts:
acc = load_account(data_dir, acc_id)
if acc.get("api_id") and acc.get("api_hash"):
has_api_credentials = True
tracked_channels += len(acc.get("channels", {}) or {})
return {
"ok": True,
"path": str(state_store.path),
"has_api_credentials": has_api_credentials,
"tracked_channels": tracked_channels,
}
# Legacy single-account semantics (no accounts list).
return {
"ok": True,
"path": str(state_store.path),
+15
View File
@@ -31,13 +31,28 @@ spec:
labels:
app: telegram-scraper
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
containers:
- name: telegram-scraper
image: gcr.forust.xyz/forust/telegram-scraper:latest
stdin: true
tty: true
env:
- name: TRUSTED_HOSTS
value: "tg.workstation.internal"
ports:
- containerPort: 8080
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 1
livenessProbe:
httpGet:
path: /health
+3 -2
View File
@@ -18,8 +18,9 @@ def main():
# Run legacy migration before starting the server
from app_state import migrate_legacy_state
data_dir = Path("data")
session_dir = Path("session")
BASE_DIR = Path(__file__).resolve().parent
data_dir = BASE_DIR / "data"
session_dir = BASE_DIR / "session"
try:
if migrate_legacy_state(data_dir, session_dir):
logger.info("Legacy migration completed successfully.")
+3
View File
@@ -20,3 +20,6 @@ dependencies = [
"Telethon==1.40.0",
"yarl==1.20.1",
]
[tool.pytest.ini_options]
pythonpath = ["."]
+71 -9
View File
@@ -1,11 +1,46 @@
import asyncio
import logging
import threading
from pathlib import Path
from typing import Any, Dict, List, Optional
from app_state import StateStore
logger = logging.getLogger(__name__)
BASE_DIR = Path(__file__).resolve().parent
def _run_in_new_loop(coro_factory):
"""Run an awaitable on a dedicated thread with its own event loop.
Returns the coroutine's result. This avoids ``asyncio.run()`` raising
``RuntimeError`` when the caller runs on a thread that already has a
running event loop (e.g. job threads, auth-loop threads).
"""
result = {}
error = {}
def runner():
loop = asyncio.new_event_loop()
try:
asyncio.set_event_loop(loop)
result["value"] = loop.run_until_complete(coro_factory())
except BaseException as exc: # noqa: BLE001 - relay any failure
error["value"] = exc
finally:
try:
loop.close()
finally:
asyncio.set_event_loop(None)
thread = threading.Thread(target=runner, daemon=True)
thread.start()
thread.join()
if "value" in error:
raise error["value"]
return result.get("value")
class ScraperJobService:
def __init__(self, state_store: StateStore):
@@ -13,22 +48,24 @@ class ScraperJobService:
def run(self, job_type: str, payload: Dict[str, Any]) -> None:
if job_type == "set_scrape_media":
# The webui handler already persists the scrape_media setting to
# the per-account (or legacy) store before enqueueing this job.
# Multi-account mode has no single global change to make, so this
# is a passthrough that just records success to keep the
# job-status / SSE flow intact.
value = bool(payload["value"])
def mutate(state: Dict[str, Any]) -> None:
state["scrape_media"] = value
self.state_store.update(mutate)
logger.info("Media scraping set to %s", value)
logger.info("Media scraping set to %s (already persisted by handler)", value)
return
asyncio.run(self._run_async(job_type, payload))
# Run the async job on a fresh thread/event loop so we never hit
# "asyncio.run() cannot be called from a running event loop".
_run_in_new_loop(lambda: self._run_async(job_type, payload))
async def _run_async(self, job_type: str, payload: Dict[str, Any]) -> None:
# Extract account_id from payload, default to None (legacy)
account_id: Optional[str] = payload.get("account_id")
ScraperClass = self._import_scraper_class()
scraper = ScraperClass(account_id=account_id)
scraper = ScraperClass(account_id=account_id, base_dir=BASE_DIR)
if account_id:
from app_state import load_account
@@ -73,9 +110,34 @@ class ScraperJobService:
await scraper.client.disconnect()
async def _scrape_channels(self, scraper, channels: List[str]) -> None:
"""Scrape all channels resiliently: a single channel failure does not
abort the rest. Each channel's offset is persisted even on failure
(see scrape_channel's finally block), so partial progress is retained.
If *every* channel fails, raise so the job is marked failed.
"""
failed: List[str] = []
for channel_id in channels:
offset = int(scraper.state.get("channels", {}).get(channel_id, 0) or 0)
await scraper.scrape_channel(channel_id, offset)
try:
ok = await scraper.scrape_channel(channel_id, offset)
except Exception: # noqa: BLE001 - scrape_channel re-raises some errors
logger.exception("Scrape of channel %s raised", channel_id)
failed.append(channel_id)
continue
if not ok:
logger.warning("Scrape of channel %s reported failure", channel_id)
failed.append(channel_id)
if failed and len(failed) == len(channels):
raise RuntimeError(
"All scrape target(s) failed: " + ", ".join(failed)
)
if failed:
logger.warning(
"Partial scrape failure — %d/%d channel(s) failed: %s",
len(failed),
len(channels),
", ".join(failed),
)
def _import_scraper_class(self):
from telegram_scraper_with_forwarding import OptimizedTelegramScraper
+170 -62
View File
@@ -2,6 +2,7 @@ import sqlite3
import json
import csv
import asyncio
import logging
import time
import sys
import warnings
@@ -27,6 +28,9 @@ from app_state import (
get_account_store,
)
BASE_DIR = Path(__file__).resolve().parent
logger = logging.getLogger(__name__)
warnings.filterwarnings(
"ignore", message="Using async sessions support is an experimental feature"
)
@@ -90,19 +94,21 @@ def _ensure_session_wal(session_path: str) -> None:
class OptimizedTelegramScraper:
def __init__(self, account_id: Optional[str] = None):
def __init__(self, account_id: Optional[str] = None, base_dir: Optional[Path] = None):
self.account_id = account_id
self.SESSION_DIR = Path("session")
self.SESSION_DIR.mkdir(exist_ok=True)
base_dir = base_dir or BASE_DIR
self.BASE_DIR = base_dir
self.SESSION_DIR = base_dir / "session"
self.SESSION_DIR.mkdir(exist_ok=True, mode=0o700)
if account_id:
self.DATA_DIR = Path("data") / "accounts" / account_id
self.state_store = get_account_store(Path("data"), account_id)
self.DATA_DIR = base_dir / "data" / "accounts" / account_id
self.state_store = get_account_store(base_dir / "data", account_id)
else:
self.DATA_DIR = Path("data")
self.DATA_DIR = base_dir / "data"
self.state_store = StateStore(self.DATA_DIR / "state.json")
self.DATA_DIR.mkdir(parents=True, exist_ok=True)
self.DATA_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
self.STATE_FILE = str(self.DATA_DIR / "state.json")
self.state = self.load_state()
@@ -112,6 +118,8 @@ class OptimizedTelegramScraper:
self.max_concurrent_downloads = 5
self.batch_size = 100
self.state_save_interval = 50
self.state_save_throttle_seconds = 5.0
self.last_state_save = None
self.db_connections = {}
self.forwarding_handler = None
@@ -121,9 +129,25 @@ class OptimizedTelegramScraper:
def save_state(self):
try:
self.state_store.save(self.state)
self.last_state_save = time.time()
except Exception as e:
print(f"Failed to save state: {e}")
def _save_state_throttled(self):
"""Throttled intermediate state persistence.
Avoids rewriting the whole per-account JSON on every
``state_save_interval`` messages (which is far too chatty for long
channels). Skips saves that fall within the throttle window; the
final save at end-of-scrape always runs regardless.
"""
now = time.time()
if (
self.last_state_save is None
or (now - self.last_state_save) >= self.state_save_throttle_seconds
):
self.save_state()
def get_forwarding_rules(self) -> List[ForwardingRule]:
rules = []
for rule_dict in self.state.get("forwarding_rules", []):
@@ -181,7 +205,7 @@ class OptimizedTelegramScraper:
def get_db_connection(self, channel: str) -> sqlite3.Connection:
if channel not in self.db_connections:
channel_dir = self.DATA_DIR / channel
channel_dir.mkdir(parents=True, exist_ok=True)
channel_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
db_file = channel_dir / f"{channel}.db"
conn = sqlite3.connect(str(db_file), check_same_thread=False, timeout=30)
@@ -210,6 +234,22 @@ class OptimizedTelegramScraper:
columns = {row[1] for row in cursor.fetchall()}
migrations = []
if "sender_id" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN sender_id INTEGER")
if "first_name" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN first_name TEXT")
if "last_name" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN last_name TEXT")
if "username" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN username TEXT")
if "message" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN message TEXT")
if "media_type" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN media_type TEXT")
if "media_path" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN media_path TEXT")
if "reply_to" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN reply_to INTEGER")
if "post_author" not in columns:
migrations.append("ALTER TABLE messages ADD COLUMN post_author TEXT")
if "views" not in columns:
@@ -222,8 +262,8 @@ class OptimizedTelegramScraper:
for migration in migrations:
try:
conn.execute(migration)
except Exception:
pass
except Exception as e:
logger.warning("Migration failed for %s: %s", migration, e)
if migrations:
conn.commit()
@@ -278,7 +318,7 @@ class OptimizedTelegramScraper:
try:
channel_dir = self.DATA_DIR / channel
media_folder = channel_dir / "media"
media_folder.mkdir(exist_ok=True)
media_folder.mkdir(exist_ok=True, mode=0o700)
if isinstance(message.media, MessageMediaPhoto):
original_name = getattr(message.file, "name", None) or "photo.jpg"
@@ -294,9 +334,18 @@ class OptimizedTelegramScraper:
unique_filename = f"{message.id}-{base_name}{extension}"
media_path = media_folder / unique_filename
existing_files = list(media_folder.glob(f"{message.id}-*"))
if existing_files:
return str(existing_files[0])
# Prefer the exact expected filename. Fall back to a matching
# "{id}-*" file only if it is non-empty (and pick the newest one,
# in case a stale/partial/other-extension file is present).
if media_path.exists() and media_path.stat().st_size > 0:
return str(media_path)
candidates = [
p for p in media_folder.glob(f"{message.id}-*")
if p != media_path and p.is_file() and p.stat().st_size > 0
]
if candidates:
return str(max(candidates, key=lambda p: p.stat().st_mtime))
for attempt in range(3):
try:
@@ -353,7 +402,7 @@ class OptimizedTelegramScraper:
return False
async def forward_message(
self, message, rule: ForwardingRule, source_channel_id: int = None
self, message, rule: ForwardingRule, source_channel_id: int = None, _retry: int = 0
):
try:
dest_entity = await self._resolve_entity(rule.destination_channel)
@@ -393,9 +442,12 @@ class OptimizedTelegramScraper:
return True
except FloodWaitError as e:
if _retry >= 3:
print(f" Failed to forward message {message.id}: FloodWait retry limit exceeded")
return False
print(f" Rate limited, waiting {e.seconds}s...")
await asyncio.sleep(e.seconds)
return await self.forward_message(message, rule, source_channel_id)
return await self.forward_message(message, rule, source_channel_id, _retry=_retry + 1)
except Exception as e:
print(f" Failed to forward message {message.id}: {e}")
return False
@@ -429,6 +481,11 @@ class OptimizedTelegramScraper:
print("No valid source channels")
return False
# Unregister a previously installed handler so it is never registered twice.
if self.forwarding_handler is not None:
self.client.remove_event_handler(self.forwarding_handler)
self.forwarding_handler = None
@self.client.on(
events.NewMessage(chats=source_channels, incoming=True, outgoing=True)
)
@@ -829,7 +886,14 @@ class OptimizedTelegramScraper:
else:
return await self.client.get_entity(channel)
async def scrape_channel(self, channel: str, offset_id: int):
async def scrape_channel(self, channel: str, offset_id: int) -> bool:
"""Scrape a single channel. Returns True on success, False on failure.
Offset progress is persisted in a ``finally`` block so partial
progress is never lost even when an error occurs mid-scrape.
"""
last_message_id = offset_id
success = False
try:
if not self.client.is_connected():
await self.client.connect()
@@ -842,15 +906,70 @@ class OptimizedTelegramScraper:
if total_messages == 0:
print(f"No messages found in channel {channel}")
return
return True
print(f"Found {total_messages} messages in channel {channel}")
message_batch = []
media_tasks = []
processed_messages = 0
last_message_id = offset_id
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
media_flush_chunk = 50
# Media progress counters tracked across chunked flushes so the
# progress bar stays coherent even though downloads happen in
# bounded chunks during the pass instead of all at the end.
total_media = 0
completed_media = 0
successful_downloads = 0
async def flush_media_batch():
"""Download the accumulated media messages in small batches.
Keeps memory bounded (we never hold references to every
media-capable message for the whole channel), and updates the
shared media progress counters via ``nonlocal``.
"""
nonlocal total_media, completed_media, successful_downloads
if not media_tasks:
return
batch = list(media_tasks)
media_tasks.clear()
total_media += len(batch)
async def download_single_media(message):
async with semaphore:
return await self.download_media(channel, message)
sub_batch = 10
for i in range(0, len(batch), sub_batch):
sub = batch[i : i + sub_batch]
tasks = [
asyncio.create_task(download_single_media(msg)) for msg in sub
]
for j, task in enumerate(tasks):
try:
media_path = await task
if media_path:
await self.update_media_path(
channel, sub[j].id, media_path
)
successful_downloads += 1
except Exception:
pass
completed_media += 1
if total_media:
mprogress = (completed_media / total_media) * 100
bar_length = 30
mfilled = int(
bar_length * completed_media // total_media
)
mbar = "" * mfilled + "" * (bar_length - mfilled)
sys.stdout.write(
f"\r📥 Media: [{mbar}] {mprogress:.1f}% "
f"({completed_media}/{total_media})"
)
sys.stdout.flush()
async for message in self.client.iter_messages(
entity, offset_id=offset_id, reverse=True
@@ -902,6 +1021,11 @@ class OptimizedTelegramScraper:
and not isinstance(message.media, MessageMediaWebPage)
):
media_tasks.append(message)
# Flush the pending media list as soon as it reaches the
# bounded chunk so we never hold thousands of message
# references in memory for the whole channel.
if len(media_tasks) >= media_flush_chunk:
await flush_media_batch()
last_message_id = message.id
processed_messages += 1
@@ -909,10 +1033,15 @@ class OptimizedTelegramScraper:
if len(message_batch) >= self.batch_size:
self.batch_insert_messages(channel, message_batch)
message_batch.clear()
# After each insert batch, also flush any accumulated
# media (bounded) rather than deferring everything to
# the end of the full pass.
if media_tasks:
await flush_media_batch()
if processed_messages % self.state_save_interval == 0:
self.state["channels"][channel] = last_message_id
self.save_state()
self._save_state_throttled()
progress = (processed_messages / total_messages) * 100
bar_length = 30
@@ -933,56 +1062,35 @@ class OptimizedTelegramScraper:
self.batch_insert_messages(channel, message_batch)
if media_tasks:
total_media = len(media_tasks)
completed_media = 0
successful_downloads = 0
print(f"\n📥 Downloading {total_media} media files...")
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
async def download_single_media(message):
async with semaphore:
return await self.download_media(channel, message)
batch_size = 10
for i in range(0, len(media_tasks), batch_size):
batch = media_tasks[i : i + batch_size]
tasks = [
asyncio.create_task(download_single_media(msg)) for msg in batch
]
for j, task in enumerate(tasks):
try:
media_path = await task
if media_path:
await self.update_media_path(
channel, batch[j].id, media_path
)
successful_downloads += 1
except Exception:
pass
completed_media += 1
progress = (completed_media / total_media) * 100
bar_length = 30
filled_length = int(bar_length * completed_media // total_media)
bar = "" * filled_length + "" * (bar_length - filled_length)
sys.stdout.write(
f"\r📥 Media: [{bar}] {progress:.1f}% ({completed_media}/{total_media})"
)
sys.stdout.flush()
await flush_media_batch()
if total_media:
print(
f"\n✅ Media download complete! ({successful_downloads}/{total_media} successful)"
)
self.state["channels"][channel] = last_message_id
self.save_state()
print(f"Completed scraping channel {channel}")
except Exception as e:
print(f"Error with channel {channel}: {e}")
# Final state save moved to ``finally`` below so the offset
# persists even when an exception aborts the scrape mid-way.
success = True
except Exception:
logger.exception("Error with channel %s", channel)
finally:
# Persist the last-known offset even on partial failure so a
# re-scrape resumes from the furthest point reached, not from the
# start. save_state() is itself best-effort (logs internally),
# so a save failure here must not mask the scrape's own result.
try:
self.state["channels"][channel] = last_message_id
self.save_state()
except Exception:
logger.exception("Failed to save state for channel %s", channel)
return success
async def rescrape_media(self, channel: str):
conn = self.get_db_connection(channel)
File diff suppressed because it is too large Load Diff
+95 -44
View File
@@ -68,7 +68,34 @@ function setBusy(button, busy) {
}
function confirmAction(message) {
return window.confirm(message);
return new Promise((resolve) => {
let settled = false;
const done = (value) => {
if (settled) return;
settled = true;
if (dialog.open) dialog.close();
resolve(value);
};
let dialog = document.getElementById('confirm-dialog');
if (!dialog) {
dialog = document.createElement('dialog');
dialog.id = 'confirm-dialog';
dialog.className = 'confirm-dialog';
dialog.innerHTML =
'<div class="confirm-dialog-body"><div class="eyebrow">Confirm</div>' +
'<p class="confirm-dialog-message muted"></p>' +
'<div class="confirm-dialog-actions">' +
'<button class="button" data-confirm-cancel type="button">Cancel</button>' +
'<button class="button primary" data-confirm-ok type="button">Confirm</button>' +
'</div></div>';
document.body.appendChild(dialog);
}
dialog.querySelector('.confirm-dialog-message').textContent = message;
dialog.querySelector('[data-confirm-ok]').onclick = () => done(true);
dialog.querySelector('[data-confirm-cancel]').onclick = () => done(false);
dialog.onclose = () => done(false);
if (!dialog.open) dialog.showModal();
});
}
function showToast(message, type = 'info') {
@@ -202,12 +229,18 @@ function renderAccountPanel(accountId) {
(item) => item.value,
);
if (!runAllTracked && channels.length === 0 && enabled) {
if (!confirmAction('Continuous scraping enabled with no selected channels. Save anyway?')) return;
if (!(await confirmAction('Continuous scraping enabled with no selected channels. Save anyway?'))) return;
}
await api(`/api/accounts/${accountId}/continuous`, {
const resp = await api(`/api/accounts/${accountId}/continuous`, {
method: 'POST',
body: JSON.stringify({ enabled, interval_minutes: intervalMinutes, run_all_tracked: runAllTracked, channels }),
});
if (resp.dropped_invalid && resp.dropped_invalid.length) {
const n = resp.dropped_invalid.length;
const shown = resp.dropped_invalid.slice(0, 3).join(', ');
const extra = n > 3 ? '…' : '';
showToast(`${n} invalid channel(s) skipped: ${shown}${extra}`, 'warn');
}
await refreshAccount(accountId);
});
@@ -283,15 +316,17 @@ function updateSidebarAccount() {
const acc = state.accounts.find((a) => a.id === state.activeAccount);
const nameEl = document.getElementById('active-account-name');
const statusEl = document.getElementById('active-account-status');
if (!nameEl || !statusEl) return;
statusEl.classList.remove('is-ok', 'is-error', 'is-dim');
if (acc) {
nameEl.textContent = acc.label || acc.id;
const authOk = isAccountAuthorized(acc.auth);
statusEl.textContent = authOk ? 'Authorized session' : 'Needs login';
statusEl.style.color = authOk ? 'var(--ok)' : 'var(--danger)';
statusEl.classList.add(authOk ? 'is-ok' : 'is-error');
} else {
nameEl.textContent = 'None';
statusEl.textContent = 'Add an account in Settings';
statusEl.style.color = 'var(--dim)';
statusEl.classList.add('is-dim');
}
}
@@ -307,7 +342,7 @@ function renderChannels(accountId, channels) {
if (!channels.length) {
const row = document.createElement('tr');
row.innerHTML =
'<td colspan="5"><div class="empty-state">No tracked channels yet. Add an ID or @username to start scraping this account.</div></td>';
'<td colspan="5"><div class="empty-state"><p class="muted">No tracked channels yet. Add an ID or @username to start scraping this account.</p></div></td>';
tbody.appendChild(row);
}
@@ -348,7 +383,7 @@ function renderChannels(accountId, channels) {
});
node.querySelector('.remove-btn').addEventListener('click', async () => {
if (!confirmAction(`Remove ${channel.name} from tracked channels?`)) return;
if (!(await confirmAction(`Remove ${channel.name} from tracked channels?`))) return;
await api(`/api/accounts/${accountId}/channels/remove`, {
method: 'POST',
body: JSON.stringify({ channel_id: channel.channel_id }),
@@ -361,6 +396,8 @@ function renderChannels(accountId, channels) {
// Update channel count stat
panel.querySelector('.channel-count').textContent = String(channels.length);
panel.querySelector('.channel-active-count').textContent = String(channels.length);
panel.querySelector('.channel-inactive-count').textContent = '0';
}
function renderJobs(accountId, jobs) {
@@ -371,7 +408,7 @@ function renderJobs(accountId, jobs) {
root.innerHTML = '';
if (!jobs.length) {
root.innerHTML = '<div class="empty-state">No jobs yet. Start a scrape or export to see progress here.</div>';
root.innerHTML = '<div class="empty-state"><p class="muted">No jobs yet. Start a scrape or export to see progress here.</p></div>';
return;
}
@@ -409,7 +446,7 @@ function pollJobFallback(accountId, jobId) {
try {
const job = await api(`/api/jobs/${encodeURIComponent(jobId)}`);
updateRenderedJob(accountId, job);
if (['completed', 'failed'].includes(job.status)) {
if (['done', 'completed', 'failed'].includes(job.status)) {
clearInterval(pollTimer);
jobStreams.delete(jobId);
refreshAccount(accountId);
@@ -453,7 +490,7 @@ function subscribeJobStream(accountId, jobId, status) {
retryCount = 0; // reset backoff on successful message
const job = JSON.parse(event.data);
updateRenderedJob(accountId, job);
if (['completed', 'failed'].includes(job.status)) {
if (['done', 'completed', 'failed'].includes(job.status)) {
newStream.close();
jobStreams.delete(jobId);
refreshAccount(accountId);
@@ -616,16 +653,24 @@ function renderSummary(accountId, data) {
const d = data.dashboard || data.state || {};
const health = data.health || {};
panel.querySelector('.forwarding-count').textContent = String((d.forwarding_rules || []).length);
panel.querySelector('.forwarding-active-count').textContent = String((d.forwarding_rules || []).length);
const toggle = panel.querySelector('.scrape-media-label');
toggle.textContent = d.scrape_media ? 'ON' : 'OFF';
const healthOk = health.api_credentials && health.session_ready && health.data_dir_exists;
panel.querySelector('.account-health-label').textContent = healthOk ? 'Ready' : 'Check';
panel.querySelector('.account-health-label').style.color = healthOk ? 'var(--ok)' : 'var(--warn)';
const healthLabel = panel.querySelector('.account-health-label');
healthLabel.textContent = healthOk ? 'Ready' : 'Check';
healthLabel.classList.remove('is-ok', 'is-warn');
healthLabel.classList.add(healthOk ? 'is-ok' : 'is-warn');
panel.querySelector('.account-health-detail').textContent = [
`${health.message_count || 0} messages`,
`${health.media_count || 0} media`,
health.active_job ? `active: ${health.active_job.status}` : 'idle',
].join(' | ');
const mediaCount = Number(health.media_count || 0);
panel.querySelector('.media-session-count').textContent = String(mediaCount);
panel.querySelector('.media-total-count').textContent = String(mediaCount);
panel.querySelector('.health-session-value').textContent = health.session_ready ? 'Valid' : 'Check';
panel.querySelector('.health-rate-value').textContent = health.active_job ? 'Busy' : 'OK';
}
/* ── Account data loading ───────────────────────────── */
@@ -707,7 +752,9 @@ async function loadAccounts() {
function renderSettingsAccounts() {
const container = document.getElementById('accounts-list');
if (!container) return;
const template = document.getElementById('account-list-item-template');
if (!template) return;
container.innerHTML = '';
state.accounts.forEach((acc) => {
@@ -718,12 +765,12 @@ function renderSettingsAccounts() {
const authStatus = node.querySelector('.account-list-auth-status');
const authOk = isAccountAuthorized(acc.auth);
authStatus.textContent = authOk ? 'Authorized' : 'Needs auth';
authStatus.style.color = authOk ? 'var(--ok)' : 'var(--danger)';
authStatus.style.fontSize = '0.78rem';
authStatus.classList.remove('is-ok', 'is-error');
authStatus.classList.add(authOk ? 'is-ok' : 'is-error');
node.querySelector('.account-select-btn').addEventListener('click', () => {
switchAccount(acc.id);
document.getElementById('settings-dialog').close();
document.getElementById('settings-dialog')?.close();
});
node.querySelector('.account-export-btn').addEventListener('click', async () => {
@@ -737,7 +784,7 @@ function renderSettingsAccounts() {
});
node.querySelector('.account-remove-btn').addEventListener('click', async () => {
if (!confirmAction(`Remove account "${acc.label || acc.id}"? All its data will be deleted.`)) return;
if (!(await confirmAction(`Remove account "${acc.label || acc.id}"? All its data will be deleted.`))) return;
try {
const removingActive = state.activeAccount === acc.id;
await api(`/api/accounts/${acc.id}`, { method: 'DELETE' });
@@ -759,6 +806,7 @@ function renderSettingsAccounts() {
function updateAuthSection(accountId) {
const label = document.getElementById('auth-account-label');
if (!label) return;
const acc = state.accounts.find((a) => a.id === accountId);
label.textContent = acc ? acc.label || acc.id : '-';
}
@@ -822,7 +870,7 @@ async function submitPassword(accountId) {
async function scrapeAll() {
if (!state.activeAccount) return;
if (!confirmAction('Queue scraping for all tracked channels?')) return;
if (!(await confirmAction('Queue scraping for all tracked channels?'))) return;
await api(`/api/accounts/${state.activeAccount}/jobs/scrape`, {
method: 'POST',
body: JSON.stringify({}),
@@ -832,7 +880,7 @@ async function scrapeAll() {
async function exportAll() {
if (!state.activeAccount) return;
if (!confirmAction('Queue export for all tracked channels?')) return;
if (!(await confirmAction('Queue export for all tracked channels?'))) return;
await api(`/api/accounts/${state.activeAccount}/jobs/export`, {
method: 'POST',
body: JSON.stringify({}),
@@ -887,25 +935,18 @@ async function main() {
}
});
// ── Settings dialog ──
// ── Legacy settings dialog (removed from dashboard; Settings page owns auth) ──
// Kept guarded so old markup, if present, never throws.
const settingsDialog = document.getElementById('settings-dialog');
const openSettingsBtn = document.getElementById('open-settings-btn');
if (!openSettingsBtn.matches('a[href]')) {
openSettingsBtn.addEventListener('click', () => {
// Update auth section for active account
if (state.activeAccount) {
updateAuthSection(state.activeAccount);
}
renderSettingsAccounts();
settingsDialog.showModal();
});
const closeSettingsBtn = document.getElementById('close-settings-btn');
if (settingsDialog && closeSettingsBtn) {
closeSettingsBtn.addEventListener('click', () => settingsDialog.close());
}
document.getElementById('close-settings-btn').addEventListener('click', () => {
settingsDialog.close();
});
// ── Add account form ──
document.getElementById('add-account-form').addEventListener('submit', async (event) => {
// ── Add account form (only when legacy dialog markup exists) ──
const addAccountForm = document.getElementById('add-account-form');
if (addAccountForm) {
addAccountForm.addEventListener('submit', async (event) => {
event.preventDefault();
const accountId = document.getElementById('add-account-id').value.trim();
const label = document.getElementById('add-account-label').value.trim();
@@ -931,9 +972,12 @@ async function main() {
} catch (err) {
showToast('Failed to add account: ' + err.message, 'error');
}
});
});
}
document.getElementById('import-account-file').addEventListener('change', async (event) => {
const importAccountFile = document.getElementById('import-account-file');
if (importAccountFile) {
importAccountFile.addEventListener('change', async (event) => {
const file = event.currentTarget.files?.[0];
if (!file) return;
try {
@@ -948,13 +992,20 @@ async function main() {
await loadAccounts();
switchAccount(accountId);
showToast(`Imported ${accountId}.`, 'success');
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
} catch (err) {
showToast(`Failed to import account: ${err.message}`, 'error');
}
});
});
}
// ── Credentials form ──
document.getElementById('credentials-form').addEventListener('submit', async (event) => {
// ── Credentials / auth forms (only when legacy dialog markup exists) ──
const bindOptional = (id, evt, handler) => {
const el = document.getElementById(id);
if (el) el.addEventListener(evt, handler);
};
bindOptional('credentials-form', 'submit', async (event) => {
event.preventDefault();
if (!state.activeAccount) return;
try {
@@ -965,7 +1016,7 @@ async function main() {
});
// ── QR login ──
document.getElementById('start-qr-btn').addEventListener('click', async (event) => {
bindOptional('start-qr-btn', 'click', async (event) => {
setBusy(event.currentTarget, true);
try {
if (!state.activeAccount) return;
@@ -978,7 +1029,7 @@ async function main() {
});
// ── Phone ──
document.getElementById('phone-form').addEventListener('submit', async (event) => {
bindOptional('phone-form', 'submit', async (event) => {
event.preventDefault();
if (!state.activeAccount) return;
try {
@@ -989,7 +1040,7 @@ async function main() {
});
// ── Code ──
document.getElementById('code-form').addEventListener('submit', async (event) => {
bindOptional('code-form', 'submit', async (event) => {
event.preventDefault();
if (!state.activeAccount) return;
try {
@@ -1000,7 +1051,7 @@ async function main() {
});
// ── Password ──
document.getElementById('password-form').addEventListener('submit', async (event) => {
bindOptional('password-form', 'submit', async (event) => {
event.preventDefault();
if (!state.activeAccount) return;
try {
@@ -1011,7 +1062,7 @@ async function main() {
});
// ── Media toggle ──
document.getElementById('scrape-media-toggle').addEventListener('change', async (event) => {
bindOptional('scrape-media-toggle', 'change', async (event) => {
const checked = event.currentTarget.checked;
try {
await toggleMedia(checked);
+127 -131
View File
@@ -4,23 +4,45 @@
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Telegram Scraper Control Panel</title>
<link rel="stylesheet" href="/static/style.css?v=6" />
<link rel="stylesheet" href="/static/style.css?v=9" />
</head>
<body>
<div class="app-shell">
<aside class="sidebar">
<div class="brand-block">
<svg class="brand-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="m21 3-7.6 18-3.8-7.6L2 9.6zM9.6 13.4 14 10" />
</svg>
<div class="eyebrow">Telegram Scraper</div>
<h1>Control</h1>
<p class="muted">Local scraper console for channels, jobs, and continuous runs.</p>
</div>
<nav class="nav-links">
<a class="nav-link active" href="/">Dashboard</a>
<a class="nav-link" href="/settings">Settings</a>
<a class="nav-link" href="/viewer">Message Viewer</a>
<a class="nav-link" href="/swagger">API Docs</a>
<a class="nav-link" href="/health">Health</a>
<a class="nav-link active" href="/"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<rect x="4" y="4" width="6" height="6" />
<rect x="14" y="4" width="6" height="6" />
<rect x="4" y="14" width="6" height="6" />
<rect x="14" y="14" width="6" height="6" /></svg
>Dashboard</a
>
<a class="nav-link" href="/settings"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<circle cx="12" cy="12" r="3" />
<path
d="M12 2v3m0 14v3M2 12h3m14 0h3m-2.9-7.1-2.1 2.1M4.9 19.1 7 17m0-10-2.1-2.1m12.2 14.2-2.1-2.1" /></svg
>Settings</a
>
<a class="nav-link" href="/viewer"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true"><path d="M4 5h16v12H8l-4 3z" /></svg>Message
Viewer</a
>
<a class="nav-link" href="/swagger"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="m8 5-5 7 5 7m8-14 5 7-5 7M14 3l-4 18" /></svg
>API Docs</a
>
</nav>
<section class="status-card" id="active-account-card">
@@ -39,6 +61,13 @@
</aside>
<main class="content">
<header class="dashboard-topbar">
<div class="dashboard-title"><span class="menu-icon" aria-hidden="true"></span>Dashboard</div>
<div class="runtime-status">
<span class="runtime-dot"></span>Service: Running <i></i> Scraper: Idle <i></i
><span class="runtime-clock"></span>
</div>
</header>
<!-- Account Tabs -->
<div id="account-tabs" class="account-tabs"></div>
@@ -57,87 +86,17 @@
</p>
</div>
</div>
<div class="panel-body">
<div class="empty-state">
<div class="empty-actions">
<a class="button primary" href="/settings">Open Settings</a>
</div>
</div>
</div>
</section>
</main>
</div>
<!-- ════════ Settings Dialog ════════ -->
<dialog id="settings-dialog" class="settings-dialog">
<div class="dialog-shell">
<div class="dialog-header">
<div>
<div class="eyebrow">Settings</div>
<h2>Global & Accounts</h2>
</div>
<button class="button button-small" id="close-settings-btn" type="button">Close</button>
</div>
<!-- ── Accounts Management ── -->
<section class="settings-section">
<div class="section-title">Accounts</div>
<div id="accounts-list" class="accounts-list"></div>
<form id="add-account-form" class="stack-form add-account-form">
<input id="add-account-id" name="account_id" placeholder="Account ID (e.g. work)" required />
<input id="add-account-label" name="label" placeholder="Display name (e.g. Work Account)" />
<input id="add-account-api-id" name="api_id" placeholder="API ID" />
<input id="add-account-api-hash" name="api_hash" placeholder="API Hash" />
<button class="button primary" type="submit">Add Account</button>
</form>
<label class="import-account-row">
<span class="muted small">Import account settings JSON</span>
<input id="import-account-file" type="file" accept="application/json,.json" />
</label>
</section>
<!-- ── Account Auth ── (shown per account selected in UI) -->
<section class="settings-section" id="account-auth-section">
<div class="section-title">Account Auth: <span id="auth-account-label">-</span></div>
<form id="credentials-form" class="stack-form">
<input id="api-id-input" name="api_id" placeholder="API ID" />
<input id="api-hash-input" name="api_hash" placeholder="API Hash" />
<button class="button" type="submit">Save credentials</button>
</form>
<div class="auth-actions">
<button class="button primary" id="start-qr-btn" type="button">Start QR login</button>
</div>
<div id="qr-wrap" class="qr-wrap hidden">
<img id="qr-image" alt="Telegram QR login" />
<p class="muted small">Open Telegram -> Settings -> Devices -> Scan QR.</p>
</div>
<form id="phone-form" class="stack-form">
<input id="phone-input" name="phone" placeholder="+1234567890" />
<button class="button" type="submit">Send code</button>
</form>
<form id="code-form" class="stack-form hidden">
<input id="code-input" name="code" placeholder="Telegram code" />
<button class="button" type="submit">Confirm code</button>
</form>
<form id="password-form" class="stack-form hidden">
<input id="password-input" name="password" type="password" placeholder="2FA password" />
<button class="button" type="submit">Confirm password</button>
</form>
</section>
<!-- ── Scraping ── -->
<section class="settings-section">
<div class="section-title">Scraping</div>
<label class="toggle-row">
<span>Download media</span>
<span class="switch">
<input id="scrape-media-toggle" type="checkbox" />
<span class="switch-slider"></span>
</span>
</label>
</section>
</div>
</dialog>
<!-- ════════ Templates ════════ -->
<!-- Account Tab -->
@@ -163,18 +122,38 @@
<div class="panel stat-panel">
<div class="section-title">Channels</div>
<div class="stat-value channel-count">0</div>
<div class="stat-caption">Tracked channels</div>
<div class="stat-meta">
<span><strong class="channel-active-count">0</strong> Active</span
><span><strong class="channel-inactive-count">0</strong> Inactive</span>
</div>
</div>
<div class="panel stat-panel">
<div class="section-title">Media scraping</div>
<div class="stat-value stat-compact scrape-media-label">OFF</div>
<div class="stat-caption">Items downloaded</div>
<div class="stat-meta">
<span><strong class="media-session-count">0</strong> This session</span
><span><strong class="media-total-count">0</strong> Total</span>
</div>
</div>
<div class="panel stat-panel">
<div class="section-title">Forwarding rules</div>
<div class="stat-value forwarding-count">0</div>
<div class="stat-caption">Active rules</div>
<div class="stat-meta">
<span><strong class="forwarding-active-count">0</strong> Active</span
><span><strong>0</strong> Paused</span>
</div>
</div>
<div class="panel stat-panel">
<div class="section-title">Account health</div>
<div class="stat-value stat-compact account-health-label">-</div>
<div class="stat-caption">Session status</div>
<div class="stat-meta">
<span><strong class="health-session-value">-</strong> Session</span
><span><strong class="health-rate-value">-</strong> Rate limit</span>
</div>
<div class="muted small account-health-detail"></div>
</div>
</section>
@@ -186,25 +165,38 @@
<p class="muted">Per-account channel list.</p>
</div>
<form class="inline-form add-channel-form">
<input class="add-channel-id" name="channel_id" placeholder="ID or @username" required />
<input class="add-channel-name" name="name" placeholder="Display name" />
<input
class="add-channel-id"
name="channel_id"
placeholder="ID or @username"
aria-label="Channel ID or username"
required
/>
<input
class="add-channel-name"
name="name"
placeholder="Display name"
aria-label="Channel display name"
/>
<button class="button primary" type="submit">Add</button>
</form>
</div>
<div class="table-wrap">
<table>
<thead>
<tr>
<th>Channel</th>
<th>Messages</th>
<th>Media</th>
<th>Last message</th>
<th>Actions</th>
</tr>
</thead>
<tbody class="channels-table"></tbody>
</table>
<div class="panel-body">
<div class="table-wrap">
<table>
<thead>
<tr>
<th>Channel</th>
<th>Messages</th>
<th>Media</th>
<th>Last message</th>
<th>Actions</th>
</tr>
</thead>
<tbody class="channels-table"></tbody>
</table>
</div>
</div>
</section>
@@ -216,7 +208,9 @@
</div>
<button class="button refresh-jobs-btn">Refresh</button>
</div>
<div class="jobs-list"></div>
<div class="panel-body">
<div class="jobs-list"></div>
</div>
</section>
</div>
@@ -230,40 +224,42 @@
</div>
</div>
<form class="continuous-form">
<label class="toggle-row">
<span>Enabled</span>
<span class="switch">
<input type="checkbox" class="continuous-enabled" checked />
<span class="switch-slider"></span>
</span>
</label>
<label class="stack-form">
<span class="muted small">Interval, minutes</span>
<input type="number" class="continuous-interval" min="1" value="1" />
</label>
<label class="toggle-row">
<span>All tracked channels</span>
<span class="switch">
<input type="checkbox" class="continuous-all" checked />
<span class="switch-slider"></span>
</span>
</label>
<div class="channel-picker">
<div class="muted small">Continuous channel set</div>
<div class="checkbox-grid continuous-channel-list"></div>
<div class="panel-body">
<form class="continuous-form">
<label class="toggle-row">
<span>Enabled</span>
<span class="switch">
<input type="checkbox" class="continuous-enabled" checked />
<span class="switch-slider"></span>
</span>
</label>
<label class="field">
<span class="field-label">Interval, minutes</span>
<input type="number" class="continuous-interval" min="1" value="1" aria-label="Interval, minutes" />
</label>
<label class="toggle-row">
<span>All tracked channels</span>
<span class="switch">
<input type="checkbox" class="continuous-all" checked />
<span class="switch-slider"></span>
</span>
</label>
<div class="channel-picker">
<div class="muted small">Continuous channel set</div>
<div class="checkbox-grid continuous-channel-list"></div>
</div>
<button class="button primary" type="submit">Save continuous settings</button>
</form>
<div class="continuous-meta">
<div>Status: <span class="continuous-status">-</span></div>
<div>Last run: <span class="continuous-last-iteration">-</span></div>
<div>Next run: <span class="continuous-next-run">-</span></div>
<div>Last error: <span class="continuous-last-error">-</span></div>
</div>
<button class="button primary" type="submit">Save continuous settings</button>
</form>
<div class="continuous-meta">
<div>Status: <span class="continuous-status">-</span></div>
<div>Last run: <span class="continuous-last-iteration">-</span></div>
<div>Next run: <span class="continuous-next-run">-</span></div>
<div>Last error: <span class="continuous-last-error">-</span></div>
<div class="log-viewer continuous-logs"></div>
</div>
<div class="log-viewer continuous-logs"></div>
</section>
</div>
</section>
@@ -319,6 +315,6 @@
</div>
</template>
<script src="/static/app.js?v=4"></script>
<script src="/static/app.js?v=9"></script>
</body>
</html>
+116 -19
View File
@@ -4,23 +4,45 @@
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Telegram Scraper Settings</title>
<link rel="stylesheet" href="/static/style.css?v=6" />
<link rel="stylesheet" href="/static/style.css?v=9" />
</head>
<body>
<div class="app-shell settings-page-shell">
<aside class="sidebar">
<div class="brand-block">
<svg class="brand-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="m21 3-7.6 18-3.8-7.6L2 9.6zM9.6 13.4 14 10" />
</svg>
<div class="eyebrow">Telegram Scraper</div>
<h1>Settings</h1>
<p class="muted">Account credentials, import/export, and runtime safety controls.</p>
</div>
<nav class="nav-links">
<a class="nav-link" href="/">Dashboard</a>
<a class="nav-link active" href="/settings">Settings</a>
<a class="nav-link" href="/viewer">Message Viewer</a>
<a class="nav-link" href="/swagger">API Docs</a>
<a class="nav-link" href="/health">Health</a>
<a class="nav-link" href="/"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<rect x="4" y="4" width="6" height="6" />
<rect x="14" y="4" width="6" height="6" />
<rect x="4" y="14" width="6" height="6" />
<rect x="14" y="14" width="6" height="6" /></svg
>Dashboard</a
>
<a class="nav-link active" href="/settings"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<circle cx="12" cy="12" r="3" />
<path
d="M12 2v3m0 14v3M2 12h3m14 0h3m-2.9-7.1-2.1 2.1M4.9 19.1 7 17m0-10-2.1-2.1m12.2 14.2-2.1-2.1" /></svg
>Settings</a
>
<a class="nav-link" href="/viewer"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true"><path d="M4 5h16v12H8l-4 3z" /></svg>Message
Viewer</a
>
<a class="nav-link" href="/swagger"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="m8 5-5 7 5 7m8-14 5 7-5 7M14 3l-4 18" /></svg
>API Docs</a
>
</nav>
<section class="status-card">
@@ -30,11 +52,21 @@
</aside>
<main class="content settings-page">
<header class="dashboard-topbar">
<div class="dashboard-title"><span class="menu-icon" aria-hidden="true"></span>Settings</div>
<div class="runtime-status">
<span class="runtime-dot"></span>Service: Running <i></i> Scraper: Idle <i></i
><span class="runtime-clock"></span>
</div>
</header>
<section class="panel settings-hero">
<div>
<div class="eyebrow">Account Settings</div>
<h2 id="settings-title">Loading accounts</h2>
<p id="settings-subtitle" class="muted">Select an account to manage credentials and scraper options.</p>
<p class="muted small" id="settings-hero-hint">
First step: add your first account below. Export and Delete unlock after you select an account.
</p>
</div>
<div class="action-row">
<button class="button" id="settings-export-btn" type="button" disabled>Export</button>
@@ -44,8 +76,12 @@
<section id="settings-empty" class="panel hidden">
<div class="empty-state">
<div class="eyebrow">Getting started</div>
<h2>Add first account</h2>
<p class="muted">Create an account below or import an exported account JSON.</p>
<p class="muted">Create an account below or import an exported account JSON. No accounts yet.</p>
<div class="empty-actions">
<a class="button primary" href="#settings-add-account-form">Add first account</a>
</div>
</div>
</section>
@@ -76,17 +112,49 @@
<p class="muted">New accounts start without continuous scraping enabled.</p>
</div>
</div>
<div class="panel-body">
<form id="settings-add-account-form" class="stack-form">
<input id="settings-add-account-id" name="account_id" placeholder="Account ID (e.g. work)" required />
<input id="settings-add-account-label" name="label" placeholder="Display name" />
<input id="settings-add-account-api-id" name="api_id" placeholder="API ID" />
<input id="settings-add-account-api-hash" name="api_hash" placeholder="API Hash" />
<button class="button primary" type="submit">Add Account</button>
<label class="field">
<span class="field-label">Account ID</span>
<input
id="settings-add-account-id"
name="account_id"
placeholder="Account ID (e.g. work)"
aria-label="Account ID"
required
/>
<span class="field-hint">Lowercase short id used in file names and URLs.</span>
</label>
<label class="field">
<span class="field-label">Display name</span>
<input
id="settings-add-account-label"
name="label"
placeholder="Display name"
aria-label="Display name"
/>
</label>
<label class="field">
<span class="field-label">API ID</span>
<input id="settings-add-account-api-id" name="api_id" placeholder="API ID" aria-label="API ID" />
<span class="field-hint">From my.telegram.org &gt; API development tools.</span>
</label>
<label class="field">
<span class="field-label">API Hash</span>
<input id="settings-add-account-api-hash" name="api_hash" placeholder="API Hash" aria-label="API Hash" />
<span class="field-hint">Paste the hash exactly as shown on my.telegram.org.</span>
</label>
<button class="button primary button-block" type="submit">Add Account</button>
</form>
<label class="import-account-row">
<span class="muted small">Import account settings JSON</span>
<span class="field-label">Import account settings JSON</span>
<span class="muted small"
>Choose a file previously exported from Settings. It must be account JSON, for example:</span
>
<span class="file-example">{"account_id": "work", "api_id": 12345, ...}</span>
<input id="settings-import-account-file" type="file" accept="application/json,.json" />
</label>
</div>
</div>
<div class="panel">
@@ -96,9 +164,20 @@
<p class="muted" id="settings-auth-label">No account selected.</p>
</div>
</div>
<div class="panel-body">
<p class="muted small" id="settings-credentials-hint">
Select an account first, then save API credentials and complete login below.
</p>
<form id="settings-credentials-form" class="stack-form">
<input id="settings-api-id-input" name="api_id" placeholder="API ID" />
<input id="settings-api-hash-input" name="api_hash" placeholder="API Hash" />
<label class="field">
<span class="field-label">API ID</span>
<input id="settings-api-id-input" name="api_id" placeholder="API ID" aria-label="API ID" />
<span class="field-hint">From my.telegram.org &gt; API development tools.</span>
</label>
<label class="field">
<span class="field-label">API Hash</span>
<input id="settings-api-hash-input" name="api_hash" placeholder="API Hash" aria-label="API Hash" />
</label>
<button class="button" type="submit">Save credentials</button>
</form>
@@ -112,19 +191,35 @@
</div>
<form id="settings-phone-form" class="stack-form">
<input id="settings-phone-input" name="phone" placeholder="+1234567890" />
<label class="field">
<span class="field-label">Phone number</span>
<input id="settings-phone-input" name="phone" placeholder="+1234567890" aria-label="Phone number" />
</label>
<button class="button" type="submit">Send code</button>
</form>
<form id="settings-code-form" class="stack-form hidden">
<input id="settings-code-input" name="code" placeholder="Telegram code" />
<label class="field">
<span class="field-label">Telegram code</span>
<input id="settings-code-input" name="code" placeholder="Telegram code" aria-label="Telegram code" />
</label>
<button class="button" type="submit">Confirm code</button>
</form>
<form id="settings-password-form" class="stack-form hidden">
<input id="settings-password-input" name="password" type="password" placeholder="2FA password" />
<label class="field">
<span class="field-label">2FA password</span>
<input
id="settings-password-input"
name="password"
type="password"
placeholder="2FA password"
aria-label="2FA password"
/>
</label>
<button class="button" type="submit">Confirm password</button>
</form>
</div>
</div>
<div class="panel">
@@ -134,6 +229,7 @@
<p class="muted">Account-level parser settings.</p>
</div>
</div>
<div class="panel-body">
<label class="toggle-row">
<span>Download media</span>
<span class="switch">
@@ -146,11 +242,12 @@
<div>Messages: <span id="settings-message-count">0</span></div>
<div>Media: <span id="settings-media-count">0</span></div>
</div>
</div>
</div>
</section>
</main>
</div>
<script src="/static/settings.js?v=1"></script>
<script src="/static/settings.js?v=9"></script>
</body>
</html>
+45 -6
View File
@@ -38,6 +38,42 @@ function showToast(message, type = 'info') {
}, 3600);
}
function confirmAction(message) {
return new Promise((resolve) => {
let settled = false;
const done = (value) => {
if (settled) return;
settled = true;
if (dialog.open) dialog.close();
resolve(value);
};
let dialog = document.getElementById('confirm-dialog');
if (!dialog) {
dialog = document.createElement('dialog');
dialog.id = 'confirm-dialog';
dialog.className = 'confirm-dialog';
dialog.innerHTML =
'<div class="confirm-dialog-body"><div class="eyebrow">Confirm</div>' +
'<p class="confirm-dialog-message muted"></p>' +
'<div class="confirm-dialog-actions">' +
'<button class="button" data-confirm-cancel type="button">Cancel</button>' +
'<button class="button primary" data-confirm-ok type="button">Confirm</button>' +
'</div></div>';
document.body.appendChild(dialog);
}
dialog.querySelector('.confirm-dialog-message').textContent = message;
dialog.querySelector('[data-confirm-ok]').onclick = () => done(true);
dialog.querySelector('[data-confirm-cancel]').onclick = () => done(false);
dialog.onclose = () => done(false);
if (!dialog.open) dialog.showModal();
});
}
function setHealthState(el, ok, warnOnly = false) {
el.classList.remove('is-ok', 'is-warn', 'is-dim');
el.classList.add(ok ? 'is-ok' : warnOnly ? 'is-dim' : 'is-warn');
}
function downloadJson(filename, payload) {
const blob = new Blob([JSON.stringify(payload, null, 2) + '\n'], {
type: 'application/json',
@@ -105,7 +141,7 @@ function renderAccountList() {
root.innerHTML = '';
if (!settingsState.accounts.length) {
root.innerHTML = '<div class="empty-state">No accounts yet.</div>';
root.innerHTML = '<div class="empty-state"><p class="muted">No accounts yet.</p></div>';
return;
}
@@ -168,17 +204,19 @@ function renderAccountData() {
: 'No account selected.';
document.getElementById('health-credentials').textContent = health.api_credentials ? 'Saved' : 'Missing';
document.getElementById('health-credentials').style.color = health.api_credentials ? 'var(--ok)' : 'var(--warn)';
setHealthState(document.getElementById('health-credentials'), Boolean(health.api_credentials));
document.getElementById('health-session').textContent =
health.session_ready || isAccountAuthorized(auth) ? 'Ready' : 'Missing';
document.getElementById('health-session').style.color =
health.session_ready || isAccountAuthorized(auth) ? 'var(--ok)' : 'var(--warn)';
setHealthState(
document.getElementById('health-session'),
Boolean(health.session_ready || isAccountAuthorized(auth)),
);
document.getElementById('health-continuous').textContent = continuousStatus.running
? 'Running'
: continuousConfig.enabled
? 'Enabled'
: 'Stopped';
document.getElementById('health-continuous').style.color = continuousStatus.running ? 'var(--ok)' : 'var(--dim)';
setHealthState(document.getElementById('health-continuous'), Boolean(continuousStatus.running), true);
document.getElementById('health-last-scrape').textContent = displayTime(
health.last_scrape || continuousStatus.last_iteration_at,
);
@@ -260,6 +298,7 @@ async function importAccount(event) {
await loadAccounts();
await loadAccount(accountId);
showToast(`Imported ${accountId}.`, 'success');
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
} catch (err) {
showToast(`Failed to import account: ${err.message}`, 'error');
}
@@ -341,7 +380,7 @@ async function exportAccount() {
async function deleteAccount() {
if (!settingsState.activeAccount) return;
const account = settingsState.accounts.find((item) => item.id === settingsState.activeAccount);
if (!window.confirm(`Remove account "${accountLabel(account)}"? All its account data will be deleted.`)) return;
if (!(await confirmAction(`Remove account "${accountLabel(account)}"? All its account data will be deleted.`))) return;
await api(`/api/accounts/${encodeURIComponent(settingsState.activeAccount)}`, { method: 'DELETE' });
setActiveAccount(null);
await loadAccounts();
+1199 -598
View File
File diff suppressed because it is too large Load Diff
+32 -6
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Telegram Scraper API</title>
<link rel="stylesheet" href="/static/style.css?v=6" />
<link rel="stylesheet" href="/static/style.css?v=9" />
</head>
<body>
<div class="viewer-shell">
@@ -17,18 +17,44 @@
<a class="button button-small" href="/">Dashboard</a>
</div>
<nav class="nav-links">
<a class="nav-link" href="/openapi.json">openapi.json</a>
<a class="nav-link" href="/health">Health</a>
<a class="nav-link" href="/viewer">Message Viewer</a>
<a class="nav-link" href="/"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<rect x="4" y="4" width="6" height="6" />
<rect x="14" y="4" width="6" height="6" />
<rect x="4" y="14" width="6" height="6" />
<rect x="14" y="14" width="6" height="6" /></svg
>Dashboard</a
>
<a class="nav-link" href="/settings"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<circle cx="12" cy="12" r="3" />
<path
d="M12 2v3m0 14v3M2 12h3m14 0h3m-2.9-7.1-2.1 2.1M4.9 19.1 7 17m0-10-2.1-2.1m12.2 14.2-2.1-2.1" /></svg
>Settings</a
>
<a class="nav-link" href="/viewer"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true"><path d="M4 5h16v12H8l-4 3z" /></svg>Message
Viewer</a
>
<a class="nav-link active" href="/swagger"
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
<path d="m8 5-5 7 5 7m8-14 5 7-5 7M14 3l-4 18" /></svg
>API Docs</a
>
</nav>
</aside>
<main class="viewer-main">
<header class="viewer-header">
<div>
<h2>Endpoints</h2>
<div class="eyebrow">OpenAPI</div>
<h2>API Docs</h2>
<p class="muted">Local API surface exposed by the web UI server.</p>
</div>
<div class="viewer-tools">
<div class="runtime-status"><span class="runtime-dot"></span>Service: Running</div>
<a class="button button-small" href="/openapi.json">openapi.json</a>
</div>
</header>
<section id="api-docs" class="api-docs"></section>
@@ -53,6 +79,6 @@
</article>
</template>
<script src="/static/swagger.js"></script>
<script src="/static/swagger.js?v=9"></script>
</body>
</html>
+52 -2
View File
@@ -19,7 +19,30 @@ function renderSpec(spec) {
const methodTemplate = document.getElementById('api-method-template');
root.innerHTML = '';
Object.entries(spec.paths || {}).forEach(([path, methods]) => {
const paths = Object.entries(spec.paths || {});
if (!paths.length) {
const section = document.createElement('section');
section.className = 'panel';
const header = document.createElement('div');
header.className = 'panel-header';
const title = document.createElement('h2');
title.textContent = 'No endpoints';
header.appendChild(title);
const body = document.createElement('div');
body.className = 'panel-body';
const empty = document.createElement('div');
empty.className = 'empty-state';
const text = document.createElement('p');
text.className = 'muted';
text.textContent = 'The OpenAPI spec contains no paths.';
empty.appendChild(text);
body.appendChild(empty);
section.append(header, body);
root.appendChild(section);
return;
}
paths.forEach(([path, methods]) => {
const section = sectionTemplate.content.firstElementChild.cloneNode(true);
section.querySelector('.api-path').textContent = path;
const methodsRoot = section.querySelector('.api-methods');
@@ -49,5 +72,32 @@ loadSpec()
.then(renderSpec)
.catch((error) => {
console.error(error);
document.getElementById('api-docs').innerHTML = `<section class="panel">${error.message}</section>`;
const docsEl = document.getElementById('api-docs');
docsEl.textContent = '';
const section = document.createElement('section');
section.className = 'panel';
const header = document.createElement('div');
header.className = 'panel-header';
const title = document.createElement('h2');
title.textContent = 'Failed to load API spec';
header.appendChild(title);
const body = document.createElement('div');
body.className = 'panel-body';
const empty = document.createElement('div');
empty.className = 'empty-state';
const text = document.createElement('p');
text.className = 'muted';
text.textContent = error.message;
const actions = document.createElement('div');
actions.className = 'empty-actions';
const retry = document.createElement('button');
retry.className = 'button';
retry.type = 'button';
retry.textContent = 'Retry';
retry.addEventListener('click', () => window.location.reload());
actions.appendChild(retry);
empty.append(text, actions);
body.appendChild(empty);
section.append(header, body);
docsEl.appendChild(section);
});
+4 -4
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Telegram Scraper Viewer</title>
<link rel="stylesheet" href="/static/style.css?v=6" />
<link rel="stylesheet" href="/static/style.css?v=9" />
</head>
<body>
<div class="viewer-shell">
@@ -15,7 +15,7 @@
<h1>Messages</h1>
<p class="muted small">
Account:
<select id="viewer-account-select"></select>
<select id="viewer-account-select" aria-label="Viewer account"></select>
</p>
</div>
<button id="sidebar-toggle" class="button button-small sidebar-toggle"></button>
@@ -31,7 +31,7 @@
<p id="viewer-subtitle" class="muted">Reading messages from the local SQLite database.</p>
</div>
<div class="viewer-tools">
<input id="viewer-search" type="search" placeholder="Search messages" />
<input id="viewer-search" type="search" placeholder="Search messages" aria-label="Search messages" />
<label class="viewer-auto-refresh">
<input id="viewer-auto-refresh" type="checkbox" checked />
<span>Auto-refresh</span>
@@ -80,6 +80,6 @@
</div>
</template>
<script src="/static/viewer.js"></script>
<script src="/static/viewer.js?v=9"></script>
</body>
</html>
+7 -2
View File
@@ -395,10 +395,15 @@ function renderMessages(payload, append = false) {
viewerState.oldestMessageId = null;
viewerState.newestMessageId = null;
const empty = document.createElement('div');
empty.className = 'viewer-empty-state';
empty.textContent = viewerState.search
empty.className = 'empty-state viewer-empty-state';
const title = document.createElement('h2');
title.textContent = viewerState.search ? 'No messages found' : 'No saved messages';
const text = document.createElement('p');
text.className = 'muted';
text.textContent = viewerState.search
? `No messages found for "${viewerState.search}".`
: 'No saved messages in this channel yet.';
empty.append(title, text);
root.appendChild(empty);
}
+1016 -278
View File
File diff suppressed because it is too large Load Diff