Compare commits
17 Commits
dev
...
2a75537fb9
| Author | SHA1 | Date | |
|---|---|---|---|
 forust
|
2a75537fb9 | ||
|
forust
|
bc2e93353a | ||
|
forust
|
e93e68db7e | ||
|
forust
|
a2468a2a2c | ||
|
forust
|
59824940c6 | ||
|
forust
|
70ac1ce73e | ||
|
forust
|
81e69865ce | ||
|
forust
|
418c4d6192 | ||
|
forust
|
5956ea36a8 | ||
|
forust
|
93c8d00af6 | ||
|
forust
|
08f8196bf9 | ||
|
forust
|
b963cf07df | ||
|
forust
|
849bf97c97 | ||
|
forust
|
145e7de7f4 | ||
|
forust
|
27296caf69 | ||
|
forust
|
9b72c3128e | ||
|
forust
|
f5ef74bda6 |
@@ -122,9 +122,29 @@ jobs:
|
||||
-summary \
|
||||
"${manifests[@]}"
|
||||
|
||||
lint-audit:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Audit Python dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
docker run --rm \
|
||||
-v "$PWD:/work" \
|
||||
-w /work \
|
||||
python:3.12-slim \
|
||||
sh -lc '
|
||||
pip install --quiet pip-audit pip-tools &&
|
||||
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
|
||||
pip-audit -r /tmp/reqs.txt
|
||||
'
|
||||
|
||||
publish:
|
||||
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
|
||||
if: github.event_name != 'pull_request'
|
||||
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
||||
@@ -122,9 +122,29 @@ jobs:
|
||||
-summary \
|
||||
"${manifests[@]}"
|
||||
|
||||
lint-audit:
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Audit Python dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
docker run --rm \
|
||||
-v "$PWD:/work" \
|
||||
-w /work \
|
||||
python:3.12-slim \
|
||||
sh -lc '
|
||||
pip install --quiet pip-audit pip-tools &&
|
||||
pip-compile --quiet --strip-extras --output-file /tmp/reqs.txt pyproject.toml &&
|
||||
pip-audit -r /tmp/reqs.txt
|
||||
'
|
||||
|
||||
publish:
|
||||
needs: [lint-prettier, lint-ruff, lint-yaml, lint-dockerfiles, validate]
|
||||
if: github.event_name != 'pull_request'
|
||||
if: github.event_name != 'pull_request' && (github.ref_name == 'main' || github.ref_name == 'dev')
|
||||
runs-on: [self-hosted, linux, arch, homelab]
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
# REVIEW.md — telegram-scraper
|
||||
|
||||
Дата: 2026-09-07
|
||||
Скоуп: полный аудит кода — безопасности и корректности (webui_server.py, app_state.py, scraper_jobs.py, telegram_scraper_with_forwarding.py, main.py, health.py, webui/*.js, деплой, тесты).
|
||||
|
||||
---
|
||||
|
||||
## Статус: что уже исправлено
|
||||
|
||||
10 пунктов (критичные/высокие) исправлены 07.09.2026. Подробности — в истории коммита.
|
||||
|
||||
| ID | Проблема | Файл | Статус |
|
||||
|---|---|---|---|
|
||||
| C-2 | SSE-поток не завершался при успешном job (`done` vs `completed`) — утечка потоков до 30 мин | webui_server.py | ✅ fixed |
|
||||
| C-3 | Пути `Path("data")`/`Path("session")` относительно CWD расходились с BASE_DIR webui — тихая рассинхронизация данных | telegram_scraper_with_forwarding.py, main.py, scraper_jobs.py | ✅ fixed |
|
||||
| H-1 | `TelegramAuthManager.lock` объявлен, но не использовался — гонки на `auth_data`/`clients` между event-loop и HTTP-потоками | webui_server.py | ✅ fixed (RLock) |
|
||||
| H-2 | Коллизия `job_id` из миллисекундного timestamp | webui_server.py | ✅ fixed (uuid4) |
|
||||
| H-3 | Экспорт секретов через `include_secrets=1` (api_hash/api_id) | webui_server.py | ✅ fixed (всегда redact) |
|
||||
| H-4 | CSRF: любой Content-Type, нет Origin/Sec-Fetch-Site проверки | webui_server.py | ✅ fixed (415 + same-origin 403) |
|
||||
| H-5 | Нет rate limiting на брутфорс phone-code/2FA | webui_server.py | ✅ fixed (5 попыток → 60s lockout, 429) |
|
||||
| H-6 | `StateStore.load()` возвращал мелкую копию — расшаренная мутация вложенных dict между потоками | app_state.py | ✅ fixed (deepcopy) |
|
||||
| H-7 | Бесконечная рекурсия `forward_message` при FloodWaitError | telegram_scraper_with_forwarding.py | ✅ fixed (cap 3 retry) |
|
||||
| H-8 | Повторная регистрация forward-хендлера → сообщения форвардились N раз | telegram_scraper_with_forwarding.py | ✅ fixed (remove_event_handler) |
|
||||
|
||||
Бонус при фиксах: `migrate_database` больше не глотает исключения (логирует), миграция покрывает все колонки MessageData; фронтенд `app.js` корректно распознаёт `'done'` как терминальный статус.
|
||||
|
||||
### Follow-up (второй проход по итогам ревью фиксов)
|
||||
|
||||
| ID | Проблема | Статус |
|
||||
|---|---|---|
|
||||
| H-6 | deepcopy теперь на **всех** путях `load()` (cache-hit + cache-miss + fallback-ветки) | ✅ fixed |
|
||||
| H-5 | cooldown 30s на успешный запрос кода (анти-SMS-флуд) + `_auth_attempts` ограничен (sweep при >10k записей) | ✅ fixed |
|
||||
| — | Origin-проверка на GET `/api/jobs/{id}/events` (403 до открытия SSE) | ✅ fixed |
|
||||
| — | SSE-потоки на аккаунт ограничены (`MAX_EVENT_STREAMS=10`, revoke самого старого) — закрыт thread-exhaustion | ✅ fixed |
|
||||
| — | `read_json_body`: кап тела 1 MB → 413 (sentinel), malformed Content-Length → не 500, не-UTF-8 → 400 | ✅ fixed (M-2 закрыт) |
|
||||
| — | +6 регрессионных тестов (H-4/H-5/H-6/C-2, sweep, oversize body) | ✅ fixed |
|
||||
|
||||
Тесты: `29 passed` (все зелёные после фиксов).
|
||||
|
||||
### Follow-up (третий проход)
|
||||
|
||||
| ID | Что исправлено | Статус |
|
||||
|---|---|---|
|
||||
| M-17 | `clean_continuous_channels()` — валидация каналов на приёме в обоих POST-эндпоинтах (per-account + legacy); невалидные отбрасываются и возвращаются как `dropped_invalid` (drop, не 400 — фронтенд api() бросает на non-OK) | ✅ fixed |
|
||||
| M-5 | `PerAccountContinuousScrapeManager.join(timeout=20)` + `remove_account` делает stop()+join перед rmtree; `stop()` идемпотентен; loop проверяет stop event на границах итераций | ✅ fixed |
|
||||
| M-6 | `self.config` присваивается под локом; `refresh_config` больше не врёт о `running` (только stop_event); `running=False` ставится в `finally` потока при реальном выходе | ✅ fixed |
|
||||
| — | +4 теста (33 passed) | ✅ fixed |
|
||||
|
||||
### Follow-up (четвёртый проход)
|
||||
|
||||
| ID | Что исправлено | Статус |
|
||||
|---|---|---|
|
||||
| M-1 | `/media/` lockdown: запрещены `state.json`/`*.db`/`*.session` + allowlist расширений (архивы, документы) | ✅ fixed |
|
||||
| M-3 | Общий `parse_bool()` — строка `"false"`/`"0"` больше не даёт `True` | ✅ fixed |
|
||||
| M-4 | HEAD `/api/jobs/{id}/events` для несуществующего job → 404 | ✅ fixed |
|
||||
| M-7 | `JobRunner.shutdown` — дренаж очереди, оставшиеся jobs → `failed`/`cancelled` | ✅ fixed |
|
||||
| M-8 | `_parse_range`: single-range edge cases (пустой файл, мульти-диапазоны) | ✅ fixed |
|
||||
| M-19 | `int(query...)` → try/except → 400 JSON; пути из ответов ред.актированы | ✅ fixed |
|
||||
| L-1 | Access log — только path, без query-параметров | ✅ fixed |
|
||||
| L-4 | Security-заголовки: CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy | ✅ fixed |
|
||||
| L-5 | QR-токен: one-time + TTL 60s | ✅ fixed |
|
||||
| L-6 | Legacy channels add/remove переведены на `StateStore.update()` — lost-update закрыт | ✅ fixed |
|
||||
| L-8 | Trusted-host allowlist для same-origin проверки — DNS-rebinding закрыт | ✅ fixed |
|
||||
| F-1 | Импорт и legacy-миграция прогоняют каналы через `clean_continuous_channels` | ✅ fixed |
|
||||
| F-3 | `start()` при drain — проверка `thread.is_alive()`/join перед стартом | ✅ fixed |
|
||||
| F-4 | `remove_account` — tombstone при таймауте join (дубли воркеров исключены) | ✅ fixed |
|
||||
| M-10 | `save()`: `flush()+fsync`, уникальные tmp (mkstemp), sweep старых tmp | ✅ fixed |
|
||||
| M-11 | Выделенный поток с `new_event_loop()`; `set_scrape_media` пишет per-account | ✅ fixed |
|
||||
| M-12 | Media пакетами с ограничением размера чанка | ✅ fixed |
|
||||
| M-13 | `save_state` — throttle (не перезапись каждые 50 сообщений) | ✅ fixed |
|
||||
| M-14 | Точное переиспользование файлов (без произвольного `{id}-*` совпадения) | ✅ fixed |
|
||||
| M-15 | `/health` агрегирует per-account проверки | ✅ fixed |
|
||||
| M-16 | k8s: `securityContext` (`runAsNonRoot`, readOnlyRootFilesystem) + `resources.limits` | ✅ fixed |
|
||||
| M-18 | chmod 700 на data/session, StateStore пишет 0600 | ✅ fixed |
|
||||
| M-9 | Legacy GET `/api/channels`/`/api/dashboard` делегируют в `legacy_account_id` | ✅ fixed |
|
||||
| — | Фронтенд: F-2 (toast `dropped_invalid`), L-9 (предупреждение о redacted кредах при импорте), L-2 (swagger.js через `textContent`) | ✅ fixed |
|
||||
| — | CI: job `pip-audit` (non-blocking) | ✅ fixed |
|
||||
| — | +17 тестов → **50 passed** | ✅ fixed |
|
||||
|
||||
Тесты: `50 passed` (все зелёные после четвёртого прохода).
|
||||
|
||||
---
|
||||
|
||||
## ОСТАВШИЕСЯ НАХОДКИ
|
||||
|
||||
### 🔴 C-1. Нет аутентификации на веб-панели, bind 0.0.0.0 + публичный ingress — **won't fix (by design)**
|
||||
|
||||
Файл/строки: `webui_server.py:47-48` (DEFAULT_HOST=0.0.0.0), весь роутинг без auth-check, `compose.yaml:11-12` (порт 7887 на всех интерфейсах), `k8s/telegram-scraper.yaml:76-89` (IngressRoute `tg.workstation.internal` без middleware/basicAuth).
|
||||
|
||||
Любой, кто достаёт порт/домен, может:
|
||||
- прочитать `api_hash`/`api_id` (через export — теперь redact, но есть и другие пути, см. M-1: `/media/accounts/<id>/state.json`),
|
||||
- прочитать QR-токен авторизации и **угнать Telegram-сессию** владельца,
|
||||
- подменить креды, удалить аккаунт (`DELETE /api/accounts/{id}` → `shutil.rmtree`),
|
||||
- читать все чаты, медиа, логи, continuous-scrape состояние.
|
||||
|
||||
**Won't fix — by design** (решение пользователя: «аутх не надо, он онли локал» — только локальный деплой; риск сознательно принят). Блок рекомендаций ниже остаётся как справочник на случай, если панель когда-нибудь станет публичной. Рекомендуемый порядок:
|
||||
1. `BasicAuth`/`ForwardAuth`/OIDC на Traefik IngressRoute (быстро, закрывает сетевой доступ).
|
||||
2. App-level сессионная авторизация (cookie + random token), проверка в `do_GET`/`do_POST`/`do_DELETE` до диспатча.
|
||||
3. Дефолт bind `127.0.0.1` + не публиковать 7887 на всех интерфейсах.
|
||||
4. После ввода auth — пересмотреть M-1 (см. ниже), который сейчас маскируется отсутствием auth.
|
||||
|
||||
---
|
||||
|
||||
### 🟠 СРЕДНИЕ
|
||||
|
||||
| # | Файл:строка (актуально) | Проблема | Предложение |
|
||||
|---|---|---|---|
|
||||
| ~~M-1~~ | ~~webui_server.py:2829+ (`serve_media`)~~ | ~~`/media/` рутится в `DATA_DIR` целиком: `GET /media/accounts/<id>/state.json` отдаёт api_hash (plaintext), `/media/accounts/<id>/<ch>/*.db` — базы. Conтент-проверки нет, только containment~~ | ✅ fixed |
|
||||
| ~~M-2~~ | ~~`read_json_body`~~ | ~~Нет капа тела, malformed Content-Length → 500~~ | ✅ закрыт follow-up: кап 1 MB → 413, try/except, не-UTF-8 → 400 |
|
||||
| ~~M-3~~ | ~~webui_server.py (много: 2168-2176, 2188, 2195, 2484, 2709, 2716)~~ | ~~`bool(body.get("value"/"enabled"/"run_all_tracked"))` — строка `"false"`/`"0"` приходит как `True`. Фиксы H-4 не тронули эти места~~ | ✅ fixed |
|
||||
| ~~M-4~~ | ~~webui_server.py `do_HEAD` (2120) + `stream_job_events` (2064)~~ | ~~HEAD на `/api/jobs/{id}/events` для несуществующего job → 200 вместо 404~~ | ✅ fixed |
|
||||
| ~~M-5~~ | ~~webui_server.py:1102+ (`ContinuousScrapeOrchestrator.remove_account`)~~ | ~~Удаление аккаунта не джойнит поток continuous scrape: `stop()` только ставит event → `shutil.rmtree` может удалить DB/media, которые поток ещё пишет~~ | ✅ fixed |
|
||||
| ~~M-6~~ | ~~webui_server.py:872+ (`refresh_config`)~~ | ~~Ставит `status["running"]=False`, пока поток ещё крутится (status врёт); `update()` присваивает `self.config` вне лока~~ | ✅ fixed |
|
||||
| ~~M-7~~ | ~~webui_server.py:404-426 (`JobRunner.shutdown`)~~ | ~~Очередные jobs остаются `"queued"` навсегда (worker выходит, не дрена́я очередь)~~ | ✅ fixed |
|
||||
| ~~M-8~~ | ~~webui_server.py:2879+ (`_parse_range`)~~ | ~~Мульти-диапазоны `bytes=0-1,5-6` → 416; `bytes=0-0` на пустом файле → 416~~ | ✅ fixed |
|
||||
| ~~M-9~~ | ~~webui_server.py legacy endpoints + `webui_server.py:116-120` (`load_state`/`save_state` через `STATE_STORE`) vs `app_state.py:123-131` (`_GLOBAL_STORE`)~~ | ~~Два независимых StateStore на один файл `data/state.json` — расхождение TTL-кэшей до 1s, конфликтные `.tmp`. Legacy GET `/api/channels`/`/api/dashboard` после миграции читают пустой глобальный state (не делегируют в migrated account)~~ | ✅ fixed |
|
||||
| ~~M-10~~ | ~~app_state.py:72-81 (`save`)~~ | ~~Нет `fsync` перед rename (потеря питания → пустой/битый файл); фиксированное имя `.tmp` (два писателя в файл клообьют друг друга)~~ | ✅ fixed |
|
||||
| ~~M-11~~ | ~~scraper_jobs.py:14-25~~ | ~~`asyncio.run()` на каждый job — `RuntimeError` при вызове из потока с существующим loop (e.g. auth loop thread); `set_scrape_media` пишет в глобальный `STATE_STORE` вместо per-account~~ | ✅ fixed |
|
||||
| ~~M-12~~ | ~~telegram_scraper_with_forwarding.py (scrape_channel)~~ | ~~Держит все media-объекты в памяти за весь проход (100k+ сообщений в большом канале)~~ | ✅ fixed |
|
||||
| ~~M-13~~ | ~~telegram_scraper_with_forwarding.py:127-131 (`save_state`)~~ | ~~Перезапись всего per-account JSON каждые 50 сообщений — сотни сериализаций на длинный канал~~ | ✅ fixed |
|
||||
| ~~M-14~~ | ~~telegram_scraper_with_forwarding.py (existing_files glob)~~ | ~~Первое произвольное совпадение `{id}-*` может быть stale/частичным файлом~~ | ✅ fixed |
|
||||
| ~~M-15~~ | ~~health.py:73-83~~ | ~~`/health` читает глобальный state: в multi-account режиме всегда `has_api_credentials: false, tracked_channels: 0` — вводит в заблуждение~~ | ✅ fixed |
|
||||
| ~~M-16~~ | ~~webui_server.py(s) + k8s~~ | ~~Контейнер в k8s без `securityContext` (root, r/w FS, нет limits); в Dockerfile нет `USER` (compose задаёт 1000:1000, k8s — нет)~~ | ✅ fixed |
|
||||
| ~~M-17~~ | ~~webui_server.py continuous endpoints (обе версии `/api/continuous` и `/api/accounts/{id}/continuous`)~~ | ~~Список каналов сохраняется сырым `str().strip()` без `normalize_channel_id` — безопасно только пока фильтрует `_resolve_channels` по tracked~~ | ✅ fixed |
|
||||
| ~~M-18~~ | ~~data/ и session/ (хост)~~ | ~~`root:root 755`, state.json пишется 644 — session-файлы Telethon (полные auth-ключи) и api_hash читаемы локальными юзерами~~ | ✅ fixed |
|
||||
| ~~M-19~~ | ~~webui_server.py:1860-1862, 2011-2013 и др.~~ | ~~`int(query...)` без try/except → ValueError убивает поток + traceback в stderr; многие хендлеры эхат `str(exc)` (абс-пути в ответах)~~ | ✅ fixed |
|
||||
| M-20 | webui_server.py (все POST) | CSRF-фикс (H-4) закрыл Origin/Content-Type, но CSRF-токенов per-session нет; при вводе реальной auth (C-1) нужны | won't fix (by design: no auth, local-only deployment) |
|
||||
|
||||
---
|
||||
|
||||
### ⚪ НИЗКИЕ
|
||||
|
||||
| # | Файл | Проблема |
|
||||
|---|---|---|
|
||||
| ~~L-1~~ | ~~webui_server.py:2776-2782 (access log)~~ | ~~Логируется весь `self.path` с query-параметрами (поисковые запросы и т.п.)~~ ✅ fixed |
|
||||
| ~~L-2~~ | ~~webui/swagger.js:52~~ | ~~`innerHTML` с ошибкой из /openapi.json (низкий риск — серверный контент)~~ ✅ fixed |
|
||||
| ~~L-3~~ | ~~requirements.txt~~ | ~~Зависимости корректны (aiohttp 3.12.14 — патч CVE-2025-53643), но Telethon 1.40.0 (есть ~1.44.x); добавить `uv audit`/`pip-audit` в CI~~ ✅ fixed |
|
||||
| ~~L-4~~ | ~~webui_server.py send_json/serve_file~~ | ~~Нет security-заголовков: CSP, X-Content-Type-Options, X-Frame-Options/frame-ancestors, Referrer-Policy (clickjacking актуален после ввода auth)~~ ✅ fixed |
|
||||
| ~~L-5~~ | ~~webui_server.py auth snapshots (582-589)~~ | ~~QR-токен и его изображение висят в snapshot до сканирования — one-time + expiry ~60s~~ ✅ fixed |
|
||||
| ~~L-6~~ | ~~webui_server.py:2147-2173 (legacy channels add/remove)~~ | ~~Паттерн load→save вместо `StateStore.update()` — lost-update race между потоками~~ ✅ fixed |
|
||||
| L-7 | telegram_scraper_with_forwarding.py:838-841 | Прогресс-бар врут на инкрементальных прогонах (total vs only-new) — не исправлено (косметика) |
|
||||
| ~~L-8~~ | ~~webui_server.py `_check_same_origin`~~ | ~~DNS-rebinding: `Host == Origin.netloc` проходит, если оба — домен атакующего (при rebinding `Sec-Fetch-Site` = same-origin). Закрыть allowlist'ом (localhost/127.0.0.1) или дефолт-bind 127.0.0.1~~ ✅ fixed |
|
||||
| ~~L-9~~ | ~~webui/app.js:944-962, webui/settings.js:248-266~~ | ~~Import/export round-trip молча теряет `api_id`/`api_hash` (H-3 redact): UI не предупреждает, что креды нужно ввести заново после импорта~~ | ✅ fixed |
|
||||
|
||||
---
|
||||
|
||||
## Follow-up findings (round 3)
|
||||
|
||||
### 🟠 СРЕДНИЕ (from round-3 review)
|
||||
|
||||
| # | Файл:строка | Проблема | Предложение |
|
||||
|---|---|---|---|
|
||||
| ~~F-1~~ | ~~webui_server.py:2592, app_state.py:263~~ | ~~Импорт аккаунта и legacy-миграция пишут `continuous_scraping.channels` как есть, минуя валидацию M-17 (латентно, т.к. `_resolve_channels` фильтрует по normalized tracked)~~ | ✅ fixed |
|
||||
| ~~F-2~~ | ~~webui_server.py:2880 + webui/app.js:207-211~~ | ~~`dropped_invalid` возвращается, но ни один JS его не читает — юзер не видит, что каналы отброшены~~ | ✅ fixed |
|
||||
| ~~F-3~~ | ~~webui_server.py:1048-1058~~ | ~~Enable во время drain: `start()` early-return по `status["running"]`, потом `finally` ставит False — аккаунт enabled=True, но мёртв до ручного переключения~~ | ✅ fixed |
|
||||
| ~~F-4~~ | ~~webui_server.py:1246-1254~~ | ~~`remove_account` удаляет менеджера даже при таймауте join — recreate того же id создаёт второй воркер поверх живого (дубли)~~ | ✅ fixed |
|
||||
|
||||
### ⚪ НИЗКИЕ (from round-3 review)
|
||||
|
||||
| # | Файл | Проблема |
|
||||
|---|---|---|
|
||||
| ~~F-5~~ | ~~webui_server.py:237-238~~ | ~~`channels: null/не-список` молча стирает весь список каналов (`([], [])`) — рассмотреть 400 на malformed payload~~ | ✅ fixed |
|
||||
| ~~F-6~~ | ~~webui_server.py:987-991~~ | ~~`refresh_config`/`_save_config` стрипают, но не нормализуют — `@`-значения с диска (import/migration) никогда не матчатся с normalized tracked, молча не скрейпятся~~ | ✅ fixed |
|
||||
| ~~F-7~~ | ~~webui_server.py:1104-1171~~ | ~~Нет верхнего `except` в `_run_loop`: исключение в refresh/auth-check убивает поток с последним_error нетронутым~~ | ✅ fixed |
|
||||
| ~~F-8~~ | ~~tests/test_integration.py:441-477~~ | ~~Новые тесты не покрывают `join()`→False (таймаут) и start-during-drain; assert `running is True` после stop завязан на GIL-timing~~ | ✅ fixed (join→False + shutdown-guard) |
|
||||
|
||||
---
|
||||
|
||||
### Round-4 residual notes — остаток после четвёртого прохода
|
||||
|
||||
Открыто после round 4:
|
||||
|
||||
- **F-5** (`channels: null` молча стирает список), **F-6** (`@`-значения с диска никогда не нормализуются), **F-7** (нет верхнего `except` в `_run_loop`), **F-8** (join-timeout / start-during-drain не покрыты тестами) — закрыты в round 5 (см. ниже).
|
||||
- **M-20** CSRF-токены — won't fix (auth нет by design).
|
||||
- **L-7** прогресс-бар — открыт (косметика).
|
||||
- Дублированная логика `clean_channel` (webui vs app_state) — документированный риск расхождения (drift).
|
||||
- Экспорт `.json`/`.csv` раздаётся через `/media/` (креды redact — риск низкий).
|
||||
- Тест-гэп: scraper-движок полностью замокан (telethon не в CI) — остаётся самым большим пробелом в тестах.
|
||||
|
||||
### Follow-up (пятый проход — residual round-3 findings)
|
||||
|
||||
| ID | Что исправлено | Статус |
|
||||
|---|---|---|
|
||||
| F-5 | Оба POST continuous-хендлера (`/api/continuous` legacy + `/api/accounts/{id}/continuous`): `channels` присутствует, но не список (строка/число/dict) → 400 без изменения хранимого списка; ключ отсутствует → список читается с диска и сохраняется (не `[]`); `null` → тоже сохраняет существующий список; `[]` остаётся явной очисткой | ✅ fixed |
|
||||
| F-6 | `.lstrip("@")` в `_load_config()`/`_save_config()`/`refresh_config()` менеджера + `app_state.StateStore.save_continuous_config()` — `@`-значения с диска (import/migration/старый код) нормализуются на чтении и записи и матчатся с normalized tracked | ✅ fixed |
|
||||
| F-7 | Верхний `try/except Exception` в `_run_loop` вокруг тела цикла (включая `refresh_config`/auth-check): `logger.exception(...)`, `last_error = "Unexpected loop error (see logs)"`, `last_iteration_at` обновлён, backoff 10s через `stop_event.wait`, продолжение цикла — поток не умирает молча | ✅ fixed |
|
||||
| F-8 | +8 тестов: non-list/`null`/absent/`[]` channels на per-account хендлере, нормализация `@` на load+refresh, `_run_loop` выживает при исключении (last_error + finally), `join()`→False на реальном таймауте и True после завершения, `create_job` в shutdown → RuntimeError | ✅ fixed |
|
||||
|
||||
Тесты: **58 passed** (все зелёные после пятого прохода).
|
||||
|
||||
---
|
||||
|
||||
### 🧪 Пробелы в тестах
|
||||
|
||||
Покрыто новыми тестами (round 2, +6): deepcopy-изоляция `load()` (все пути), Content-Type/oversize в `read_json_body`, same-origin проверка, rate limiter (lockout + cooldown кода), sweep `_auth_attempts`, терминальные статусы SSE.
|
||||
|
||||
Осталось:
|
||||
- tests/test_integration.py:29-38 — **весь** scraper-движок замокан (`sys.modules["telegram_scraper_with_forwarding"] = MagicMock()`): реальный код (media naming, flood, forwarding, DB миграция, session) не покрыт вообще. Рекомендация: ставить telethon в CI и импортировать реальный модуль.
|
||||
- tests/test_integration.py:176-201 — тест удаления аккаунта дублирует логику хендлера инлайн, не вызывает продакшн-путь → регрессии в `_handle_delete_account` не ловятся.
|
||||
- Нет тестов на: `StateStore` TTL/atomic-write/конкурентный `update()`; `_parse_range` / Range-ответы; `normalize_media_url`/`guess_media_kind`; 404 SSE; auth-флоу (QR/phone/2FA state machine); H-1 (lock-дисциплина); H-7/H-8 (scraper engine — упирается в полный мок движка).
|
||||
- tests/test_integration.py:48-50 — мутация `os.environ` на уровне импорта (leak между модулями). Лучше `monkeypatch`.
|
||||
- tests/test_integration.py:337-397 — `START_CONTINUOUS = False` выставляется в setup и не восстанавливается.
|
||||
|
||||
---
|
||||
|
||||
### ✅ Проверено — уязвимостей НЕТ
|
||||
|
||||
- **Path traversal**: `serve_media`/`serve_static` — `resolve()` + `relative_to()` (корректно, включая symlink); `normalize_channel_id` отвергает `/`, `\`, control chars, `.`/`..`.
|
||||
- **SQL injection**: все запросы параметризованы, `search` — через `LIKE ?`.
|
||||
- **XSS**: viewer.js рендерит контент через `textContent`/`createTextNode`; media-URL всегда префиксуется `/media/` (нет `javascript:` схемы).
|
||||
- **SSRF**: юзер-контролируемого фетча URL нет (только MTProto).
|
||||
- **Десериализация**: только JSON, без pickle/yaml.
|
||||
- **Command injection**: нет subprocess/os.system в продакшн-путях.
|
||||
- **Secrets в image**: `.dockerignore` исключает `data/` и `session/`.
|
||||
---
|
||||
|
||||
### Follow-up (шестой проход — k8s-rollout audit живого кластера)
|
||||
|
||||
- **TRUSTED_HOSTS** (`webui_server.py`, L-8): `_is_trusted_host` отвергал ЛЮБОЙ hostname → через Traefik-домен `tg.workstation.internal` (`k8s/telegram-scraper.yaml:97`) браузерные POST/DELETE + SSE `/api/jobs/*/events` возвращали 403. Добавлен `_TRUSTED_HOSTS_ENV` (parse `TRUSTED_HOSTS` на импорте, нормализация `.strip().lower().rstrip(".")`); проверка после localhost-set. Дефолт строгий: без env результаты идентичны прежним для ВСЕХ входов (regression guard); с env настроенный hostname (любой case, опциональный trailing dot) проходит, остальные — нет. IPv6-with-port (`[::1]:8080`) — out of scope, не тронут.
|
||||
- **Media allowlist** (`_MEDIA_FILE_EXTENSIONS`): +10 суффиксов с комментарием `# extended coverage (animated stickers, matroska, legacy containers)` — `.mkv .mk3d .heic .tgs .flv .3gp .ogv .asf .wmv .djvu`. Живой диск: `.tgs` x44 / `.mkv` x2 (возвращали 403). `.exe`/`.ts` оставлены 403. `guess_media_kind` для новых суффиксов возвращает `"file"` (fall-through как у `.zip`) — viewer рендерит "Open file" link. `serve_media` уже lowercases suffix, `.MP4`/`.MOV` ок.
|
||||
- **Cache-buster**: `webui/index.html` `app.js?v=4` → `app.js?v=5` (иначе stale JS после deploy).
|
||||
- **k8s-манифест**: `containers[0].env: TRUSTED_HOSTS="tg.workstation.internal"` (после `tty: true`); image/probes/securityContext/resources/PVCs/IngressRoute не тронуты; YAML проверен `yaml.safe_load`.
|
||||
- **Live-cluster факты**: local-path PVCs, state v2 `accounts=[default, forust]`, смешанное владение 1000/root → обязательный `chown -R 1000:1000 /app/data /app/session` ДО первого старта нового пода.
|
||||
|
||||
Тесты: **58 → 64 passed** (+3 TRUSTED_HOSTS env, +3 extended media).
|
||||
+139
-23
@@ -1,6 +1,8 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from copy import deepcopy
|
||||
@@ -50,12 +52,12 @@ class StateStore:
|
||||
with self.lock:
|
||||
now = time.time()
|
||||
if self._cache is not None and (now - self._cache_time) < self._cache_ttl:
|
||||
return dict(self._cache)
|
||||
return deepcopy(self._cache)
|
||||
if not self.path.exists():
|
||||
result = deepcopy(self.defaults)
|
||||
self._cache = result
|
||||
self._cache_time = now
|
||||
return result
|
||||
return deepcopy(result)
|
||||
try:
|
||||
with self.path.open("r", encoding="utf-8") as handle:
|
||||
state: Dict[str, Any] = json.load(handle)
|
||||
@@ -63,23 +65,82 @@ class StateStore:
|
||||
result = deepcopy(self.defaults)
|
||||
self._cache = result
|
||||
self._cache_time = now
|
||||
return result
|
||||
return deepcopy(result)
|
||||
result = self._merge_defaults(state)
|
||||
self._cache = result
|
||||
self._cache_time = now
|
||||
return result
|
||||
return deepcopy(result)
|
||||
|
||||
def save(self, state: Dict[str, Any]) -> None:
|
||||
with self.lock:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp_path = self.path.with_suffix(self.path.suffix + ".tmp")
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
merged = self._merge_defaults(state)
|
||||
with tmp_path.open("w", encoding="utf-8") as handle:
|
||||
json.dump(merged, handle, ensure_ascii=False, indent=2)
|
||||
handle.write("\n")
|
||||
tmp_path.replace(self.path)
|
||||
|
||||
# Unique temp file so two writers to the same path cannot clobber
|
||||
# each other's in-progress file, plus fsync before atomic rename so
|
||||
# a power loss cannot leave an empty/corrupt state file behind.
|
||||
fd, tmp_name = tempfile.mkstemp(
|
||||
dir=str(self.path.parent),
|
||||
prefix=self.path.name + ".tmp-",
|
||||
suffix=".tmp",
|
||||
)
|
||||
tmp_path = Path(tmp_name)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(merged, handle, ensure_ascii=False, indent=2)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
# Restrict permissions on the state file to the owning user.
|
||||
try:
|
||||
os.chmod(tmp_path, 0o600)
|
||||
except OSError:
|
||||
pass
|
||||
os.replace(tmp_path, self.path)
|
||||
finally:
|
||||
# Ensure no leftover stale temp file if something went wrong.
|
||||
if tmp_path.exists():
|
||||
try:
|
||||
tmp_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# Best-effort directory fsync for full durability (POSIX only).
|
||||
try:
|
||||
dir_fd = os.open(str(self.path.parent), os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(dir_fd)
|
||||
finally:
|
||||
os.close(dir_fd)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# After a successful save, best-effort sweep leftover stale tmp
|
||||
# files (e.g. from a crashed writer) older than an hour. Throttled
|
||||
# to avoid scanning the dir on every save.
|
||||
self._sweep_stale_tmp(now=time.time())
|
||||
|
||||
self._cache = None
|
||||
|
||||
_STALE_TMP_MAX_AGE = 3600.0 # 1 hour
|
||||
_STALE_TMP_SWEEP_INTERVAL = 60.0
|
||||
_stale_sweep_last: float = 0.0
|
||||
|
||||
def _sweep_stale_tmp(self, now: float) -> None:
|
||||
if (now - self._stale_sweep_last) < self._STALE_TMP_SWEEP_INTERVAL:
|
||||
return
|
||||
self._stale_sweep_last = now
|
||||
try:
|
||||
cutoff = now - self._STALE_TMP_MAX_AGE
|
||||
for stale in self.path.parent.glob(self.path.name + ".tmp-*.tmp"):
|
||||
try:
|
||||
if stale.stat().st_mtime < cutoff:
|
||||
stale.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def update(self, mutator: Callable[[Dict[str, Any]], None]) -> Dict[str, Any]:
|
||||
with self.lock:
|
||||
state = self.load()
|
||||
@@ -97,7 +158,7 @@ class StateStore:
|
||||
"enabled": bool(config.get("enabled", True)),
|
||||
"interval_minutes": max(1, int(config.get("interval_minutes", 1) or 1)),
|
||||
"channels": [
|
||||
str(item).strip()
|
||||
str(item).strip().lstrip("@")
|
||||
for item in config.get("channels", [])
|
||||
if str(item).strip()
|
||||
],
|
||||
@@ -124,6 +185,18 @@ _GLOBAL_STORE: Optional[StateStore] = None
|
||||
|
||||
|
||||
def get_global_store(data_dir: Path) -> StateStore:
|
||||
"""Return the process-wide global state store singleton.
|
||||
|
||||
The store is cached in a module-global and the SAME instance is returned
|
||||
for the same process, regardless of how many times this is called with the
|
||||
same (or any) data_dir. webui_server's ``STATE_STORE`` should delegate to
|
||||
this function so there is exactly one authoritative global store per
|
||||
process rather than a second, potentially divergent instance.
|
||||
|
||||
Backward-compat note: callers that cache their own ``_GLOBAL_STORE=None``
|
||||
sentinel (e.g. tests resetting state between cases) still work because we
|
||||
re-create the singleton lazily on first call.
|
||||
"""
|
||||
global _GLOBAL_STORE
|
||||
if _GLOBAL_STORE is None:
|
||||
_GLOBAL_STORE = StateStore(data_dir / "state.json", defaults=GLOBAL_DEFAULTS)
|
||||
@@ -186,13 +259,47 @@ def account_session_path(session_dir: Path, account_id: str) -> str:
|
||||
# ── MIGRATION (with data copy) ─────────────────────────────────────────
|
||||
|
||||
|
||||
def _is_valid_channel_id(channel_id: str) -> bool:
|
||||
"""Path-safe channel id check.
|
||||
|
||||
Mirrors webui_server.normalize_channel_id's validation: reject entries
|
||||
containing ``/`` or ``\\``, control chars, ``.``/``..``, empty; keep
|
||||
numbers and plain names. Defined locally (not imported from webui_server,
|
||||
which would be circular) so it can be shared by migration.
|
||||
"""
|
||||
channel_id = str(channel_id or "").strip()
|
||||
if (
|
||||
not channel_id
|
||||
or "/" in channel_id
|
||||
or "\\" in channel_id
|
||||
or channel_id in {".", ".."}
|
||||
or any(ord(ch) < 32 for ch in channel_id)
|
||||
):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _clean_continuous_channels(channels: Any) -> List[str]:
|
||||
"""Normalize/drop invalid continuous-scraping channel entries during
|
||||
migration. Mirrors webui_server.clean_continuous_channels: strips a
|
||||
leading ``@``, keeps numbers/names, and drops unsafe entries so they can
|
||||
never become a path-traversal vector.
|
||||
"""
|
||||
cleaned: List[str] = []
|
||||
if not isinstance(channels, list):
|
||||
return cleaned
|
||||
for item in channels:
|
||||
cleaned.append(str(item).strip().lstrip("@"))
|
||||
return [c for c in cleaned if _is_valid_channel_id(c)]
|
||||
|
||||
|
||||
def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
|
||||
"""Copy a single channel's DB + media from src_root to dst_root."""
|
||||
src_ch = src_root / channel_id
|
||||
dst_ch = dst_root / channel_id
|
||||
if not src_ch.exists():
|
||||
return
|
||||
dst_ch.mkdir(parents=True, exist_ok=True)
|
||||
dst_ch.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
# SQLite DB
|
||||
db_name = f"{channel_id}.db"
|
||||
@@ -206,7 +313,7 @@ def _copy_channel_data(src_root: Path, dst_root: Path, channel_id: str) -> None:
|
||||
src_media = src_ch / "media"
|
||||
dst_media = dst_ch / "media"
|
||||
if src_media.exists() and src_media.is_dir():
|
||||
dst_media.mkdir(parents=True, exist_ok=True)
|
||||
dst_media.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for item in src_media.iterdir():
|
||||
if item.is_file():
|
||||
dst_file = dst_media / item.name
|
||||
@@ -250,7 +357,24 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
|
||||
|
||||
# ── 1. Create per-account state for "default" ──────────────────────
|
||||
acc_dir = data_dir / "accounts" / "default"
|
||||
acc_dir.mkdir(parents=True, exist_ok=True)
|
||||
acc_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
# Normalize/drop invalid continuous-scraping channel entries during
|
||||
# migration so unsafe values can never become a path-traversal vector.
|
||||
continuous_cfg = raw.get(
|
||||
"continuous_scraping",
|
||||
{
|
||||
"enabled": True,
|
||||
"interval_minutes": 1,
|
||||
"channels": [],
|
||||
"run_all_tracked": True,
|
||||
},
|
||||
)
|
||||
if not isinstance(continuous_cfg, dict):
|
||||
continuous_cfg = {}
|
||||
continuous_cfg["channels"] = _clean_continuous_channels(
|
||||
continuous_cfg.get("channels")
|
||||
)
|
||||
|
||||
acc_state = {
|
||||
"label": "Default",
|
||||
@@ -260,15 +384,7 @@ def migrate_legacy_state(data_dir: Path, session_dir: Path) -> bool:
|
||||
"channel_names": raw.get("channel_names", {}),
|
||||
"scrape_media": raw.get("scrape_media", True),
|
||||
"forwarding_rules": raw.get("forwarding_rules", []),
|
||||
"continuous_scraping": raw.get(
|
||||
"continuous_scraping",
|
||||
{
|
||||
"enabled": True,
|
||||
"interval_minutes": 1,
|
||||
"channels": [],
|
||||
"run_all_tracked": True,
|
||||
},
|
||||
),
|
||||
"continuous_scraping": continuous_cfg,
|
||||
}
|
||||
|
||||
acc_state_path = acc_dir / "state.json"
|
||||
|
||||
@@ -2,7 +2,7 @@ import sqlite3
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app_state import StateStore
|
||||
from app_state import StateStore, load_account
|
||||
|
||||
|
||||
def health_payload(
|
||||
@@ -73,6 +73,28 @@ def _dir_check(path: Path, writable: bool = False) -> Dict[str, Any]:
|
||||
def _state_check(state_store: StateStore) -> Dict[str, Any]:
|
||||
try:
|
||||
state = state_store.load()
|
||||
data_dir = state_store.path.parent
|
||||
accounts = state.get("accounts") or []
|
||||
|
||||
if accounts:
|
||||
# Multi-account mode: the global store no longer holds api
|
||||
# credentials / channels. Aggregate those from each account's own
|
||||
# state file so the reported values are meaningful.
|
||||
has_api_credentials = False
|
||||
tracked_channels = 0
|
||||
for acc_id in accounts:
|
||||
acc = load_account(data_dir, acc_id)
|
||||
if acc.get("api_id") and acc.get("api_hash"):
|
||||
has_api_credentials = True
|
||||
tracked_channels += len(acc.get("channels", {}) or {})
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(state_store.path),
|
||||
"has_api_credentials": has_api_credentials,
|
||||
"tracked_channels": tracked_channels,
|
||||
}
|
||||
|
||||
# Legacy single-account semantics (no accounts list).
|
||||
return {
|
||||
"ok": True,
|
||||
"path": str(state_store.path),
|
||||
|
||||
@@ -31,13 +31,28 @@ spec:
|
||||
labels:
|
||||
app: telegram-scraper
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
containers:
|
||||
- name: telegram-scraper
|
||||
image: gcr.forust.xyz/forust/telegram-scraper:latest
|
||||
stdin: true
|
||||
tty: true
|
||||
env:
|
||||
- name: TRUSTED_HOSTS
|
||||
value: "tg.workstation.internal"
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
resources:
|
||||
requests:
|
||||
memory: 128Mi
|
||||
cpu: 100m
|
||||
limits:
|
||||
memory: 512Mi
|
||||
cpu: 1
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
|
||||
@@ -18,8 +18,9 @@ def main():
|
||||
# Run legacy migration before starting the server
|
||||
from app_state import migrate_legacy_state
|
||||
|
||||
data_dir = Path("data")
|
||||
session_dir = Path("session")
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
data_dir = BASE_DIR / "data"
|
||||
session_dir = BASE_DIR / "session"
|
||||
try:
|
||||
if migrate_legacy_state(data_dir, session_dir):
|
||||
logger.info("Legacy migration completed successfully.")
|
||||
|
||||
@@ -20,3 +20,6 @@ dependencies = [
|
||||
"Telethon==1.40.0",
|
||||
"yarl==1.20.1",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
pythonpath = ["."]
|
||||
|
||||
+71
-9
@@ -1,11 +1,46 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app_state import StateStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def _run_in_new_loop(coro_factory):
|
||||
"""Run an awaitable on a dedicated thread with its own event loop.
|
||||
|
||||
Returns the coroutine's result. This avoids ``asyncio.run()`` raising
|
||||
``RuntimeError`` when the caller runs on a thread that already has a
|
||||
running event loop (e.g. job threads, auth-loop threads).
|
||||
"""
|
||||
result = {}
|
||||
error = {}
|
||||
|
||||
def runner():
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
asyncio.set_event_loop(loop)
|
||||
result["value"] = loop.run_until_complete(coro_factory())
|
||||
except BaseException as exc: # noqa: BLE001 - relay any failure
|
||||
error["value"] = exc
|
||||
finally:
|
||||
try:
|
||||
loop.close()
|
||||
finally:
|
||||
asyncio.set_event_loop(None)
|
||||
|
||||
thread = threading.Thread(target=runner, daemon=True)
|
||||
thread.start()
|
||||
thread.join()
|
||||
if "value" in error:
|
||||
raise error["value"]
|
||||
return result.get("value")
|
||||
|
||||
|
||||
class ScraperJobService:
|
||||
def __init__(self, state_store: StateStore):
|
||||
@@ -13,22 +48,24 @@ class ScraperJobService:
|
||||
|
||||
def run(self, job_type: str, payload: Dict[str, Any]) -> None:
|
||||
if job_type == "set_scrape_media":
|
||||
# The webui handler already persists the scrape_media setting to
|
||||
# the per-account (or legacy) store before enqueueing this job.
|
||||
# Multi-account mode has no single global change to make, so this
|
||||
# is a passthrough that just records success to keep the
|
||||
# job-status / SSE flow intact.
|
||||
value = bool(payload["value"])
|
||||
|
||||
def mutate(state: Dict[str, Any]) -> None:
|
||||
state["scrape_media"] = value
|
||||
|
||||
self.state_store.update(mutate)
|
||||
logger.info("Media scraping set to %s", value)
|
||||
logger.info("Media scraping set to %s (already persisted by handler)", value)
|
||||
return
|
||||
|
||||
asyncio.run(self._run_async(job_type, payload))
|
||||
# Run the async job on a fresh thread/event loop so we never hit
|
||||
# "asyncio.run() cannot be called from a running event loop".
|
||||
_run_in_new_loop(lambda: self._run_async(job_type, payload))
|
||||
|
||||
async def _run_async(self, job_type: str, payload: Dict[str, Any]) -> None:
|
||||
# Extract account_id from payload, default to None (legacy)
|
||||
account_id: Optional[str] = payload.get("account_id")
|
||||
ScraperClass = self._import_scraper_class()
|
||||
scraper = ScraperClass(account_id=account_id)
|
||||
scraper = ScraperClass(account_id=account_id, base_dir=BASE_DIR)
|
||||
|
||||
if account_id:
|
||||
from app_state import load_account
|
||||
@@ -73,9 +110,34 @@ class ScraperJobService:
|
||||
await scraper.client.disconnect()
|
||||
|
||||
async def _scrape_channels(self, scraper, channels: List[str]) -> None:
|
||||
"""Scrape all channels resiliently: a single channel failure does not
|
||||
abort the rest. Each channel's offset is persisted even on failure
|
||||
(see scrape_channel's finally block), so partial progress is retained.
|
||||
If *every* channel fails, raise so the job is marked failed.
|
||||
"""
|
||||
failed: List[str] = []
|
||||
for channel_id in channels:
|
||||
offset = int(scraper.state.get("channels", {}).get(channel_id, 0) or 0)
|
||||
await scraper.scrape_channel(channel_id, offset)
|
||||
try:
|
||||
ok = await scraper.scrape_channel(channel_id, offset)
|
||||
except Exception: # noqa: BLE001 - scrape_channel re-raises some errors
|
||||
logger.exception("Scrape of channel %s raised", channel_id)
|
||||
failed.append(channel_id)
|
||||
continue
|
||||
if not ok:
|
||||
logger.warning("Scrape of channel %s reported failure", channel_id)
|
||||
failed.append(channel_id)
|
||||
if failed and len(failed) == len(channels):
|
||||
raise RuntimeError(
|
||||
"All scrape target(s) failed: " + ", ".join(failed)
|
||||
)
|
||||
if failed:
|
||||
logger.warning(
|
||||
"Partial scrape failure — %d/%d channel(s) failed: %s",
|
||||
len(failed),
|
||||
len(channels),
|
||||
", ".join(failed),
|
||||
)
|
||||
|
||||
def _import_scraper_class(self):
|
||||
from telegram_scraper_with_forwarding import OptimizedTelegramScraper
|
||||
|
||||
@@ -2,6 +2,7 @@ import sqlite3
|
||||
import json
|
||||
import csv
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
import sys
|
||||
import warnings
|
||||
@@ -27,6 +28,9 @@ from app_state import (
|
||||
get_account_store,
|
||||
)
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
warnings.filterwarnings(
|
||||
"ignore", message="Using async sessions support is an experimental feature"
|
||||
)
|
||||
@@ -90,19 +94,21 @@ def _ensure_session_wal(session_path: str) -> None:
|
||||
|
||||
|
||||
class OptimizedTelegramScraper:
|
||||
def __init__(self, account_id: Optional[str] = None):
|
||||
def __init__(self, account_id: Optional[str] = None, base_dir: Optional[Path] = None):
|
||||
self.account_id = account_id
|
||||
self.SESSION_DIR = Path("session")
|
||||
self.SESSION_DIR.mkdir(exist_ok=True)
|
||||
base_dir = base_dir or BASE_DIR
|
||||
self.BASE_DIR = base_dir
|
||||
self.SESSION_DIR = base_dir / "session"
|
||||
self.SESSION_DIR.mkdir(exist_ok=True, mode=0o700)
|
||||
|
||||
if account_id:
|
||||
self.DATA_DIR = Path("data") / "accounts" / account_id
|
||||
self.state_store = get_account_store(Path("data"), account_id)
|
||||
self.DATA_DIR = base_dir / "data" / "accounts" / account_id
|
||||
self.state_store = get_account_store(base_dir / "data", account_id)
|
||||
else:
|
||||
self.DATA_DIR = Path("data")
|
||||
self.DATA_DIR = base_dir / "data"
|
||||
self.state_store = StateStore(self.DATA_DIR / "state.json")
|
||||
|
||||
self.DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
self.DATA_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
self.STATE_FILE = str(self.DATA_DIR / "state.json")
|
||||
|
||||
self.state = self.load_state()
|
||||
@@ -112,6 +118,8 @@ class OptimizedTelegramScraper:
|
||||
self.max_concurrent_downloads = 5
|
||||
self.batch_size = 100
|
||||
self.state_save_interval = 50
|
||||
self.state_save_throttle_seconds = 5.0
|
||||
self.last_state_save = None
|
||||
self.db_connections = {}
|
||||
self.forwarding_handler = None
|
||||
|
||||
@@ -121,9 +129,25 @@ class OptimizedTelegramScraper:
|
||||
def save_state(self):
|
||||
try:
|
||||
self.state_store.save(self.state)
|
||||
self.last_state_save = time.time()
|
||||
except Exception as e:
|
||||
print(f"Failed to save state: {e}")
|
||||
|
||||
def _save_state_throttled(self):
|
||||
"""Throttled intermediate state persistence.
|
||||
|
||||
Avoids rewriting the whole per-account JSON on every
|
||||
``state_save_interval`` messages (which is far too chatty for long
|
||||
channels). Skips saves that fall within the throttle window; the
|
||||
final save at end-of-scrape always runs regardless.
|
||||
"""
|
||||
now = time.time()
|
||||
if (
|
||||
self.last_state_save is None
|
||||
or (now - self.last_state_save) >= self.state_save_throttle_seconds
|
||||
):
|
||||
self.save_state()
|
||||
|
||||
def get_forwarding_rules(self) -> List[ForwardingRule]:
|
||||
rules = []
|
||||
for rule_dict in self.state.get("forwarding_rules", []):
|
||||
@@ -181,7 +205,7 @@ class OptimizedTelegramScraper:
|
||||
def get_db_connection(self, channel: str) -> sqlite3.Connection:
|
||||
if channel not in self.db_connections:
|
||||
channel_dir = self.DATA_DIR / channel
|
||||
channel_dir.mkdir(parents=True, exist_ok=True)
|
||||
channel_dir.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
|
||||
db_file = channel_dir / f"{channel}.db"
|
||||
conn = sqlite3.connect(str(db_file), check_same_thread=False, timeout=30)
|
||||
@@ -210,6 +234,22 @@ class OptimizedTelegramScraper:
|
||||
columns = {row[1] for row in cursor.fetchall()}
|
||||
|
||||
migrations = []
|
||||
if "sender_id" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN sender_id INTEGER")
|
||||
if "first_name" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN first_name TEXT")
|
||||
if "last_name" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN last_name TEXT")
|
||||
if "username" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN username TEXT")
|
||||
if "message" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN message TEXT")
|
||||
if "media_type" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN media_type TEXT")
|
||||
if "media_path" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN media_path TEXT")
|
||||
if "reply_to" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN reply_to INTEGER")
|
||||
if "post_author" not in columns:
|
||||
migrations.append("ALTER TABLE messages ADD COLUMN post_author TEXT")
|
||||
if "views" not in columns:
|
||||
@@ -222,8 +262,8 @@ class OptimizedTelegramScraper:
|
||||
for migration in migrations:
|
||||
try:
|
||||
conn.execute(migration)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Migration failed for %s: %s", migration, e)
|
||||
|
||||
if migrations:
|
||||
conn.commit()
|
||||
@@ -278,7 +318,7 @@ class OptimizedTelegramScraper:
|
||||
try:
|
||||
channel_dir = self.DATA_DIR / channel
|
||||
media_folder = channel_dir / "media"
|
||||
media_folder.mkdir(exist_ok=True)
|
||||
media_folder.mkdir(exist_ok=True, mode=0o700)
|
||||
|
||||
if isinstance(message.media, MessageMediaPhoto):
|
||||
original_name = getattr(message.file, "name", None) or "photo.jpg"
|
||||
@@ -294,9 +334,18 @@ class OptimizedTelegramScraper:
|
||||
unique_filename = f"{message.id}-{base_name}{extension}"
|
||||
media_path = media_folder / unique_filename
|
||||
|
||||
existing_files = list(media_folder.glob(f"{message.id}-*"))
|
||||
if existing_files:
|
||||
return str(existing_files[0])
|
||||
# Prefer the exact expected filename. Fall back to a matching
|
||||
# "{id}-*" file only if it is non-empty (and pick the newest one,
|
||||
# in case a stale/partial/other-extension file is present).
|
||||
if media_path.exists() and media_path.stat().st_size > 0:
|
||||
return str(media_path)
|
||||
|
||||
candidates = [
|
||||
p for p in media_folder.glob(f"{message.id}-*")
|
||||
if p != media_path and p.is_file() and p.stat().st_size > 0
|
||||
]
|
||||
if candidates:
|
||||
return str(max(candidates, key=lambda p: p.stat().st_mtime))
|
||||
|
||||
for attempt in range(3):
|
||||
try:
|
||||
@@ -353,7 +402,7 @@ class OptimizedTelegramScraper:
|
||||
return False
|
||||
|
||||
async def forward_message(
|
||||
self, message, rule: ForwardingRule, source_channel_id: int = None
|
||||
self, message, rule: ForwardingRule, source_channel_id: int = None, _retry: int = 0
|
||||
):
|
||||
try:
|
||||
dest_entity = await self._resolve_entity(rule.destination_channel)
|
||||
@@ -393,9 +442,12 @@ class OptimizedTelegramScraper:
|
||||
|
||||
return True
|
||||
except FloodWaitError as e:
|
||||
if _retry >= 3:
|
||||
print(f" Failed to forward message {message.id}: FloodWait retry limit exceeded")
|
||||
return False
|
||||
print(f" Rate limited, waiting {e.seconds}s...")
|
||||
await asyncio.sleep(e.seconds)
|
||||
return await self.forward_message(message, rule, source_channel_id)
|
||||
return await self.forward_message(message, rule, source_channel_id, _retry=_retry + 1)
|
||||
except Exception as e:
|
||||
print(f" Failed to forward message {message.id}: {e}")
|
||||
return False
|
||||
@@ -429,6 +481,11 @@ class OptimizedTelegramScraper:
|
||||
print("No valid source channels")
|
||||
return False
|
||||
|
||||
# Unregister a previously installed handler so it is never registered twice.
|
||||
if self.forwarding_handler is not None:
|
||||
self.client.remove_event_handler(self.forwarding_handler)
|
||||
self.forwarding_handler = None
|
||||
|
||||
@self.client.on(
|
||||
events.NewMessage(chats=source_channels, incoming=True, outgoing=True)
|
||||
)
|
||||
@@ -829,7 +886,14 @@ class OptimizedTelegramScraper:
|
||||
else:
|
||||
return await self.client.get_entity(channel)
|
||||
|
||||
async def scrape_channel(self, channel: str, offset_id: int):
|
||||
async def scrape_channel(self, channel: str, offset_id: int) -> bool:
|
||||
"""Scrape a single channel. Returns True on success, False on failure.
|
||||
|
||||
Offset progress is persisted in a ``finally`` block so partial
|
||||
progress is never lost even when an error occurs mid-scrape.
|
||||
"""
|
||||
last_message_id = offset_id
|
||||
success = False
|
||||
try:
|
||||
if not self.client.is_connected():
|
||||
await self.client.connect()
|
||||
@@ -842,15 +906,70 @@ class OptimizedTelegramScraper:
|
||||
|
||||
if total_messages == 0:
|
||||
print(f"No messages found in channel {channel}")
|
||||
return
|
||||
return True
|
||||
|
||||
print(f"Found {total_messages} messages in channel {channel}")
|
||||
|
||||
message_batch = []
|
||||
media_tasks = []
|
||||
processed_messages = 0
|
||||
last_message_id = offset_id
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
|
||||
media_flush_chunk = 50
|
||||
|
||||
# Media progress counters tracked across chunked flushes so the
|
||||
# progress bar stays coherent even though downloads happen in
|
||||
# bounded chunks during the pass instead of all at the end.
|
||||
total_media = 0
|
||||
completed_media = 0
|
||||
successful_downloads = 0
|
||||
|
||||
async def flush_media_batch():
|
||||
"""Download the accumulated media messages in small batches.
|
||||
|
||||
Keeps memory bounded (we never hold references to every
|
||||
media-capable message for the whole channel), and updates the
|
||||
shared media progress counters via ``nonlocal``.
|
||||
"""
|
||||
nonlocal total_media, completed_media, successful_downloads
|
||||
if not media_tasks:
|
||||
return
|
||||
batch = list(media_tasks)
|
||||
media_tasks.clear()
|
||||
total_media += len(batch)
|
||||
|
||||
async def download_single_media(message):
|
||||
async with semaphore:
|
||||
return await self.download_media(channel, message)
|
||||
|
||||
sub_batch = 10
|
||||
for i in range(0, len(batch), sub_batch):
|
||||
sub = batch[i : i + sub_batch]
|
||||
tasks = [
|
||||
asyncio.create_task(download_single_media(msg)) for msg in sub
|
||||
]
|
||||
for j, task in enumerate(tasks):
|
||||
try:
|
||||
media_path = await task
|
||||
if media_path:
|
||||
await self.update_media_path(
|
||||
channel, sub[j].id, media_path
|
||||
)
|
||||
successful_downloads += 1
|
||||
except Exception:
|
||||
pass
|
||||
completed_media += 1
|
||||
if total_media:
|
||||
mprogress = (completed_media / total_media) * 100
|
||||
bar_length = 30
|
||||
mfilled = int(
|
||||
bar_length * completed_media // total_media
|
||||
)
|
||||
mbar = "█" * mfilled + "░" * (bar_length - mfilled)
|
||||
sys.stdout.write(
|
||||
f"\r📥 Media: [{mbar}] {mprogress:.1f}% "
|
||||
f"({completed_media}/{total_media})"
|
||||
)
|
||||
sys.stdout.flush()
|
||||
|
||||
async for message in self.client.iter_messages(
|
||||
entity, offset_id=offset_id, reverse=True
|
||||
@@ -902,6 +1021,11 @@ class OptimizedTelegramScraper:
|
||||
and not isinstance(message.media, MessageMediaWebPage)
|
||||
):
|
||||
media_tasks.append(message)
|
||||
# Flush the pending media list as soon as it reaches the
|
||||
# bounded chunk so we never hold thousands of message
|
||||
# references in memory for the whole channel.
|
||||
if len(media_tasks) >= media_flush_chunk:
|
||||
await flush_media_batch()
|
||||
|
||||
last_message_id = message.id
|
||||
processed_messages += 1
|
||||
@@ -909,10 +1033,15 @@ class OptimizedTelegramScraper:
|
||||
if len(message_batch) >= self.batch_size:
|
||||
self.batch_insert_messages(channel, message_batch)
|
||||
message_batch.clear()
|
||||
# After each insert batch, also flush any accumulated
|
||||
# media (bounded) rather than deferring everything to
|
||||
# the end of the full pass.
|
||||
if media_tasks:
|
||||
await flush_media_batch()
|
||||
|
||||
if processed_messages % self.state_save_interval == 0:
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
self._save_state_throttled()
|
||||
|
||||
progress = (processed_messages / total_messages) * 100
|
||||
bar_length = 30
|
||||
@@ -933,56 +1062,35 @@ class OptimizedTelegramScraper:
|
||||
self.batch_insert_messages(channel, message_batch)
|
||||
|
||||
if media_tasks:
|
||||
total_media = len(media_tasks)
|
||||
completed_media = 0
|
||||
successful_downloads = 0
|
||||
print(f"\n📥 Downloading {total_media} media files...")
|
||||
|
||||
semaphore = asyncio.Semaphore(self.max_concurrent_downloads)
|
||||
|
||||
async def download_single_media(message):
|
||||
async with semaphore:
|
||||
return await self.download_media(channel, message)
|
||||
|
||||
batch_size = 10
|
||||
for i in range(0, len(media_tasks), batch_size):
|
||||
batch = media_tasks[i : i + batch_size]
|
||||
tasks = [
|
||||
asyncio.create_task(download_single_media(msg)) for msg in batch
|
||||
]
|
||||
|
||||
for j, task in enumerate(tasks):
|
||||
try:
|
||||
media_path = await task
|
||||
if media_path:
|
||||
await self.update_media_path(
|
||||
channel, batch[j].id, media_path
|
||||
)
|
||||
successful_downloads += 1
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
completed_media += 1
|
||||
progress = (completed_media / total_media) * 100
|
||||
bar_length = 30
|
||||
filled_length = int(bar_length * completed_media // total_media)
|
||||
bar = "█" * filled_length + "░" * (bar_length - filled_length)
|
||||
|
||||
sys.stdout.write(
|
||||
f"\r📥 Media: [{bar}] {progress:.1f}% ({completed_media}/{total_media})"
|
||||
)
|
||||
sys.stdout.flush()
|
||||
await flush_media_batch()
|
||||
|
||||
if total_media:
|
||||
print(
|
||||
f"\n✅ Media download complete! ({successful_downloads}/{total_media} successful)"
|
||||
)
|
||||
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
print(f"Completed scraping channel {channel}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error with channel {channel}: {e}")
|
||||
# Final state save moved to ``finally`` below so the offset
|
||||
# persists even when an exception aborts the scrape mid-way.
|
||||
success = True
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error with channel %s", channel)
|
||||
|
||||
finally:
|
||||
# Persist the last-known offset even on partial failure so a
|
||||
# re-scrape resumes from the furthest point reached, not from the
|
||||
# start. save_state() is itself best-effort (logs internally),
|
||||
# so a save failure here must not mask the scrape's own result.
|
||||
try:
|
||||
self.state["channels"][channel] = last_message_id
|
||||
self.save_state()
|
||||
except Exception:
|
||||
logger.exception("Failed to save state for channel %s", channel)
|
||||
|
||||
return success
|
||||
|
||||
async def rescrape_media(self, channel: str):
|
||||
conn = self.get_db_connection(channel)
|
||||
|
||||
+967
-3
File diff suppressed because it is too large
Load Diff
+18
-3
@@ -204,10 +204,16 @@ function renderAccountPanel(accountId) {
|
||||
if (!runAllTracked && channels.length === 0 && enabled) {
|
||||
if (!confirmAction('Continuous scraping enabled with no selected channels. Save anyway?')) return;
|
||||
}
|
||||
await api(`/api/accounts/${accountId}/continuous`, {
|
||||
const resp = await api(`/api/accounts/${accountId}/continuous`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ enabled, interval_minutes: intervalMinutes, run_all_tracked: runAllTracked, channels }),
|
||||
});
|
||||
if (resp.dropped_invalid && resp.dropped_invalid.length) {
|
||||
const n = resp.dropped_invalid.length;
|
||||
const shown = resp.dropped_invalid.slice(0, 3).join(', ');
|
||||
const extra = n > 3 ? '…' : '';
|
||||
showToast(`${n} invalid channel(s) skipped: ${shown}${extra}`, 'warn');
|
||||
}
|
||||
await refreshAccount(accountId);
|
||||
});
|
||||
|
||||
@@ -361,6 +367,8 @@ function renderChannels(accountId, channels) {
|
||||
|
||||
// Update channel count stat
|
||||
panel.querySelector('.channel-count').textContent = String(channels.length);
|
||||
panel.querySelector('.channel-active-count').textContent = String(channels.length);
|
||||
panel.querySelector('.channel-inactive-count').textContent = '0';
|
||||
}
|
||||
|
||||
function renderJobs(accountId, jobs) {
|
||||
@@ -409,7 +417,7 @@ function pollJobFallback(accountId, jobId) {
|
||||
try {
|
||||
const job = await api(`/api/jobs/${encodeURIComponent(jobId)}`);
|
||||
updateRenderedJob(accountId, job);
|
||||
if (['completed', 'failed'].includes(job.status)) {
|
||||
if (['done', 'completed', 'failed'].includes(job.status)) {
|
||||
clearInterval(pollTimer);
|
||||
jobStreams.delete(jobId);
|
||||
refreshAccount(accountId);
|
||||
@@ -453,7 +461,7 @@ function subscribeJobStream(accountId, jobId, status) {
|
||||
retryCount = 0; // reset backoff on successful message
|
||||
const job = JSON.parse(event.data);
|
||||
updateRenderedJob(accountId, job);
|
||||
if (['completed', 'failed'].includes(job.status)) {
|
||||
if (['done', 'completed', 'failed'].includes(job.status)) {
|
||||
newStream.close();
|
||||
jobStreams.delete(jobId);
|
||||
refreshAccount(accountId);
|
||||
@@ -616,6 +624,7 @@ function renderSummary(accountId, data) {
|
||||
const d = data.dashboard || data.state || {};
|
||||
const health = data.health || {};
|
||||
panel.querySelector('.forwarding-count').textContent = String((d.forwarding_rules || []).length);
|
||||
panel.querySelector('.forwarding-active-count').textContent = String((d.forwarding_rules || []).length);
|
||||
const toggle = panel.querySelector('.scrape-media-label');
|
||||
toggle.textContent = d.scrape_media ? 'ON' : 'OFF';
|
||||
const healthOk = health.api_credentials && health.session_ready && health.data_dir_exists;
|
||||
@@ -626,6 +635,11 @@ function renderSummary(accountId, data) {
|
||||
`${health.media_count || 0} media`,
|
||||
health.active_job ? `active: ${health.active_job.status}` : 'idle',
|
||||
].join(' | ');
|
||||
const mediaCount = Number(health.media_count || 0);
|
||||
panel.querySelector('.media-session-count').textContent = String(mediaCount);
|
||||
panel.querySelector('.media-total-count').textContent = String(mediaCount);
|
||||
panel.querySelector('.health-session-value').textContent = health.session_ready ? 'Valid' : 'Check';
|
||||
panel.querySelector('.health-rate-value').textContent = health.active_job ? 'Busy' : 'OK';
|
||||
}
|
||||
|
||||
/* ── Account data loading ───────────────────────────── */
|
||||
@@ -948,6 +962,7 @@ async function main() {
|
||||
await loadAccounts();
|
||||
switchAccount(accountId);
|
||||
showToast(`Imported ${accountId}.`, 'success');
|
||||
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
|
||||
} catch (err) {
|
||||
showToast(`Failed to import account: ${err.message}`, 'error');
|
||||
}
|
||||
|
||||
+60
-7
@@ -4,23 +4,49 @@
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Telegram Scraper Control Panel</title>
|
||||
<link rel="stylesheet" href="/static/style.css?v=6" />
|
||||
<link rel="stylesheet" href="/static/style.css?v=8" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="app-shell">
|
||||
<aside class="sidebar">
|
||||
<div class="brand-block">
|
||||
<svg class="brand-icon" viewBox="0 0 24 24" aria-hidden="true">
|
||||
<path d="m21 3-7.6 18-3.8-7.6L2 9.6zM9.6 13.4 14 10" />
|
||||
</svg>
|
||||
<div class="eyebrow">Telegram Scraper</div>
|
||||
<h1>Control</h1>
|
||||
<p class="muted">Local scraper console for channels, jobs, and continuous runs.</p>
|
||||
</div>
|
||||
|
||||
<nav class="nav-links">
|
||||
<a class="nav-link active" href="/">Dashboard</a>
|
||||
<a class="nav-link" href="/settings">Settings</a>
|
||||
<a class="nav-link" href="/viewer">Message Viewer</a>
|
||||
<a class="nav-link" href="/swagger">API Docs</a>
|
||||
<a class="nav-link" href="/health">Health</a>
|
||||
<a class="nav-link active" href="/"
|
||||
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
|
||||
<rect x="4" y="4" width="6" height="6" />
|
||||
<rect x="14" y="4" width="6" height="6" />
|
||||
<rect x="4" y="14" width="6" height="6" />
|
||||
<rect x="14" y="14" width="6" height="6" /></svg
|
||||
>Dashboard</a
|
||||
>
|
||||
<a class="nav-link" href="/settings"
|
||||
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
|
||||
<circle cx="12" cy="12" r="3" />
|
||||
<path
|
||||
d="M12 2v3m0 14v3M2 12h3m14 0h3m-2.9-7.1-2.1 2.1M4.9 19.1 7 17m0-10-2.1-2.1m12.2 14.2-2.1-2.1" /></svg
|
||||
>Settings</a
|
||||
>
|
||||
<a class="nav-link" href="/viewer"
|
||||
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true"><path d="M4 5h16v12H8l-4 3z" /></svg>Message
|
||||
Viewer</a
|
||||
>
|
||||
<a class="nav-link" href="/swagger"
|
||||
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true">
|
||||
<path d="m8 5-5 7 5 7m8-14 5 7-5 7M14 3l-4 18" /></svg
|
||||
>API Docs</a
|
||||
>
|
||||
<a class="nav-link" href="/health"
|
||||
><svg class="nav-icon" viewBox="0 0 24 24" aria-hidden="true"><path d="M3 12h4l2-6 4 12 2-6h6" /></svg
|
||||
>Health</a
|
||||
>
|
||||
</nav>
|
||||
|
||||
<section class="status-card" id="active-account-card">
|
||||
@@ -39,6 +65,13 @@
|
||||
</aside>
|
||||
|
||||
<main class="content">
|
||||
<header class="dashboard-topbar">
|
||||
<div class="dashboard-title"><span class="menu-icon" aria-hidden="true">☰</span>Dashboard</div>
|
||||
<div class="runtime-status">
|
||||
<span class="runtime-dot"></span>Service: Running <i></i> Scraper: Idle <i></i
|
||||
><span class="runtime-clock"></span>
|
||||
</div>
|
||||
</header>
|
||||
<!-- Account Tabs -->
|
||||
<div id="account-tabs" class="account-tabs"></div>
|
||||
|
||||
@@ -163,18 +196,38 @@
|
||||
<div class="panel stat-panel">
|
||||
<div class="section-title">Channels</div>
|
||||
<div class="stat-value channel-count">0</div>
|
||||
<div class="stat-caption">Tracked channels</div>
|
||||
<div class="stat-meta">
|
||||
<span><strong class="channel-active-count">0</strong> Active</span
|
||||
><span><strong class="channel-inactive-count">0</strong> Inactive</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="panel stat-panel">
|
||||
<div class="section-title">Media scraping</div>
|
||||
<div class="stat-value stat-compact scrape-media-label">OFF</div>
|
||||
<div class="stat-caption">Items downloaded</div>
|
||||
<div class="stat-meta">
|
||||
<span><strong class="media-session-count">0</strong> This session</span
|
||||
><span><strong class="media-total-count">0</strong> Total</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="panel stat-panel">
|
||||
<div class="section-title">Forwarding rules</div>
|
||||
<div class="stat-value forwarding-count">0</div>
|
||||
<div class="stat-caption">Active rules</div>
|
||||
<div class="stat-meta">
|
||||
<span><strong class="forwarding-active-count">0</strong> Active</span
|
||||
><span><strong>0</strong> Paused</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="panel stat-panel">
|
||||
<div class="section-title">Account health</div>
|
||||
<div class="stat-value stat-compact account-health-label">-</div>
|
||||
<div class="stat-caption">Session status</div>
|
||||
<div class="stat-meta">
|
||||
<span><strong class="health-session-value">-</strong> Session</span
|
||||
><span><strong class="health-rate-value">-</strong> Rate limit</span>
|
||||
</div>
|
||||
<div class="muted small account-health-detail"></div>
|
||||
</div>
|
||||
</section>
|
||||
@@ -319,6 +372,6 @@
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script src="/static/app.js?v=4"></script>
|
||||
<script src="/static/app.js?v=5"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Telegram Scraper Settings</title>
|
||||
<link rel="stylesheet" href="/static/style.css?v=6" />
|
||||
<link rel="stylesheet" href="/static/style.css?v=8" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="app-shell settings-page-shell">
|
||||
|
||||
@@ -260,6 +260,7 @@ async function importAccount(event) {
|
||||
await loadAccounts();
|
||||
await loadAccount(accountId);
|
||||
showToast(`Imported ${accountId}.`, 'success');
|
||||
showToast('Credentials (api_id/api_hash) are not exported for security — re-enter them in Settings if needed.', 'warn');
|
||||
} catch (err) {
|
||||
showToast(`Failed to import account: ${err.message}`, 'error');
|
||||
}
|
||||
|
||||
+860
@@ -1380,3 +1380,863 @@ tbody tr:hover {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
/* ── GPU Radar visual system ────────────────────────────
|
||||
A restrained, data-first monochrome treatment inspired by gpu.chernuha.space. */
|
||||
:root {
|
||||
--bg-color: #080808;
|
||||
--panel: #101010;
|
||||
--panel-solid: #151515;
|
||||
--panel-alt: #0b0b0b;
|
||||
--text-color: #f5f5f5;
|
||||
--dim: #9a9a9a;
|
||||
--line: #2b2b2b;
|
||||
--accent: #f1f1f1;
|
||||
--accent-hover: #ffffff;
|
||||
--accent-soft: rgba(255, 255, 255, 0.08);
|
||||
--accent-line: #3b3b3b;
|
||||
--accent-2: #d8d8d8;
|
||||
--bubble-own: #181818;
|
||||
--bubble-other: #101010;
|
||||
--danger: #d0d0d0;
|
||||
--ok: #e0e0e0;
|
||||
--warn: #b8b8b8;
|
||||
--info: #d7d7d7;
|
||||
--radius: 8px;
|
||||
--shadow: none;
|
||||
--font-ui: Inter, 'SF Pro Display', 'Helvetica Neue', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
|
||||
}
|
||||
|
||||
body {
|
||||
font-size: 14px;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
a {
|
||||
border-bottom: 0;
|
||||
}
|
||||
a:hover,
|
||||
button:hover {
|
||||
border-color: var(--accent-line);
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
|
||||
.app-shell {
|
||||
grid-template-columns: 236px minmax(0, 1fr);
|
||||
min-height: 100vh;
|
||||
}
|
||||
.sidebar {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
height: 100vh;
|
||||
padding: 18px 12px;
|
||||
background: var(--panel-alt);
|
||||
border-right-color: var(--line);
|
||||
}
|
||||
.brand-block {
|
||||
position: relative;
|
||||
display: block;
|
||||
min-height: 64px;
|
||||
padding: 5px 9px 16px;
|
||||
}
|
||||
.brand-block::before {
|
||||
content: '◈';
|
||||
position: absolute;
|
||||
top: 8px;
|
||||
left: 9px;
|
||||
color: var(--text-color);
|
||||
font-size: 25px;
|
||||
line-height: 1;
|
||||
}
|
||||
.brand-block h1,
|
||||
.dialog-header h2 {
|
||||
color: var(--text-color);
|
||||
font-size: 15px;
|
||||
line-height: 1.12;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
.brand-block h1 {
|
||||
margin-left: 36px;
|
||||
}
|
||||
.brand-block .eyebrow {
|
||||
margin: 0 0 3px 36px;
|
||||
color: var(--muted, #747474);
|
||||
font-size: 12px;
|
||||
letter-spacing: 0;
|
||||
text-transform: none;
|
||||
}
|
||||
.brand-block .muted {
|
||||
display: none;
|
||||
}
|
||||
.eyebrow,
|
||||
.section-title {
|
||||
color: #747474;
|
||||
font-size: 10px;
|
||||
font-weight: 650;
|
||||
letter-spacing: 0.08em;
|
||||
}
|
||||
.muted {
|
||||
color: #9a9a9a;
|
||||
}
|
||||
|
||||
.nav-links {
|
||||
gap: 5px;
|
||||
margin: 10px 0 24px;
|
||||
}
|
||||
.nav-link,
|
||||
.button,
|
||||
input,
|
||||
.content-tab,
|
||||
.viewer-channel-item,
|
||||
.account-tab {
|
||||
border-color: transparent;
|
||||
border-radius: 6px;
|
||||
background: transparent;
|
||||
}
|
||||
.nav-link {
|
||||
justify-content: flex-start;
|
||||
min-height: 46px;
|
||||
padding: 0 13px;
|
||||
color: #979797;
|
||||
font-size: 13px;
|
||||
font-weight: 650;
|
||||
}
|
||||
.nav-link.active {
|
||||
border-color: #3b3b3b;
|
||||
color: var(--text-color);
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
.nav-link::before {
|
||||
display: none;
|
||||
}
|
||||
.nav-icon {
|
||||
display: inline-grid;
|
||||
width: 18px;
|
||||
place-items: center;
|
||||
color: #c6c6c6;
|
||||
font-size: 17px;
|
||||
font-weight: 400;
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.status-card,
|
||||
.panel,
|
||||
.stat-panel,
|
||||
.job-card,
|
||||
.message-card,
|
||||
.settings-dialog {
|
||||
border-color: var(--line);
|
||||
border-radius: var(--radius);
|
||||
background: var(--panel);
|
||||
box-shadow: none;
|
||||
}
|
||||
.status-card {
|
||||
padding: 14px 10px 4px;
|
||||
background: transparent;
|
||||
border: 0;
|
||||
border-top: 1px solid rgba(255, 255, 255, 0.09);
|
||||
}
|
||||
.status-card + .status-card {
|
||||
margin-top: 12px;
|
||||
}
|
||||
.status-card .button {
|
||||
width: auto;
|
||||
justify-content: flex-start;
|
||||
padding: 0 13px;
|
||||
}
|
||||
.status-badge {
|
||||
padding: 0;
|
||||
border: 0;
|
||||
border-radius: 0;
|
||||
color: var(--text-color);
|
||||
background: transparent;
|
||||
font-size: 12px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.content {
|
||||
max-width: none;
|
||||
padding: 0 28px 40px;
|
||||
}
|
||||
.content::before {
|
||||
display: none;
|
||||
}
|
||||
.dashboard-topbar {
|
||||
display: flex;
|
||||
min-height: 58px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 20px;
|
||||
margin: 0 -28px 20px;
|
||||
padding: 0 28px;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.dashboard-title,
|
||||
.runtime-status {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
color: #efefef;
|
||||
font-size: 13px;
|
||||
font-weight: 650;
|
||||
}
|
||||
.menu-icon {
|
||||
color: #dedede;
|
||||
font-size: 20px;
|
||||
font-weight: 400;
|
||||
}
|
||||
.runtime-status {
|
||||
color: #b8b8b8;
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
}
|
||||
.runtime-status i {
|
||||
width: 1px;
|
||||
height: 17px;
|
||||
background: #363636;
|
||||
}
|
||||
.runtime-dot {
|
||||
width: 7px;
|
||||
height: 7px;
|
||||
border-radius: 50%;
|
||||
background: #e2e2e2;
|
||||
}
|
||||
.runtime-clock::before {
|
||||
content: '◷';
|
||||
color: #e1e1e1;
|
||||
font-size: 17px;
|
||||
}
|
||||
.content-tabs,
|
||||
.account-tabs {
|
||||
gap: 0;
|
||||
margin-bottom: 18px;
|
||||
padding-bottom: 0;
|
||||
border-bottom: 0;
|
||||
}
|
||||
.content-tab,
|
||||
.account-tab {
|
||||
min-height: 38px;
|
||||
padding: 8px 14px;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 0;
|
||||
color: #aaa;
|
||||
font-size: 13px;
|
||||
}
|
||||
.content-tab:first-child,
|
||||
.account-tab:first-child {
|
||||
border-radius: 6px 0 0 6px;
|
||||
}
|
||||
.content-tab + .content-tab,
|
||||
.account-tab + .account-tab {
|
||||
margin-left: -1px;
|
||||
}
|
||||
.content-tab:last-child,
|
||||
.account-tab:last-child {
|
||||
border-radius: 0 6px 6px 0;
|
||||
}
|
||||
.content-tab.active,
|
||||
.account-tab.active {
|
||||
position: relative;
|
||||
z-index: 1;
|
||||
border-color: #535353;
|
||||
color: var(--text-color);
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
|
||||
.panel-grid {
|
||||
gap: 14px;
|
||||
margin-bottom: 14px;
|
||||
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||
}
|
||||
.settings-health-grid {
|
||||
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||
}
|
||||
.panel,
|
||||
.status-card,
|
||||
.job-card,
|
||||
.message-card {
|
||||
padding: 18px;
|
||||
}
|
||||
.panel {
|
||||
margin-bottom: 14px;
|
||||
}
|
||||
.stat-panel {
|
||||
position: relative;
|
||||
min-height: 187px;
|
||||
padding: 18px;
|
||||
}
|
||||
.stat-panel::after {
|
||||
content: '';
|
||||
position: absolute;
|
||||
right: 18px;
|
||||
bottom: 65px;
|
||||
left: 18px;
|
||||
height: 1px;
|
||||
background: #343434;
|
||||
}
|
||||
.stat-panel::before {
|
||||
position: absolute;
|
||||
top: 17px;
|
||||
right: 18px;
|
||||
display: grid;
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
place-items: center;
|
||||
border: 1px solid #9c9c9c;
|
||||
border-radius: 50%;
|
||||
color: #e4e4e4;
|
||||
font-size: 15px;
|
||||
font-weight: 400;
|
||||
}
|
||||
.stat-panel:nth-child(1)::before {
|
||||
content: '⌘';
|
||||
}
|
||||
.stat-panel:nth-child(2)::before {
|
||||
content: '⇩';
|
||||
}
|
||||
.stat-panel:nth-child(3)::before {
|
||||
content: '⇄';
|
||||
}
|
||||
.stat-panel:nth-child(4)::before {
|
||||
content: '◇';
|
||||
}
|
||||
.stat-value {
|
||||
margin-top: 24px;
|
||||
font-size: 28px;
|
||||
font-weight: 650;
|
||||
letter-spacing: -0.04em;
|
||||
}
|
||||
.stat-compact {
|
||||
font-size: 19px;
|
||||
}
|
||||
.stat-panel .account-health-detail {
|
||||
position: absolute;
|
||||
right: 18px;
|
||||
bottom: 17px;
|
||||
left: 18px;
|
||||
}
|
||||
.stat-panel .section-title {
|
||||
max-width: calc(100% - 34px);
|
||||
}
|
||||
.panel-header,
|
||||
.dialog-header,
|
||||
.job-head {
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
}
|
||||
.panel-header h2,
|
||||
.settings-hero h2 {
|
||||
font-size: 16px;
|
||||
font-weight: 650;
|
||||
letter-spacing: -0.02em;
|
||||
}
|
||||
.panel-header .muted {
|
||||
margin-top: 4px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.button {
|
||||
min-height: 36px;
|
||||
padding: 7px 12px;
|
||||
border: 1px solid var(--line);
|
||||
color: #d4d4d4;
|
||||
font-size: 12px;
|
||||
font-weight: 650;
|
||||
}
|
||||
.button.primary {
|
||||
border-color: #d9d9d9;
|
||||
color: #111;
|
||||
background: #f1f1f1;
|
||||
}
|
||||
.button.primary:hover {
|
||||
border-color: #fff;
|
||||
background: #fff;
|
||||
color: #080808;
|
||||
}
|
||||
.button.danger {
|
||||
color: #c5c5c5;
|
||||
}
|
||||
.button.button-small {
|
||||
min-height: 32px;
|
||||
padding: 5px 9px;
|
||||
}
|
||||
input {
|
||||
min-height: 36px;
|
||||
padding: 8px 11px;
|
||||
border: 1px solid var(--line);
|
||||
color: var(--text-color);
|
||||
background: #0d0d0d;
|
||||
font-size: 12px;
|
||||
}
|
||||
input:focus {
|
||||
border-color: #777;
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
padding: 12px 10px;
|
||||
border-bottom-color: var(--line);
|
||||
font-size: 12px;
|
||||
}
|
||||
th {
|
||||
color: #888;
|
||||
font-size: 10px;
|
||||
font-weight: 650;
|
||||
letter-spacing: 0.06em;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
tbody tr:hover {
|
||||
background: rgba(255, 255, 255, 0.035);
|
||||
}
|
||||
.job-card {
|
||||
border-radius: 6px;
|
||||
}
|
||||
.job-status {
|
||||
color: #aaa;
|
||||
font-size: 10px;
|
||||
}
|
||||
.job-logs,
|
||||
.log-viewer {
|
||||
border-color: var(--line);
|
||||
border-radius: 6px;
|
||||
background: #0b0b0b;
|
||||
font-family: var(--font-mono);
|
||||
font-size: 12px;
|
||||
}
|
||||
.log-line {
|
||||
border-bottom-color: #202020;
|
||||
}
|
||||
.log-line:hover {
|
||||
background: rgba(255, 255, 255, 0.04);
|
||||
}
|
||||
|
||||
.switch {
|
||||
width: 42px;
|
||||
min-width: 42px;
|
||||
height: 22px;
|
||||
}
|
||||
.switch-slider {
|
||||
border-color: #3a3a3a;
|
||||
background: #181818;
|
||||
}
|
||||
.switch-slider::before {
|
||||
top: 3px;
|
||||
left: 3px;
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
background: #8d8d8d;
|
||||
}
|
||||
.switch input:checked + .switch-slider {
|
||||
border-color: #858585;
|
||||
background: #f1f1f1;
|
||||
}
|
||||
.switch input:checked + .switch-slider::before {
|
||||
transform: translateX(20px);
|
||||
background: #111;
|
||||
}
|
||||
.checkbox-row,
|
||||
.settings-account-button,
|
||||
.account-list-item,
|
||||
.add-account-form,
|
||||
.import-account-row,
|
||||
.qr-wrap {
|
||||
border-color: var(--line);
|
||||
border-radius: 6px;
|
||||
background: #111;
|
||||
}
|
||||
.settings-account-button.active {
|
||||
border-color: #535353;
|
||||
background: var(--accent-soft);
|
||||
}
|
||||
|
||||
.settings-hero {
|
||||
border-top: 0;
|
||||
}
|
||||
.settings-dialog {
|
||||
border-color: #3b3b3b;
|
||||
}
|
||||
.dialog-shell {
|
||||
padding: 20px;
|
||||
}
|
||||
.settings-section {
|
||||
border-top-color: var(--line);
|
||||
}
|
||||
.toast {
|
||||
border-color: #4a4a4a;
|
||||
border-radius: 6px;
|
||||
background: #151515;
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.panel-grid {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
}
|
||||
@media (max-width: 768px) {
|
||||
.app-shell {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.sidebar {
|
||||
position: static;
|
||||
height: auto;
|
||||
padding: 12px;
|
||||
border-right: 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
.brand-block {
|
||||
min-height: 46px;
|
||||
padding-bottom: 5px;
|
||||
}
|
||||
.nav-links {
|
||||
display: flex;
|
||||
overflow-x: auto;
|
||||
margin: 8px 0 0;
|
||||
}
|
||||
.nav-link {
|
||||
min-height: 34px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.status-card {
|
||||
display: none;
|
||||
}
|
||||
.content {
|
||||
padding: 16px;
|
||||
}
|
||||
.content::before {
|
||||
height: 0;
|
||||
margin: -16px -16px 16px;
|
||||
}
|
||||
}
|
||||
@media (max-width: 560px) {
|
||||
.panel-grid,
|
||||
.settings-health-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.content-tabs,
|
||||
.account-tabs {
|
||||
overflow-x: auto;
|
||||
flex-wrap: nowrap;
|
||||
}
|
||||
.content-tab,
|
||||
.account-tab {
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
/* Dashboard reference fidelity pass */
|
||||
:root {
|
||||
--sidebar-width: 244px;
|
||||
}
|
||||
|
||||
.app-shell {
|
||||
grid-template-columns: var(--sidebar-width) minmax(0, 1fr);
|
||||
width: 100vw;
|
||||
max-width: 100vw;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
width: var(--sidebar-width);
|
||||
padding: 18px 12px;
|
||||
}
|
||||
|
||||
.brand-block {
|
||||
min-height: 62px;
|
||||
padding: 5px 9px 14px;
|
||||
}
|
||||
|
||||
.brand-block::before {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.brand-icon {
|
||||
position: absolute;
|
||||
top: 9px;
|
||||
left: 9px;
|
||||
width: 27px;
|
||||
height: 27px;
|
||||
fill: none;
|
||||
stroke: #f1f1f1;
|
||||
stroke-width: 1.65;
|
||||
stroke-linecap: round;
|
||||
stroke-linejoin: round;
|
||||
}
|
||||
|
||||
.brand-block h1 {
|
||||
margin-left: 37px;
|
||||
font-size: 15px;
|
||||
}
|
||||
|
||||
.brand-block .eyebrow {
|
||||
margin-left: 37px;
|
||||
color: #f5f5f5;
|
||||
font-size: 15px;
|
||||
font-weight: 650;
|
||||
}
|
||||
|
||||
.nav-links {
|
||||
gap: 5px;
|
||||
margin: 6px 0 0;
|
||||
}
|
||||
|
||||
.nav-link {
|
||||
min-height: 45px;
|
||||
gap: 12px;
|
||||
padding: 0 13px;
|
||||
color: #a1a1a1;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.nav-link.active {
|
||||
border-color: transparent;
|
||||
background: #202020;
|
||||
}
|
||||
|
||||
a.nav-link {
|
||||
border-bottom-color: transparent;
|
||||
}
|
||||
|
||||
.nav-icon {
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
flex: 0 0 20px;
|
||||
fill: none;
|
||||
stroke: currentColor;
|
||||
stroke-width: 1.7;
|
||||
stroke-linecap: round;
|
||||
stroke-linejoin: round;
|
||||
}
|
||||
|
||||
.status-card:first-of-type {
|
||||
margin-top: auto;
|
||||
}
|
||||
|
||||
.status-card {
|
||||
padding: 15px 10px 5px;
|
||||
}
|
||||
|
||||
.content {
|
||||
min-width: 0;
|
||||
padding: 0 20px 28px;
|
||||
}
|
||||
|
||||
.dashboard-topbar {
|
||||
min-height: 58px;
|
||||
margin: 0 -20px 20px;
|
||||
padding: 0 28px;
|
||||
}
|
||||
|
||||
.dashboard-title {
|
||||
gap: 22px;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.menu-icon {
|
||||
font-size: 21px;
|
||||
}
|
||||
|
||||
.runtime-status {
|
||||
gap: 13px;
|
||||
}
|
||||
|
||||
.account-tabs {
|
||||
margin-bottom: 18px;
|
||||
}
|
||||
|
||||
.account-tab {
|
||||
min-height: 36px;
|
||||
padding: 7px 16px;
|
||||
}
|
||||
|
||||
.content-tabs {
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.content-tab {
|
||||
min-height: 35px;
|
||||
padding: 6px 13px;
|
||||
}
|
||||
|
||||
.panel-grid {
|
||||
gap: 16px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.panel,
|
||||
.stat-panel {
|
||||
border-color: #303030;
|
||||
border-radius: 7px;
|
||||
}
|
||||
|
||||
.stat-panel {
|
||||
min-height: 208px;
|
||||
padding: 18px 18px 16px;
|
||||
}
|
||||
|
||||
.stat-panel::after {
|
||||
right: 16px;
|
||||
bottom: 66px;
|
||||
left: 16px;
|
||||
background: #363636;
|
||||
}
|
||||
|
||||
.stat-panel::before {
|
||||
top: 16px;
|
||||
right: 16px;
|
||||
width: 25px;
|
||||
height: 25px;
|
||||
}
|
||||
|
||||
.section-title {
|
||||
color: #ebebeb;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
letter-spacing: -0.01em;
|
||||
text-transform: none;
|
||||
}
|
||||
|
||||
.stat-value {
|
||||
margin-top: 20px;
|
||||
font-size: 27px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.stat-caption {
|
||||
margin-top: 3px;
|
||||
color: #bdbdbd;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.stat-meta {
|
||||
position: absolute;
|
||||
right: 18px;
|
||||
bottom: 15px;
|
||||
left: 18px;
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 12px;
|
||||
color: #bdbdbd;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.stat-meta span + span {
|
||||
padding-left: 12px;
|
||||
border-left: 1px solid #363636;
|
||||
}
|
||||
|
||||
.stat-meta strong {
|
||||
display: block;
|
||||
margin-bottom: 1px;
|
||||
color: #ededed;
|
||||
font-size: 14px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.stat-panel .account-health-detail {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.panel {
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.panel-header {
|
||||
min-height: 52px;
|
||||
padding: 0 18px;
|
||||
border-bottom: 1px solid #343434;
|
||||
}
|
||||
|
||||
.panel-header h2 {
|
||||
font-size: 15px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.panel-header .muted {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.inline-form {
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.inline-form input {
|
||||
width: 168px;
|
||||
}
|
||||
|
||||
.table-wrap {
|
||||
padding: 0 8px;
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
height: 38px;
|
||||
padding: 9px 10px;
|
||||
border-bottom-color: #303030;
|
||||
}
|
||||
|
||||
th {
|
||||
color: #e6e6e6;
|
||||
font-size: 11px;
|
||||
font-weight: 500;
|
||||
letter-spacing: 0;
|
||||
text-transform: none;
|
||||
}
|
||||
|
||||
td {
|
||||
color: #d0d0d0;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.action-row .button,
|
||||
td .button,
|
||||
.job-card .button {
|
||||
min-height: 29px;
|
||||
padding: 4px 9px;
|
||||
border-radius: 5px;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.jobs-list {
|
||||
padding: 12px 16px 16px;
|
||||
}
|
||||
|
||||
.job-card {
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.app-shell {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
width: auto;
|
||||
}
|
||||
|
||||
.content {
|
||||
padding: 0 16px 20px;
|
||||
}
|
||||
|
||||
.dashboard-topbar {
|
||||
margin: 0 -16px 16px;
|
||||
padding: 0 16px;
|
||||
}
|
||||
}
|
||||
|
||||
/* Stat panel: restore left padding & drop card icons */
|
||||
.stat-panel {
|
||||
padding: 18px 18px 16px;
|
||||
}
|
||||
.stat-panel .section-title {
|
||||
max-width: none;
|
||||
}
|
||||
.stat-panel .stat-value,
|
||||
.stat-panel .stat-caption {
|
||||
padding-left: 2px;
|
||||
}
|
||||
.stat-panel .stat-caption {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
.stat-panel::before {
|
||||
content: none !important;
|
||||
}
|
||||
|
||||
+6
-1
@@ -49,5 +49,10 @@ loadSpec()
|
||||
.then(renderSpec)
|
||||
.catch((error) => {
|
||||
console.error(error);
|
||||
document.getElementById('api-docs').innerHTML = `<section class="panel">${error.message}</section>`;
|
||||
const docsEl = document.getElementById('api-docs');
|
||||
docsEl.textContent = '';
|
||||
const section = document.createElement('section');
|
||||
section.className = 'panel';
|
||||
section.textContent = error.message;
|
||||
docsEl.appendChild(section);
|
||||
});
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Telegram Scraper Viewer</title>
|
||||
<link rel="stylesheet" href="/static/style.css?v=6" />
|
||||
<link rel="stylesheet" href="/static/style.css?v=8" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="viewer-shell">
|
||||
|
||||
+1016
-278
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user