telegram-scraper/README.md

Telegram Scraper

Telegram scraper on top of Telethon with:

• CLI workflow for scraping, exports, media recovery, and forwarding
• lightweight multi-account Web UI served by Python
• SQLite storage per tracked chat/channel
• Docker/Compose / Kubernetes setup for "run it and leave it on the server"

What It Does

• Scrapes messages with metadata: views, forwards, reactions, post author
• Downloads media into per-channel folders
• Stores messages in SQLite
• Exports to CSV and JSON
• Supports continuous scraping
• Supports forwarding rules
• Multi-account support — multiple Telegram sessions side by side
• Lets you browse tracked channels and local exports in a web panel with account tabs

Project Layout

.
├── main.py                           # starts the web server
├── webui_server.py                   # HTTP server + REST API
├── webui/                            # plain HTML/CSS/JS frontend
│   ├── index.html / app.js           # main dashboard
│   ├── viewer.html / viewer.js       # message viewer with account selector
│   └── swagger.html / swagger.js     # API docs
├── telegram_scraper_with_forwarding.py
├── app_state.py                      # multi-account state management
├── scraper_jobs.py                   # async job runner
├── health.py                         # health check endpoints
├── data/
│   ├── state.json                    # global state (accounts list)
│   └── accounts/<id>/                # per-account data
│       ├── state.json                # per-account settings & credentials
│       └── <channel_id>/             # SQLite DB + media
└── session/
    ├── <account_id>.session          # per-account Telethon session
    └── ...                            # (WAL journal mode for concurrency)

Requirements

• Python 3.11+
• Telegram account
• api_id and api_hash from https://my.telegram.org

Telegram API Credentials

1. Go to https://my.telegram.org/auth
2. Sign in with your Telegram account
3. Open API development tools
4. Create an application
5. Save:
   • api_id
   • api_hash

Install

Local

pip install -r requirements.txt

Or with uv:

uv sync

Smoke Test

A smoke test is a quick "does it start and answer basic requests?" check. It does not replace full tests, but it catches broken imports, routes, and JSON responses.

python scripts/smoke_test.py

The smoke test starts the web server on 127.0.0.1:18080 and disables automatic continuous scraping for that process.

Running

Web UI

Starts on 0.0.0.0:8080 by default:

python main.py

Or:

uv run python main.py

Open:

http://<server-ip>:8080

The current web panel includes:

• multi-account support with account tabs
• tracked channels overview per account
• background job queue
• scrape/export/media actions per account
• shared scrape_media toggle
• local message viewer with account selector
• API docs at /swagger and /openapi.json
• health checks at /health and /health/continuous

CLI

If you still want the old interactive mode:

python telegram_scraper_with_forwarding.py

Docker / Kubernetes

Docker Compose

docker compose up -d --build

Kubernetes

kubectl apply -f k8s/telegram-scraper.yaml

Then open:

http://<server-ip>:7887

Volumes (Docker)

• ./data:/app/data for databases, exports, and state.json
• session:/app/session for Telethon sessions

If you already logged in before with the same compose volume and did not remove it, the session should be reused.

Multi-Account

The app supports multiple Telegram accounts side by side. Each account has:

• its own state.json under data/accounts/<id>/
• its own Telethon session file session/<id>.session
• its own channel databases and media

Use the Web UI account tabs to switch between accounts, or create new ones in Settings.

Legacy single-account data is automatically migrated to the default account on first startup.

Shared State

The Web UI and CLI share the same files:

• data/state.json (global — accounts list)
• data/accounts/<id>/state.json (per-account credentials and settings)
• session/<id>.session (per-account Telethon session, WAL journal mode)

That means:

• credentials persist in state files
• both entry points use the same tracked channels per account
• both entry points can reuse the same Telegram login sessions

Data Storage

SQLite

Each tracked channel per account gets its own database:

data/accounts/<account_id>/<channel_id>/<channel_id>.db

Main fields include:

• message_id
• date
• sender_id
• first_name
• last_name
• username
• message
• media_type
• media_path
• reply_to
• post_author
• views
• forwards
• reactions

Media

Media files are stored in:

data/accounts/<account_id>/<channel_id>/media/

Exports

Generated into the same channel folder:

• data/accounts/<account_id>/<channel_id>/<channel_id>_<username>.csv
• data/accounts/<account_id>/<channel_id>/<channel_id>_<username>.json

Notes

• The web server is intentionally simple: plain HTML/CSS/JS, no frontend framework
• The viewer reads local SQLite data and supports account switching via a dropdown
• Session SQLite files use WAL journal mode to prevent "database is locked" errors when concurrent scraping and jobs run
• The active account tab is persisted in localStorage across page reloads
• If dependencies are installed but the Telegram session is missing, the Web UI falls back to read-only status until login is available

Disclaimer

Use responsibly and make sure your usage complies with Telegram rules, local law, and any privacy obligations.