Telegram Scraper

Telegram scraper on top of Telethon with:

• CLI workflow for scraping, exports, media recovery, and forwarding
• lightweight Web UI served by Python
• SQLite storage per tracked chat/channel
• Docker/Compose setup for "run it and leave it on the server"

What It Does

• Scrapes messages with metadata: views, forwards, reactions, post author
• Downloads media into per-channel folders
• Stores messages in SQLite
• Exports to CSV and JSON
• Supports continuous scraping
• Supports forwarding rules
• Lets you browse tracked channels and local exports in a web panel

Project Layout

.
├── main.py                           # starts the web server
├── webui_server.py                   # HTTP server + API
├── webui/                            # plain HTML/CSS/JS frontend
├── telegram_scraper_with_forwarding.py
├── data/
│   ├── state.json                    # shared settings and credentials
│   └── <channel_id>/<channel_id>.db  # SQLite databases
└── session/                          # Telethon session files

Requirements

• Python 3.11+
• Telegram account
• api_id and api_hash from https://my.telegram.org

Telegram API Credentials

1. Go to https://my.telegram.org/auth
2. Sign in with your Telegram account
3. Open API development tools
4. Create an application
5. Save:
   • api_id
   • api_hash

Install

Local

pip install -r requirements.txt

Or with uv:

uv sync

Smoke Test

A smoke test is a quick "does it start and answer basic requests?" check. It does not replace full tests, but it catches broken imports, routes, and JSON responses.

python scripts/smoke_test.py

The smoke test starts the web server on 127.0.0.1:18080 and disables automatic continuous scraping for that process.

Running

Web UI

Starts on 0.0.0.0:8080 by default:

python main.py

Or:

uv run python main.py

Open:

http://<server-ip>:8080

The current web panel includes:

• tracked channels overview
• background job queue
• scrape/export/media actions
• shared scrape_media toggle
• local message viewer powered by SQLite + media files
• API docs at /swagger and /openapi.json
• health checks at /health and /health/continuous

CLI

If you still want the old interactive mode:

python telegram_scraper_with_forwarding.py

Docker

Build and run:

docker compose up -d --build

Then open:

http://<server-ip>:8080

Volumes

• ./data:/app/data for databases, exports, and state.json
• session:/app/session for Telethon sessions

If you already logged in before with the same compose volume and did not remove it, the session should be reused.

Shared State

The Web UI and CLI share the same files:

• data/state.json
• session/session.session

That means:

• credentials can stay in state.json
• both entry points use the same tracked channels
• both entry points can reuse the same Telegram login session

Data Storage

SQLite

Each tracked channel gets its own database:

data/<channel_id>/<channel_id>.db

Main fields include:

• message_id
• date
• sender_id
• first_name
• last_name
• username
• message
• media_type
• media_path
• reply_to
• post_author
• views
• forwards
• reactions

Media

Media files are stored in:

data/<channel_id>/media/

Exports

Generated into the same channel folder:

• data/<channel_id>/<channel_id>_<username>.csv
• data/<channel_id>/<channel_id>_<username>.json

Notes

• The web server is intentionally simple: plain HTML/CSS/JS, no frontend framework
• The viewer currently reads local SQLite data, not Telegram export HTML directly
• If dependencies are installed but the Telegram session is missing, the Web UI falls back to read-only status until login is available

Disclaimer

Use responsibly and make sure your usage complies with Telegram rules, local law, and any privacy obligations.