187 lines
3.6 KiB
Markdown
187 lines
3.6 KiB
Markdown
# Telegram Scraper
|
|
|
|
Telegram scraper on top of Telethon with:
|
|
|
|
- CLI workflow for scraping, exports, media recovery, and forwarding
|
|
- lightweight Web UI served by Python
|
|
- SQLite storage per tracked chat/channel
|
|
- Docker/Compose setup for "run it and leave it on the server"
|
|
|
|
## What It Does
|
|
|
|
- Scrapes messages with metadata: views, forwards, reactions, post author
|
|
- Downloads media into per-channel folders
|
|
- Stores messages in SQLite
|
|
- Exports to CSV and JSON
|
|
- Supports continuous scraping
|
|
- Supports forwarding rules
|
|
- Lets you browse tracked channels and local exports in a web panel
|
|
|
|
## Project Layout
|
|
|
|
```text
|
|
.
|
|
├── main.py # starts the web server
|
|
├── webui_server.py # HTTP server + API
|
|
├── webui/ # plain HTML/CSS/JS frontend
|
|
├── telegram_scraper_with_forwarding.py
|
|
├── data/
|
|
│ ├── state.json # shared settings and credentials
|
|
│ └── <channel_id>/<channel_id>.db # SQLite databases
|
|
└── session/ # Telethon session files
|
|
```
|
|
|
|
## Requirements
|
|
|
|
- Python 3.11+
|
|
- Telegram account
|
|
- `api_id` and `api_hash` from https://my.telegram.org
|
|
|
|
## Telegram API Credentials
|
|
|
|
1. Go to https://my.telegram.org/auth
|
|
2. Sign in with your Telegram account
|
|
3. Open `API development tools`
|
|
4. Create an application
|
|
5. Save:
|
|
- `api_id`
|
|
- `api_hash`
|
|
|
|
## Install
|
|
|
|
### Local
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Or with `uv`:
|
|
|
|
```bash
|
|
uv sync
|
|
```
|
|
|
|
## Running
|
|
|
|
### Web UI
|
|
|
|
Starts on `0.0.0.0:8080` by default:
|
|
|
|
```bash
|
|
python main.py
|
|
```
|
|
|
|
Or:
|
|
|
|
```bash
|
|
uv run python main.py
|
|
```
|
|
|
|
Open:
|
|
|
|
```text
|
|
http://<server-ip>:8080
|
|
```
|
|
|
|
The current web panel includes:
|
|
|
|
- tracked channels overview
|
|
- background job queue
|
|
- scrape/export/media actions
|
|
- shared `scrape_media` toggle
|
|
- local message viewer powered by SQLite + media files
|
|
|
|
### CLI
|
|
|
|
If you still want the old interactive mode:
|
|
|
|
```bash
|
|
python telegram_scraper_with_forwarding.py
|
|
```
|
|
|
|
## Docker
|
|
|
|
Build and run:
|
|
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Then open:
|
|
|
|
```text
|
|
http://<server-ip>:8080
|
|
```
|
|
|
|
### Volumes
|
|
|
|
- `./data:/app/data` for databases, exports, and `state.json`
|
|
- `session:/app/session` for Telethon sessions
|
|
|
|
If you already logged in before with the same compose volume and did not remove it, the session should be reused.
|
|
|
|
## Shared State
|
|
|
|
The Web UI and CLI share the same files:
|
|
|
|
- `data/state.json`
|
|
- `session/session.session`
|
|
|
|
That means:
|
|
|
|
- credentials can stay in `state.json`
|
|
- both entry points use the same tracked channels
|
|
- both entry points can reuse the same Telegram login session
|
|
|
|
## Data Storage
|
|
|
|
### SQLite
|
|
|
|
Each tracked channel gets its own database:
|
|
|
|
```text
|
|
data/<channel_id>/<channel_id>.db
|
|
```
|
|
|
|
Main fields include:
|
|
|
|
- `message_id`
|
|
- `date`
|
|
- `sender_id`
|
|
- `first_name`
|
|
- `last_name`
|
|
- `username`
|
|
- `message`
|
|
- `media_type`
|
|
- `media_path`
|
|
- `reply_to`
|
|
- `post_author`
|
|
- `views`
|
|
- `forwards`
|
|
- `reactions`
|
|
|
|
### Media
|
|
|
|
Media files are stored in:
|
|
|
|
```text
|
|
data/<channel_id>/media/
|
|
```
|
|
|
|
### Exports
|
|
|
|
Generated into the same channel folder:
|
|
|
|
- `data/<channel_id>/<channel_id>_<username>.csv`
|
|
- `data/<channel_id>/<channel_id>_<username>.json`
|
|
|
|
## Notes
|
|
|
|
- The web server is intentionally simple: plain HTML/CSS/JS, no frontend framework
|
|
- The viewer currently reads local SQLite data, not Telegram export HTML directly
|
|
- If dependencies are installed but the Telegram session is missing, the Web UI falls back to read-only status until login is available
|
|
|
|
## Disclaimer
|
|
|
|
Use responsibly and make sure your usage complies with Telegram rules, local law, and any privacy obligations.
|