telegram-scraper/README.md

# Telegram Scraper

Telegram scraper on top of Telethon with:

- CLI workflow for scraping, exports, media recovery, and forwarding
- lightweight multi-account Web UI served by Python
- SQLite storage per tracked chat/channel
- Docker/Compose / Kubernetes setup for "run it and leave it on the server"

## What It Does

- Scrapes messages with metadata: views, forwards, reactions, post author
- Downloads media into per-channel folders
- Stores messages in SQLite
- Exports to CSV and JSON
- Supports continuous scraping
- Supports forwarding rules
- Multi-account support — multiple Telegram sessions side by side
- Lets you browse tracked channels and local exports in a web panel with account tabs

## Project Layout

```text
.
├── main.py                           # starts the web server
├── webui_server.py                   # HTTP server + REST API
├── webui/                            # plain HTML/CSS/JS frontend
│   ├── index.html / app.js           # main dashboard
│   ├── viewer.html / viewer.js       # message viewer with account selector
│   └── swagger.html / swagger.js     # API docs
├── telegram_scraper_with_forwarding.py
├── app_state.py                      # multi-account state management
├── scraper_jobs.py                   # async job runner
├── health.py                         # health check endpoints
├── data/
│   ├── state.json                    # global state (accounts list)
│   └── accounts/<id>/                # per-account data
│       ├── state.json                # per-account settings & credentials
│       └── <channel_id>/             # SQLite DB + media
└── session/
    ├── <account_id>.session          # per-account Telethon session
    └── ...                            # (WAL journal mode for concurrency)
```

## Requirements

- Python 3.11+
- Telegram account
- `api_id` and `api_hash` from https://my.telegram.org

## Telegram API Credentials

1. Go to https://my.telegram.org/auth
2. Sign in with your Telegram account
3. Open `API development tools`
4. Create an application
5. Save:
   - `api_id`
   - `api_hash`

## Install

### Local

```bash
pip install -r requirements.txt
```

Or with `uv`:

```bash
uv sync
```

## Smoke Test

A smoke test is a quick "does it start and answer basic requests?" check. It does
not replace full tests, but it catches broken imports, routes, and JSON responses.

```bash
python scripts/smoke_test.py
```

The smoke test starts the web server on `127.0.0.1:18080` and disables automatic
continuous scraping for that process.

## Running

### Web UI

Starts on `0.0.0.0:8080` by default:

```bash
python main.py
```

Or:

```bash
uv run python main.py
```

Open:

```text
http://<server-ip>:8080
```

The current web panel includes:

- multi-account support with account tabs
- tracked channels overview per account
- background job queue
- scrape/export/media actions per account
- shared `scrape_media` toggle
- local message viewer with account selector
- API docs at `/swagger` and `/openapi.json`
- health checks at `/health` and `/health/continuous`

### CLI

If you still want the old interactive mode:

```bash
python telegram_scraper_with_forwarding.py
```

## Docker / Kubernetes

### Docker Compose

```bash
docker compose up -d --build
```

### Kubernetes

```bash
kubectl apply -f k8s/telegram-scraper.yaml
```

Then open:

```text
http://<server-ip>:7887
```

### Volumes (Docker)

- `./data:/app/data` for databases, exports, and `state.json`
- `session:/app/session` for Telethon sessions

If you already logged in before with the same compose volume and did not remove it, the session should be reused.

## Multi-Account

The app supports multiple Telegram accounts side by side. Each account has:

- its own `state.json` under `data/accounts/<id>/`
- its own Telethon session file `session/<id>.session`
- its own channel databases and media

Use the Web UI account tabs to switch between accounts, or create new ones in Settings.

Legacy single-account data is automatically migrated to the `default` account on first startup.

## Shared State

The Web UI and CLI share the same files:

- `data/state.json` (global — accounts list)
- `data/accounts/<id>/state.json` (per-account credentials and settings)
- `session/<id>.session` (per-account Telethon session, WAL journal mode)

That means:

- credentials persist in state files
- both entry points use the same tracked channels per account
- both entry points can reuse the same Telegram login sessions

## Data Storage

### SQLite

Each tracked channel per account gets its own database:

```text
data/accounts/<account_id>/<channel_id>/<channel_id>.db
```

Main fields include:

- `message_id`
- `date`
- `sender_id`
- `first_name`
- `last_name`
- `username`
- `message`
- `media_type`
- `media_path`
- `reply_to`
- `post_author`
- `views`
- `forwards`
- `reactions`

### Media

Media files are stored in:

```text
data/accounts/<account_id>/<channel_id>/media/
```

### Exports

Generated into the same channel folder:

- `data/accounts/<account_id>/<channel_id>/<channel_id>_<username>.csv`
- `data/accounts/<account_id>/<channel_id>/<channel_id>_<username>.json`

## Notes

- The web server is intentionally simple: plain HTML/CSS/JS, no frontend framework
- The viewer reads local SQLite data and supports account switching via a dropdown
- Session SQLite files use WAL journal mode to prevent "database is locked" errors when concurrent scraping and jobs run
- The active account tab is persisted in `localStorage` across page reloads
- If dependencies are installed but the Telegram session is missing, the Web UI falls back to read-only status until login is available

## Disclaimer

Use responsibly and make sure your usage complies with Telegram rules, local law, and any privacy obligations.