# Telegram Scraper

Telegram scraper on top of Telethon with:

- CLI workflow for scraping, exports, media recovery, and forwarding
- lightweight Web UI served by Python
- SQLite storage per tracked chat/channel
- Docker/Compose setup for "run it and leave it on the server"

## What It Does

- Scrapes messages with metadata: views, forwards, reactions, post author
- Downloads media into per-channel folders
- Stores messages in SQLite
- Exports to CSV and JSON
- Supports continuous scraping
- Supports forwarding rules
- Lets you browse tracked channels and local exports in a web panel

## Project Layout

```text
.
├── main.py                           # starts the web server
├── webui_server.py                   # HTTP server + API
├── webui/                            # plain HTML/CSS/JS frontend
├── telegram_scraper_with_forwarding.py
├── data/
│   ├── state.json                    # shared settings and credentials
│   └── <channel_id>/<channel_id>.db  # SQLite databases
└── session/                          # Telethon session files
```

## Requirements

- Python 3.11+
- Telegram account
- `api_id` and `api_hash` from https://my.telegram.org

## Telegram API Credentials

1. Go to https://my.telegram.org/auth
2. Sign in with your Telegram account
3. Open `API development tools`
4. Create an application
5. Save:
   - `api_id`
   - `api_hash`

## Install

### Local

```bash
pip install -r requirements.txt
```

Or with `uv`:

```bash
uv sync
```

## Running

### Web UI

Starts on `0.0.0.0:8080` by default:

```bash
python main.py
```

Or:

```bash
uv run python main.py
```

Open:

```text
http://<server-ip>:8080
```

The current web panel includes:

- tracked channels overview
- background job queue
- scrape/export/media actions
- shared `scrape_media` toggle
- local message viewer powered by SQLite + media files

### CLI

If you still want the old interactive mode:

```bash
python telegram_scraper_with_forwarding.py
```

## Docker

Build and run:

```bash
docker compose up -d --build
```

Then open:

```text
http://<server-ip>:8080
```

### Volumes

- `./data:/app/data` for databases, exports, and `state.json`
- `session:/app/session` for Telethon sessions

If you already logged in before with the same compose volume and did not remove it, the session should be reused.

## Shared State

The Web UI and CLI share the same files:

- `data/state.json`
- `session/session.session`

That means:

- credentials can stay in `state.json`
- both entry points use the same tracked channels
- both entry points can reuse the same Telegram login session

## Data Storage

### SQLite

Each tracked channel gets its own database:

```text
data/<channel_id>/<channel_id>.db
```

Main fields include:

- `message_id`
- `date`
- `sender_id`
- `first_name`
- `last_name`
- `username`
- `message`
- `media_type`
- `media_path`
- `reply_to`
- `post_author`
- `views`
- `forwards`
- `reactions`

### Media

Media files are stored in:

```text
data/<channel_id>/media/
```

### Exports

Generated into the same channel folder:

- `data/<channel_id>/<channel_id>_<username>.csv`
- `data/<channel_id>/<channel_id>_<username>.json`

## Notes

- The web server is intentionally simple: plain HTML/CSS/JS, no frontend framework
- The viewer currently reads local SQLite data, not Telegram export HTML directly
- If dependencies are installed but the Telegram session is missing, the Web UI falls back to read-only status until login is available

## Disclaimer

Use responsibly and make sure your usage complies with Telegram rules, local law, and any privacy obligations.