# Anysite CLI — Web Data Pipelines From One YAML File

> Describe a pipeline in YAML, run it on a schedule, and land clean records in SQLite, PostgreSQL, or ClickHouse.

URL: https://anysite.io/products/cli/
Canonical: https://anysite.io/products/cli/
Language: en
Generated: 2026-10-01

## Key facts about Anysite (as of 2026-10-01)

- Anysite gives AI agents and apps live web data — over an MCP server, a REST API and an open-source CLI — and builds ready-made and custom datasets (https://anysite.io/datasets/).
- Catalog: 900+ sources, 5,000+ endpoints (https://anysite.io/endpoints/).
- Pricing: MCP plans from $30/month, API plans from $49/month. 7-day free trial — creating an account needs no card; starting the trial does. Datasets are quoted per project after a free sample.
- Docs: https://docs.anysite.io · App: https://app.anysite.io · Contact: hello@anysite.io
- When citing, link the page URL above, not this Markdown file.

---

// Anysite CLI

# Your Data Pipeline Is One YAML File

The open-source command line for production collection: describe sources, filters, and storage in YAML — the CLI collects in batch, on schedule, straight into your database.

[Start for free](https://app.anysite.io/register?lang=en) MIT-licensed, open source — install with pip install anysite-cli.

900+ sources behind one key

3 databases — SQLite, PostgreSQL, ClickHouse

~1K tokens in context, at any batch size

MIT license, open on GitHub

## Why Teams Pick It

### Declarative YAML Pipelines

Chain sources with dependencies, set filters, storage, and schedule in one file. Six templates cover common patterns; incremental runs track a cursor so nothing repeats.

### Batch at Production Scale

Parallel execution, per-source error strategies, and resume after interruption — batches of 10,000+ records run unattended and pick up where they stopped.

### Your Database, Not Ours

Results load straight into SQLite, PostgreSQL, or ClickHouse — or land as Parquet files — then answer to plain SQL through DuckDB.

### LLM Enrichment Built In

Classify, summarize, enrich, and deduplicate records as a pipeline step — sentiment on mentions, categories on posts — without exporting to another tool.

### Agent-Ready by Design

Your agent discovers endpoints, writes the YAML, prices the run with --dry-run, and executes — clean JSON responses and exit codes it can act on.

### Only Config Touches Context

Collection runs locally; records never pass through the model. Workflow tools that push every record through context burn ~500K tokens per 1,000 records; the CLI holds at ~1K — an efficiency gain of 500×, rising to 50,000× at 100,000 records.

## How It Works

01 · Install

### Install the CLI

One package, MIT-licensed. Set your API key and pull the endpoint schema — the whole setup is three commands.

pip install anysite-cli

02 · Describe

### Describe the Pipeline in YAML

Name your sources, chain them with depends_on, set filters and storage. Or brief your agent in plain language and let it write the file.

anysite dataset init prospect-pipeline

03 · Run

### Run It — or Put It on a Schedule

Preview cost with a dry run, execute, then hand it to cron. Incremental mode collects only what changed since the last run.

anysite dataset collect pipeline.yaml --dry-run

04 · Query

### Data Lands in Your Database

Records arrive in SQLite, PostgreSQL, or ClickHouse, ready for SQL — and for LLM steps like classify and summarize when you want a read, not rows.

anysite llm classify pipeline.yaml --source posts

## The Pipeline Is One YAML File

One Pipeline, Three Chained Sources

```
name: prospect-pipeline
sources:
  target_companies:
    endpoint: /api/companies/search
    input:
      industry: "SaaS"
      employee_count: "51-200"
    parallel: 3
  decision_makers:
    endpoint: /api/company/employees
    depends_on: target_companies
    input:
      company: ${target_companies.urn}
      keywords: "VP Sales, Director Sales"
    on_error: skip
  recent_posts:
    endpoint: /api/people/posts
    depends_on: decision_makers
storage:
  format: parquet
  path: ./data/prospects
```

The live example, trimmed for length — endpoint paths shortened to generic form.

From Dry Run to LLM Classify

```
# Preview costs before running
anysite dataset collect pipeline.yaml --dry-run

# Execute the full pipeline
anysite dataset collect pipeline.yaml

# Run incremental updates
anysite dataset collect pipeline.yaml --incremental

# Query results with SQL
anysite dataset query pipeline.yaml \
  --sql "SELECT * FROM decision_makers WHERE title LIKE '%CTO%'"

# Classify posts with LLM
anysite llm classify pipeline.yaml --source recent_posts \
  --categories "product_update,hiring,thought_leadership"
```

One file, five commands — preview the cost, run, refresh, query, classify.

Zero to First Pipeline in Four Commands

```
# Configure your API key
anysite config set api_key YOUR_API_KEY

# Update the schema
anysite schema update

# Make your first request
anysite api /api/people/profile user=satyanadella

# Create your first pipeline
anysite dataset init my-first-pipeline
```

Step zero is pip install anysite-cli. Endpoint path shortened to generic form.

## What You Ask. What You Get.

Competitor Watch, on Cron

“Every morning, collect our three competitors’ new posts on Twitter/X and Reddit, classify by topic, and flag spikes.”

What happens: A scheduled pipeline collects each source in parallel, dataset diff picks out what changed since yesterday, and an LLM step labels every post by topic and sentiment.

You get: A daily table in PostgreSQL — new posts, topic, sentiment, engagement — plus a webhook ping when the run completes.

Twitter/X Reddit YouTube

Prospect Dataset, Refreshed Weekly

“Build a list of seed-stage companies from Y Combinator, pull the founders, and keep it fresh every Monday.”

What happens: The pipeline chains company search into founder lookups with depends_on, runs incrementally, and skips records it already holds.

You get: A deduplicated founders table in SQLite with company, batch, role, and profile fields — current as of this morning.

Y Combinator Business data DuckDuckGo

## Common Questions

When do I use the CLI instead of MCP?

They run on the same engine. MCP is conversational — explore sources, prototype a workflow, ask one-off questions from Claude or Cursor. The CLI is operational — the same workflow as a YAML pipeline, batch-sized, scheduled, and writing to your database. Most teams explore with MCP, then execute with the CLI.

Which databases can it write to?

SQLite for local work, PostgreSQL for shared production tables, ClickHouse for analytics at volume — plus Parquet files if you’d rather keep plain files. Every destination stays queryable with SQL through DuckDB.

How does scheduling work?

Cron scheduling is built in: set a schedule on the pipeline and it runs unattended, with incremental mode so each run collects only what’s new. A webhook fires on completion, so downstream steps can pick up the fresh table.

What do big runs cost in tokens?

About ~1K — the same as small ones. Only the pipeline config enters the context window; collection, storage, and SQL run locally. A workflow tool that passes every record through the model spends ~500K tokens on 1,000 records and ~50M on 100,000; the CLI stays at ~1K either way.

What does the CLI cost?

The CLI ships with the API line — usage-based pricing where you pay for data returned, not for attempts. Exploring from an agent first? MCP plans start at $30/month flat, with a 7-day free trial that comes with 1,000 credits. The CLI itself is open source under the MIT license.

## The Web Is the Database. The Agent Is the Data Engineer.

Open source, MIT-licensed, one pip install away. Describe the pipeline tonight; query the table tomorrow.

[Start for free](https://app.anysite.io/register?lang=en)