Ingestion Pipeline
Data flows from provider APIs via CLI connectors into PostgreSQL.
Run make ingest-hetzner, make ingest-azure, etc. Each connector fetches locations, instance types, and prices; the ingestion service upserts into PostgreSQL and normalizes to EUR via FX rates. Idempotent — safe to re-run anytime.
To double-check ingested prices, run make verify-gcp (GCP), make verify-hetzner (Hetzner), or make verify-all. See Pricing data sources for the APIs and docs used per provider.
Pricing data sources
Instance pricing is ingested from the following APIs or config:
| Provider | API / Docs | Auth |
|---|---|---|
| GCP | Cloud Billing Catalog API | ADC |
| AWS | Price List API, EC2 API | AWS credentials |
| Azure | Retail Prices API | None (public) |
| Hetzner | Hetzner Cloud API | API token |
| DigitalOcean | DigitalOcean API | API token |
| Scaleway | Instance API | API keys |
| OVH | OVH API / Public Catalog | OVH keys / none |
| UpCloud | UpCloud API | Username + password |
| Open Telekom Cloud | OTC Price API | None (public) |
| Seeweb | Seeweb ECS API | API token |
| Aruba, Exoscale, IONOS, gridscale, STACKIT, Elastx, Cyso Cloud | Config (pricing pages) | None |
graph LR
CLI["make ingest-* (CLI)"]
CON["Connectors (GCP, AWS, Hetzner, Scaleway, …)"]
SVC["Ingestion Service (upsert + FX)"]
DB[("PostgreSQL")]
CLI --> CON --> SVC --> DB