Ingest Paginated API Data into Postgres in n8n with Retries

Ingest any paginated API into PostgreSQL in n8n—automatically, with retries

This n8n workflow pulls data from any paginated HTTP API, normalizes each record, and upserts it into your PostgreSQL table—looping through pages with built-in retries and an optional wait between requests.

What this workflow does

• Runs manually (start the workflow whenever you need a fresh ingest).
• Builds an API request configuration in a code step, including URL, headers/auth, pagination strategy, limits, and retry settings.
• Calls the API using an HTTP Request (via the workflow’s configured call), and retries on failures using your max-retries and retry-wait settings.
• Parses the response, extracts records using your configured JSON dataPath, maps fields into the target database schema, and computes the next page/cursor/link.
• Upserts each batch into PostgreSQL using an INSERT ... ON CONFLICT strategy (only when the page contains records).
• Checks if more pages remain (including a max-pages safety limit), optionally waits to respect rate limits, and continues until pagination ends.
• Outputs a final summary including pages processed and total records ingested.

Use cases

• Daily/adhoc ingestion from a SaaS API that returns results in pages (cursor or link-based).
• Backfilling historical data into PostgreSQL while staying resilient to transient HTTP errors.
• Keeping a Postgres table synchronized by upserting on a configured unique key.

Technical details

• Nodes/steps: if, code, no op, wait, postgres, sticky note
• PostgreSQL: upsert via INSERT ... ON CONFLICT
• Configurable settings: base URL, auth header/token, pagination type/parameters, response dataPath/cursorPath, limit, maxPages, maxRetries, and retryWaitM

Set your API and table mappings once, then reuse this n8n workflow for reliable paginated API ingestion into PostgreSQL.