{"product_id":"ingest-paginated-api-data-into-postgres-in-n8n-with-retries","title":"Ingest Paginated API Data into Postgres in n8n with Retries","description":"\u003ch3\u003eIngest any paginated API into PostgreSQL in n8n—automatically, with retries\u003c\/h3\u003e\n\u003cp\u003eThis n8n workflow pulls data from \u003cstrong\u003eany paginated HTTP API\u003c\/strong\u003e, normalizes each record, and \u003cstrong\u003eupserts\u003c\/strong\u003e it into your \u003cstrong\u003ePostgreSQL\u003c\/strong\u003e table—looping through pages with \u003cstrong\u003ebuilt-in retries\u003c\/strong\u003e and an optional wait between requests.\u003c\/p\u003e\n\n\u003ch3\u003eWhat this workflow does\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cstrong\u003eRuns manually\u003c\/strong\u003e (start the workflow whenever you need a fresh ingest).\u003c\/li\u003e\n  \u003cli\u003eBuilds an API request configuration in a \u003cstrong\u003ecode\u003c\/strong\u003e step, including \u003cem\u003eURL, headers\/auth\u003c\/em\u003e, pagination strategy, limits, and retry settings.\u003c\/li\u003e\n  \u003cli\u003eCalls the API using an \u003cstrong\u003eHTTP Request\u003c\/strong\u003e (via the workflow’s configured call), and \u003cstrong\u003eretries on failures\u003c\/strong\u003e using your max-retries and retry-wait settings.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eParses the response\u003c\/strong\u003e, extracts records using your configured \u003cem\u003eJSON dataPath\u003c\/em\u003e, maps fields into the target database schema, and computes the \u003cem\u003enext page\/cursor\/link\u003c\/em\u003e.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eUpserts each batch\u003c\/strong\u003e into PostgreSQL using an \u003cstrong\u003eINSERT ... ON CONFLICT\u003c\/strong\u003e strategy (only when the page contains records).\u003c\/li\u003e\n  \u003cli\u003eChecks if more pages remain (including a \u003cstrong\u003emax-pages safety limit\u003c\/strong\u003e), optionally waits to respect rate limits, and continues until pagination ends.\u003c\/li\u003e\n  \u003cli\u003eOutputs a final summary including \u003cstrong\u003epages processed\u003c\/strong\u003e and \u003cstrong\u003etotal records ingested\u003c\/strong\u003e.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eUse cases\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eDaily\/adhoc ingestion from a SaaS API that returns results in pages (cursor or link-based).\u003c\/li\u003e\n  \u003cli\u003eBackfilling historical data into PostgreSQL while staying resilient to transient HTTP errors.\u003c\/li\u003e\n  \u003cli\u003eKeeping a Postgres table synchronized by upserting on a configured \u003cstrong\u003eunique key\u003c\/strong\u003e.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eTechnical details\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cstrong\u003eNodes\/steps:\u003c\/strong\u003e if, code, no op, wait, postgres, sticky note\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003ePostgreSQL:\u003c\/strong\u003e upsert via \u003ccode\u003eINSERT ... ON CONFLICT\u003c\/code\u003e\n\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eConfigurable settings:\u003c\/strong\u003e base URL, auth header\/token, pagination type\/parameters, response dataPath\/cursorPath, limit, maxPages, maxRetries, and retryWaitM\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003cp\u003eSet your API and table mappings once, then reuse this n8n workflow for reliable paginated API ingestion into PostgreSQL.\u003c\/p\u003e","brand":"N8N Commerce","offers":[{"title":"Default Title","offer_id":46172995125427,"sku":"N8N-20550","price":77.99,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0749\/6279\/6723\/files\/1MHcLiAIBAsv5bUSgkXKI_mV8JscmQ.png?v=1791365261","url":"https:\/\/buyflowscripts.com\/products\/ingest-paginated-api-data-into-postgres-in-n8n-with-retries","provider":"N8N Commerce","version":"1.0","type":"link"}