Postgres to BigQuery loads with schema mapping and incremental sync
The Postgres to BigQuery pipeline from Adapters loads tables incrementally into your warehouse with schema mapping, type casting, and automatic retries, at a flat monthly price instead of per-row usage billing. It reads changed rows on a watermark so each run stays cheap and no code is required.
No credit card required.
Field mapping auto-plugged · tap a port to rewire
Plug a source port into
Transform on this cable
JSON in
JSON out
5 sample records ready
Last updated September 2026
What running Postgres to BigQuery by hand costs you
- A nightly pg_dump and load rewrites whole tables and balloons BigQuery storage and query cost.
- Postgres and BigQuery types do not line up, so numeric, jsonb, and timestamp columns land wrong.
- Home-grown loaders have no retries, so one network blip leaves the warehouse half-loaded and silent.
The field mapping, out of the box
These cables are pre-wired when you pick the pair. Rewire any of them, or add your own, in the same visual data mapping tool you use for every adapter.
Input / POSTGRES
Output / BIGQUERY
Transforms included
Incremental loads read rows past an updated_at watermark so each run only moves what changed; Postgres types cast to BigQuery types (numeric to NUMERIC, jsonb to JSON, timestamptz to TIMESTAMP in UTC), cents convert to decimal amounts, and tables partition and cluster so queries scan less and cost less.
Postgres to BigQuery: the quota that breaks naive pipelines, and the types that corrupt data quietly
This lane fails in two predictable ways: a loader that fires too many load jobs and gets throttled, and a schema that loads without error but is wrong. Below: the BigQuery quotas that decide the design, what Datastream needs from PostgreSQL, the type conversions worth setting by hand, and when change data capture earns its configuration cost. Google Cloud documentation read 15 August 2026.
The 1,500 load jobs per table per day ceiling
This is the number that decides your architecture, and almost nobody reads it before writing the loader. BigQuery allows 1,500 load jobs per table per day and 100,000 per project per day, and failed jobs count toward both. A pipeline that submits one load job per file, or worse per batch of rows, exhausts a table's daily allowance in an afternoon on an active source. There are three ways out: batch aggressively so each job carries many files, use the Storage Write API which streams rows and does not touch the load job quota at all, or run continuous replication with Datastream. Google's own published guidance is explicit that if you regularly exceed the load job limits because of frequent updates, you should be streaming instead of batching.
The other load limits worth knowing before the first backfill
A single load job caps at 15 TB across all its input files and fails after 6 hours of execution, which is exactly what a full history dump of a large table hits. Split the initial backfill by date range into several jobs rather than submitting one. A job can reference up to 10,000 source URIs and read up to 10 million files. On the row side, CSV and newline-delimited JSON cap at 100 MB per row and per cell, so a wide jsonb document can exceed it. Compressed CSV files cap at 4 GB while uncompressed go to 5 TB, which means gzipping a very large export can be the thing that breaks a load that would have worked uncompressed.
What Datastream needs from PostgreSQL
If you want change data capture rather than a watermark query, Datastream is Google's managed option and it reads the write-ahead log through logical replication. That means wal_level = logical, available since PostgreSQL 11, plus a replication slot and a publication covering the tables you want. On Amazon RDS, Aurora or Cloud SQL these are parameter group settings and changing wal_level requires a restart, so plan it into a maintenance window. The trap that takes production down is the replication slot itself: PostgreSQL retains WAL from the slot position until the consumer confirms it has read past it, so a paused or broken stream over a weekend grows WAL until the disk fills. Monitor slot lag as a first-class alert, and drop the slot manually when you remove a stream, because removing the stream does not remove the slot.
The type conversions that load cleanly and are wrong
Schema autodetect is the main culprit here, because it infers types from a sample of the file. A Postgres numeric money column whose first few hundred rows happen to be whole numbers is detected as INTEGER, and every cent after that is gone with no error raised. Declare NUMERIC or BIGNUMERIC explicitly. Timestamps are the second trap: timestamptz should map to BigQuery TIMESTAMP, which is an absolute point in time, and never to DATETIME, which stores no zone and will shift every value by your offset. Postgres jsonb maps to the native JSON type. Arrays map to REPEATED fields. And identifier rules differ: Postgres folds unquoted names to lowercase, while BigQuery field names allow only letters, numbers and underscores and cannot begin with a digit, so a column named 2024_total needs renaming in the load schema.
Partition and cluster on day one, because BigQuery bills by bytes scanned
This is a cost decision disguised as a schema decision. An unpartitioned table means every dashboard query scans the entire history, and BigQuery charges by bytes scanned. Partitioning on the ingestion date or on an event timestamp cuts that to the range each query actually needs, and clustering on the columns you filter by, such as customer or account ID, narrows it further. Both are trivial to declare when the table is created and genuinely tedious to retrofit, because you have to create a replacement table and repoint every scheduled query, dashboard and downstream model that reads the old one. Set it before the first load, not after the first surprising bill.
Watermark or CDC, and how Adapters fits
A watermark query filtering on updated_at needs no database configuration and misses two things: hard deletes, and rows changed by any process that does not touch the timestamp. CDC catches both and costs you logical replication setup plus slot monitoring. The honest rule: use a watermark when the table is append-mostly and a missed delete does not change a reported number, use CDC when deletes matter or rows are updated in place by application logic, and just reload small dimension tables in full. Adapters runs scheduled incremental syncs with visual field mapping, explicit type casting you can see before the first run, batched loads that stay well inside the job quota, retries, alerting on the run that did not happen, and per-record error logs, on a flat monthly price from $49. It is not the right pick if you need sub-second freshness, where Datastream or the Storage Write API is the correct answer, or if the wider project needs dozens of additional SaaS sources.
How it goes live
Three steps, minutes end to end, covered by flat data integration pricing from $49 a month.
STEP 01
Pick the pair
Connect Postgres and BigQuery with scoped credentials. About a minute each.
STEP 02
Confirm the mapping
The cables above are pre-wired. Adjust any field, preview the transform on sample records, done.
STEP 03
Schedule the sync
Hourly down to every minute, with retries, alerting, and a full log on every run.
Prefer to understand the moving parts first? Our long-form guide to loading Postgres into BigQuery incrementally covers the field-by-field detail, the failure cases, and what changes at volume.
Postgres to BigQuery sync: common questions
How do I load Postgres data into BigQuery?
Read rows past an indexed updated_at watermark, stage them, then MERGE into the target table on the primary key. Full reloads are simple but expensive once tables get large, and they scale badly. Incremental loads keep both the runtime and the query bill flat.
Can BigQuery query a Postgres database directly?
Through federated queries with EXTERNAL_QUERY, and only for Cloud SQL instances, not any PostgreSQL server you happen to run. For a self-managed or third-party hosted Postgres, something has to extract and load the rows, which is what a scheduled connector does.
How do PostgreSQL data types map to BigQuery?
Integers become INT64, numeric keeps its precision as NUMERIC, text becomes STRING, boolean stays BOOL, jsonb becomes JSON, and timestamptz becomes TIMESTAMP normalized to UTC. Money columns belong in NUMERIC rather than FLOAT64, since FLOAT64 cannot represent cents exactly.
How do I keep BigQuery costs down when replicating Postgres?
BigQuery bills by bytes scanned, so partition large tables on the date column analysts filter by and cluster on the columns they group by. An unpartitioned copy of a big table turns a cheap daily dashboard into a recurring line item nobody budgeted for.
How does the Postgres to BigQuery sync work?
The Postgres to BigQuery pipeline from Adapters loads tables incrementally into your warehouse with schema mapping, type casting, and automatic retries, at a flat monthly price instead of per-row usage billing. It reads changed rows on a watermark so each run stays cheap and no code is required.
Is there a prebuilt Postgres connector for BigQuery?
Yes. This Postgres to BigQuery connector ships prebuilt: the field mapping is wired the moment you pick the pair, transforms are included, and you can try it against sample records in the live demo. No code or engineering sprint required.
How much does the Postgres BigQuery integration cost?
Pricing is flat and monthly: Starter at $49, Growth at $149, Scale at $399. Every plan includes this pair, visual field mapping, and per-record logs. There are no per-task or per-row fees, so the bill stays the same as volume grows.
How often can Adapters sync Postgres to BigQuery?
Hourly on Starter, every 5 minutes on Growth, and down to every minute on Scale. Failed records retry automatically with backoff, and alerting plus a full per-record log come standard on every run.
Do I need to write code to connect Postgres and BigQuery?
No. Fields are auto-mapped the moment you pick the pair, and you can rewire any mapping visually before the first sync. Incremental loads read rows past an updated_at watermark so each run only moves what changed; Postgres types cast to BigQuery types (numeric to NUMERIC, jsonb to JSON, timestamptz to TIMESTAMP in UTC), cents convert to decimal amounts, and tables partition and cluster so queries scan less and cost less.
More pairs from the API connector library
Browse the full api connector library, or request a pair you do not see.
Postgres and BigQuery, finally in agreement
Map the pair once and let it sync on schedule. Flat price from $49 a month, no per-task fees.
No credit card required.