Data integration blog: guides on sync, ETL and APIs
Practical writing on data integration from the team building production-grade adapters:
ETL vs ELT, webhooks vs polling, data mapping rules, and what point-to-point scripts
really cost. No filler, plenty of field names.
Stripe allows 100 requests a second in live mode and 25 in a sandbox. Those are the numbers everyone quotes. The reason Stripe integrations break is that six genuinely different problems all return HTTP 429, and their fixes contradict each other. The current limits, how to tell the six apart in one header, and seven changes that fix them.
QuickBooks throttles at 500 requests a minute per company file and allows ten concurrent connections to it. Almost every 429 we get asked about is the second limit, not the first, which is why adding workers makes a slow QuickBooks sync slower. The current numbers, the five failures that all look like one problem, and the six changes that fix them.
Shopify does not count your requests, it prices them. The GraphQL Admin API meters calculated query cost in points per second and your plan sets the rate, which is why the same connector behaves like a different product on Standard and on Plus. The current numbers, and the six changes that fix a throttled sync.
Almost every stalled HubSpot integration is stalled for one of two reasons, and neither is the daily call limit everybody worries about. The current numbers by tier, what a 429 actually costs you, and the six changes that fix a rate-limited sync.
Enterprise, Unlimited and Performance orgs get five of these at no extra cost, and most of them sit unclaimed while an integration runs as somebody's admin account. What the license permits, what it costs, where it gets in your way, and the setup order that trips up almost every first attempt.
Zero-ETL is described in two wrong ways: as marketing for a faster connector, or as a universal replacement for ETL. It is neither. Here is exactly what AWS replicates for you, what you give up in exchange, and the failures that show up quietly in week one.
The commands are three lines long and teams still end up throttled by a quota nobody read, money columns that lost their cents, and a pipeline that stopped a month ago. Every realistic path into BigQuery, with the numbers from Google's own documentation.
The commands are short and almost every team still ends up with slow loads, money columns that lost their cents, and a pipeline that stopped three weeks ago. Every realistic path into Snowflake, with the numbers from Snowflake's own documentation.
Two completely different things get called connecting Snowflake to Salesforce, and picking the wrong one costs a month. Here is the difference between a write that lands on records and zero copy federation that does not, plus the API limits and type mismatches that break the first sync.
PostgreSQL ships real logical replication, and it is the right answer whenever both ends are Postgres. It also silently refuses to replicate your schema, your sequences and your large objects. Here is the setup, the documented restrictions, and the slot that fills your disk.
PostgreSQL gives you real log-based CDC through logical decoding, and it gives you a replication slot that will retain write-ahead log files until your disk is full. Here is how the pieces fit, what to set on RDS, and when a watermark query is the better answer.
Stripe charges into QuickBooks with the processor fee on its own account. A Shopify order that decrements NetSuite inventory before the warehouse oversells it. Twelve integrations teams actually run, and the three disagreements that make each one hard.
PostgreSQL folds unquoted identifiers to lowercase while SQL Server keeps the case you typed, and the MONEY type on both sides means different things. Here is the phased migration that lets you prove the totals match before you move the application.
MONEY columns that land as a float lose cents nobody notices until the quarter is closed, and SQL Server change data capture holds the transaction log open when nothing is consuming it. Here is the incremental migration that survives both.
BigQuery bills by bytes scanned, so an unpartitioned copy of a large MySQL table turns a cheap dashboard into a recurring line item. Here is the incremental load that keeps analysts off production and keeps the query bill flat.
A big-bang UNLOAD and COPY means a frozen reporting window nobody signs off on, and Redshift folds identifiers to lowercase while Snowflake folds them to uppercase. Here is the parallel-run migration that lets you validate totals before you cut over.
Analysts querying the production MySQL replica slow down the app at month end, and hand-rolled load scripts double revenue after a retry. Here is how to replicate MySQL into Snowflake incrementally with type mapping that survives contact with real money columns.
A mysqldump is stale the second it finishes, and MySQL types do not land cleanly in Postgres: TINYINT(1) is a boolean in practice, unsigned integers overflow, and 0000-00-00 has no equivalent. Here is the migration that runs both databases in parallel until the numbers agree.
The Square API pages by cursor and returns money as integer cents in a nested object, and its reports cannot join to your inventory or accounting tables. Here is how to replicate payments into Postgres incrementally so reporting queries transaction data with plain SQL.
The PayPal Transaction Search API only returns a bounded date window per call and splits gross, fee, and net across objects, so a naive pull drops rows and breaks reconciliation. Here is how to replicate transactions into Postgres incrementally and tie out to the payout.
Querying a Snowflake table on every page load is slow and expensive, so teams cache it by hand and the copy drifts. Here is how to push modeled data into Postgres incrementally so your app reads it with a fast local SQL query.
Qlik retired Talend Open Studio and moved Talend onto capacity-based cloud tiers with no public list price. Here is how the capacity model works, what contracts actually run, and what pushes the number up.
A big-bang export and reload freezes reporting and hides type bugs until someone spots a wrong number. Here is how to migrate BigQuery to Snowflake incrementally, handle nested fields, validate both warehouses in parallel, and cut over safely.
Calling the QuickBooks Online API live is slow and throttled, and its reports cannot join to your product or CRM data. Here is how to replicate the ledger into Postgres incrementally so finance dashboards query it with plain SQL.
The Xero API caps you at 60 calls a minute, and its reports cannot join to your product or billing tables. Here is how to replicate the ledger into Postgres incrementally so reporting and app features query it with plain SQL.
A big-bang unload and reload freezes reporting and hides type bugs until someone spots a wrong number. Here is how to migrate Snowflake to BigQuery incrementally, validate both warehouses in parallel, and cut over safely.
Calling the Shopify Admin API live is slow and rate limited. Here is how to replicate orders, line items, customers, and refunds into Postgres incrementally so your dashboards and app features query store data with plain SQL.
Matillion sells the Data Productivity Cloud on a credit-based consumption model with no public list price. Here is how credits work, what the Developer, Teams, and Scale editions include, what a deployment runs, and what pushes the bill up.
Xero reports are per organization and cannot join to product or ad data. Here is how to load Xero into BigQuery incrementally so revenue, cash, and margin models read one clean ledger across every entity.
PayPal Activity exports cannot join to your orders or ad spend. Here is how to load PayPal into BigQuery incrementally so revenue, fee, and net-settlement models read one clean table you can trust.
SnapLogic sells through sales and scales price with data volume, environments, and connectors. Here is how a typical annual contract stacks up, what a small deal versus an enterprise deal runs, and what inflates the quote.
Square reports are per location and cannot join to other channels. Here is how to load Square into Snowflake incrementally so sales, fee, and margin models read one clean table across every location.
Querying NetSuite live through SuiteTalk hits governance limits. Here is how to replicate NetSuite into Postgres incrementally so your apps, tools, and reports read live ERP data without throttling the API.
Informatica sells IDMC through sales on an IPU consumption model with no public list price. Here is how the meter works, what an entry versus enterprise deployment runs, what PowerCenter adds, and what inflates the bill.
Querying Salesforce live burns API calls against a daily cap. Here is how to replicate Salesforce into Postgres incrementally so your apps, tools, and reports read live CRM data with plain SQL.
HubSpot reports cannot join to product usage or ad spend. Here is how to load HubSpot into BigQuery incrementally so pipeline, attribution, and funnel models read one clean, partitioned history.
Jitterbit sells Harmony through sales and prices it mainly by connection count. Here is how the tiers stack up, what a typical annual contract runs, and what inflates the quote.
Stripe dashboards show today, not history you can model. Here is how to load Stripe into BigQuery incrementally so revenue, fee, and MRR models read one clean table.
Shopify reports cannot join to ad spend or fees, and CSV exports go stale. Here is how to load Shopify into BigQuery incrementally so cohort, LTV, and margin models run on a full order history.
Tray.ai sells through sales and bills by task, where each workflow step counts. Here is how the plans stack up, what a task really is, and what inflates the bill.
Monthly CSV exports give you a snapshot, not a live model. Here is how to load QuickBooks Online into Snowflake incrementally so finance models read today's numbers.
HubSpot reports cannot join to product or billing data, and list exports go stale. Here is how to load HubSpot into Snowflake incrementally so attribution and pipeline models run on a full history.
Airbyte Cloud bills by data volume and the community edition is free to self-host. Here is how the credit model works, what pushes the bill up, and what running it yourself really costs.
Monthly CSV exports give you a snapshot, not a live model. Here is how to load QuickBooks Online into BigQuery incrementally so finance dashboards read today's numbers.
The Stripe API paginates 100 objects at a time and rate-limits, so backfilling by hand is brittle. Here is how to load charges, fees, and payouts into Snowflake so revenue models tie out.
n8n cloud bills by workflow execution and the community edition is free to download. Here is what each tier actually covers, what pushes the meter up, and what self-hosting really costs to run.
Nightly full dumps rewrite whole tables and inflate BigQuery cost. Here is how to load Postgres incrementally, map the types correctly, and handle updates, deletes, and schema drift.
Saved-search exports are manual and stale, and SuiteAnalytics Connect carries its own license. Here is how to load NetSuite transactions into Snowflake incrementally so the numbers tie out.
Celigo prices by edition, endpoints, and the number of integration flows, through a sales quote. Here is how the tiers work, what pushes the bill up, and where a flat price wins.
Full table pulls burn Salesforce API calls and land stale data in the warehouse. Here is how to load accounts, opportunities, and activity into Snowflake incrementally and keep it fresh.
Square pays you net of fees, so the deposit never equals gross sales. Here is how to post Square sales, refunds, and fees into NetSuite so each payout reconciles on the first try.
A CSV dump lands accounts in the wrong type and duplicates your contacts. Here is how to move customers, the chart of accounts, and open balances from QuickBooks to Xero cleanly.
Sales closes a deal in HubSpot and finance rekeys it into QuickBooks. Here is how to sync contacts, turn closed-won deals into invoices, and push payment status back to the deal.
Boomi prices mainly by connection through a sales quote, and each endpoint-to-endpoint link counts. Here is how the tiers work, what drives the bill, and where a flat price wins.
PayPal takes its fee before it pays you, so the deposit never equals the sale. Here is how to post sales, fees, and refunds so QuickBooks reconciles on the first try.
Fivetran charges by Monthly Active Rows, so the bill tracks change, not size. Here is how MAR is counted, what the 2026 plans cost, and why a busy month gets expensive.
Shopify pays out net of fees, so the deposit never equals the order total. Here is how to post orders, fees, refunds, and sales tax so QuickBooks reconciles on the first try.
Self-serve platforms run tens to hundreds of dollars a month, enterprise suites run six figures a year, and in-house looks free until you price the maintenance. The arithmetic, in full.
The build is the cheap part. Auth, pagination, retries, backfills, and every upstream API version bump are the bill you pay forever. A framework for deciding which side you are on.
iPaaS is a hosted platform that moves data between your apps so you do not write and babysit point-to-point scripts. Here is how it works and when it pays off.
An API adapter sits between two systems and translates one data shape into another. It is the piece of glue code every team rewrites, and the piece we productized.
Most sync failures are mapping failures. Nine rules, from picking a stable match key to versioning every change, that keep records landing where they should.
Every pair of connected systems adds another script someone has to own. At ten systems you are maintaining dozens of fragile links. The math gets ugly fast.
7 min read
Done reading about glue code? Retire it
Connect your first pair and watch records land where they should. Flat price from $49 a month.