Platform & Integration/2020
Node Loader
Chunked Salesforce to PostgreSQL migration with scheduled batches
- Role
- Back-End Developer
- Year
- 2020
- Discipline
- Platform & Integration
- Stack
- 6 technologies
A Node and Express service that migrates data out of Salesforce into the client's PostgreSQL database over GraphQL, logging what moved, with endpoints for scheduled triggers and batch runs.
01The problem
The client needed to migrate data from Salesforce into their own PostgreSQL database over GraphQL, and to have a record of what had actually moved. A single bulk pull was not viable against the volumes involved.
02The approach
After working the requirement through with the client, we built a Node and Express application exposing an endpoint for scheduling triggers or batches. It fetches updated data from Salesforce, uses JSforce to retrieve the historical records, divides them into chunks, and uploads them to the client's server — so a run is made of many small, observable steps rather than one large opaque one. Deployed to AWS EC2.
03The outcome
Migration ran in scheduled, chunked batches with logging of migrated data. This sat within the broader Salesforce data migration and syncing work, where I built 30+ REST endpoints for front-end integration.
04Decisions worth noting
- 01
Chunked transfers rather than one bulk operation, so a failure costs one chunk instead of the whole run.
- 02
Endpoint-driven scheduling for triggers and batches.
- 03
JSforce for historical retrieval out of Salesforce.
- 04
Part of a data migration and syncing platform exposing 30+ REST endpoints.