API Last Modified Date - and sync performance
Hi,
I'd like to better synchronise your data into our data lake.
Currently we need to do a full refresh and load your entire data every day via your API. It takes us 1 hour 40 minutes each time, daily. I'd much rather do an incremental sync.
This will really benefit Cin7 as well as us.
Refer to: https://docs.airbyte.com/platform/connector-development/connector-builder-ui/incremental-sync
The Issue: Looking at your API documentation (https://dearinventory.docs.apiary.io/) and the existing Airbyte connector, it appears that none of your Cin7 Core API endpoints currently support incremental data filtering via modifiedSince or updatedSince parameters. This means data integration tools must perform full refreshes of entire datasets rather than efficiently syncing only changed records.
What's needed: To enable incremental synchronization, your API endpoints would need:
- A
modifiedSinceparameter that filters records modified after a specified timestamp - A consistent
lastUpdatedormodifiedDatefield in all record responses - ISO 8601 timestamp formatting for reliable cursor tracking
Significant benefits for both of us:
For your infrastructure:
- Dramatically reduced API load - instead of transferring entire datasets daily, we'd only request changed records
- Lower bandwidth costs and server resource usage
- Improved API performance for all users as fewer full dataset requests are made
- Better scalability as your customer base grows
- Reduced risk of rate limiting issues during peak usage periods
For data integration:
- Much faster sync times - minutes instead of hours for large datasets
- More frequent syncs become feasible without overwhelming your servers
- Near real-time data availability in downstream systems
- Lower infrastructure costs for data processing
- Improved data freshness for business intelligence and reporting
Business impact:
- Better customer experience through reduced API response times
- Enhanced reliability of data integrations
- Ability to support more customers and larger data volumes
- Competitive advantage as many modern APIs offer incremental sync capabilities
This enhancement would align Cin7 Core with modern API best practices and significantly improve the integration experience for your customers.
Would it be possible to add modifiedSince filtering parameters to your main endpoints? I'd be happy to discuss the technical requirements in more detail.
Thanks for considering this important enhancement!
Best regards,
Related to
Comments
1 comment
Following. We also perform daily (nightly rather) sync's of all product data. For PO and SO data, we have a webhook which informs us of changes and then we read the relevant updated records.
Please sign in to leave a comment.