Several years ago, we built a FileMaker system for an advertising firm that needed to pull in data from several external sources: phone calls, QR scans, sales data, and so on.
They were able to obtain spreadsheet exports from each source. We then built the tools to import and process those spreadsheets. That was cost-effective and worked well for the first few years.
But as they grew, that process was increasingly unsustainable. The amount of data grew, both in the exported sheets and in the tables, over time. The run times of these imports got longer and longer. And the employees doing the imports had increasing amounts of other work occupying their time. Yet, their need grew for more frequent updates to the data to get the most up-to-the-minute insights.
We needed a way to automate (to help the users) and streamline the processing (to help the server).
Soliant Sidecars
Enter Soliant’s Sidecar product. Sidecars can be created on Soliant.Cloud servers and configured to listen for webhooks sent by external platforms. They then accept the data payload sent by the webhook and trigger FileMaker scripts to process that data payload.
The external data platform had developed an API and offered the ability to create webhooks triggered by changes to their data.
As an example, consider using a phone call service provider:
- We set up a webhook at the call provider to send call data as it was generated.
- That webhook called our Sidecar, sending over the call data.
- The Sidecar passed that data into our existing call processing script.
With this method, call data was updated automatically every five minutes. Nobody had to do any imports or wait around for processing. The server load was reduced, and processing spikes smoothed out.
Before Soliant.Cloud Sidecars: Manual Imports
Spreadsheets contained thousands of rows and took 15+ minutes to import and process, consuming server resources and slowing the application for all users.
Due to possible changes to data (adding notes, marking invalid), overlapping batches of calls were imported over and over to make sure nothing was missed. Users needed to import 2-3 times per day, or more if needed.
Once a month, that entire month’s worth of calls had to be re-imported, taking 30+ minutes to process. Users had to wait for completion before proceeding with a busy day of invoicing work
Automating This Process Using a Sidecar
A call arrives or is edited on the call provider platform. A webhook sends a notification to the Sidecar endpoint with call data.
Sidecar features Amazon Web Services’s SQS (queuing service) to ensure that all notifications are delivered, even if the FileMaker server is down for maintenance.
A scheduled script on the FileMaker server then processes webhooks received in batches every 5 minutes.
These smaller batches more frequently mean no visible load on the server.
Changes to a call are also sent via webhook. There’s no need to import the same data repeatedly, reducing overall server load.
The client loves the new process. The newest data is flowing in throughout the day, with no brain space allocated to remembering to do imports. There’s no drag on the server, freeing it up to handle all their other work more responsively.
An Inexpensive and Simple Long-term Solution
These gains were not expensive. Sidecar incurs a one-time orchestration fee of $900 and carries a monthly subscription of $10 per month for up to 1 Billion API requests. With that and a minor investment in tweaking existing logic to handle inbound webhooks rather than imports, this client was up and running with fully automated call imports, reaping benefits to both their employees and their server. 
If you’re interested in this Sidecar, you can learn more about it here. If you’re looking to streamline your data and applications in a way that doesn’t fit with this Sidecar, our AWS architects may be able to help with a custom solution. Contact us here to schedule a call with our team.
