If you use the FileMaker platform, you are likely familiar with a type of field known as a container. Containers are roughly the equivalent of BLOB (Binary Large OBject) fields in other database systems. There are some technical differences between the two, but for the most part, they serve the same purpose. Aside from the technical differences, there are arguments both for and against storing this kind of data in a database. The right path is not always clear, so it is important to understand your options.
Keeping Data in Container Fields in FileMaker
Keeping data in container fields in your FileMaker database retains ease of use, functionality, and record atomicity. All of those are valid and may be further informed by the requirements for your deployment. However, there are some considerations that become especially important as your deployment scales.
FileMaker can either store container data within a FileMaker file or can be stored externally. We typically discourage internally stored container data. It can bloat file size and quickly become hard to manage.
You can also consider externally storing container data, especially for on-premises deployments. Unfortunately, your FileMaker system will still need to manage that data at times when backing up a system. For example, when your database stores binary data, the overall size of your system becomes unmanageable. You should store your object data (files) on a system dedicated to that process.
FileMaker’s ability to customize organization of internal documents drives enormous value to businesses. They can rely on the platform to manage and retrieve information that they need quickly. Imagine, for example, a sales department that uses hundreds of PDFs in its sales process. These files would be unmanageable on a file server. A database that organizes them with a searchable and easy-to-use system transforms the process. It also allows for additional customizations, security, and more.
Leveraging Cloud Storage for Your FileMaker Application
Cloud storage, on the other hand, changes how you store, manage, and access your FileMaker files. We find it simplifies file storage. It also speeds up retrieval time and ensures backup recovery for your documents. Our clients who need to access files from any location and at any time often opt for cloud integrations. They also gain easy access to data visualization and automation tools that most cloud platforms provide.
Block Storage (EBS)
Block storage is a way of storing data in small, fixed-sized chunks. Each chunk has its own special ID and can be accessed separately. Big companies often use this type of storage because of its speed. Block storage allows you to work with individual chunks of data, which is helpful for databases. It’s like breaking up a big puzzle into smaller pieces that you can move around and work with easily.
Unfortunately, block storage is expensive. Launching it is also more complicated than purchasing a hard drive and connecting it to a server. For servers hosted on AWS, volume storage is provided by a service called Elastic Block Storage (EBS). You can dynamically provision and configure it to attach to compute instances (servers) for file storage accessible to an operating system, much like traditional hard drives. While you have options in choosing what type of EBS volume to provision, the cost is billed out monthly. If you manage many files, like a FileMaker system that handles a lot of container data, it can get costly fast.
Object Storage (S3)
Amazon S3 (Simple Storage Service) is a massively scalable service for object storage in the cloud. It comes with global infrastructure, incredible durability, and an impressive feature set. You can think of object storage as an enormous database table with a couple of fields, the key or file name, and the object data itself (the file being stored.) Additionally, objects stored in S3 automatically get copied to every data center, known as an Availability Zone, in a given AWS Region.
Compared to block storage, the cost differences are stark. And with S3, you never need to worry about running out of space. Your storage grows according to your usage.
For example, a one terabyte EBS volume will cost somewhere starting around $80 per month, before tax. The same one terabyte of data stored in S3, even at the most expensive price tier, costs about $23 per month. And that is if you used the entire EBS volume. In S3, you do not need to provision additional space, as each object is stored independently. Very few FileMaker deployments even have that much container data to manage; most have much less. Without having to provision additional space, the difference in cost savings is that much greater.
Additionally, S3 has many additional features for you to enable and utilize. Object Versioning allows you to overwrite an object with the same name and automatically keep a copy of the original, giving you two versions. Subsequent overwrites create new versions, and those just stack up.
Similarly, when you delete an object, a marker is placed at the top of that stack, acting as the current version of that object. Unless individual versions are deleted permanently, you can still access these.
SideCar for Amazon S3 and FileMaker
We find S3 to be the clear winner for our FileMaker clients. Unfortunately, integrating S3 with a FileMaker file can be challenging. You can calculate the necessary signatures to integrate directly with the AWS API. But it’s difficult. And even if you calculate the correct values to use, you may struggle to support everything S3 has to offer. Some are only able to upload/download files and not list the contents of a bucket (where S3 objects are stored, per AWS account and region). Or, they can’t return object version information, etc.
We’ve simplified this process our clients in Soliant.cloud, our FileMaker Server hosting platform. We built our own microservice infrastructure to simplify integrating S3 with FileMaker solutions. The SideCar (what we call our custom-built suite of add-ons) provides an API endpoint that takes care of the heavy lifting. It returns results in an easy- to-consume format – JSON — for a FileMaker script to use. We also have a sample file available for SideCar subscribers to use as a starting point to use in their own solutions. It makes it easy to copy/paste what they need and customize from there.
Using SideCar for S3, you don’t need to rotate and manage API keys. The underlying infrastructure delivers the proper identity access roles required, with access scoped to the provisioned S3 bucket. You can easily integrate with FileMaker using an issued API Key with an endpoint.
Managed Environments
We manage the AWS environment for our Soliant.cloud clients. Each gets a dedicated AWS account as part of our overall AWS Organization. Most find the process of evaluating features to enable overwhelming. We can either manually or programmatically provision an S3 bucket with default features set. Our defaults will work for most use cases. They serve most client needs and provide a solid foundation to use in production settings. We can always adjust those settings.
Disaster Recovery Use Case
We recently helped a client who was storing images and ran into a big problem. Some scripting had inadvertently overwritten some object data across hundreds of records. Under normal circumstances, this could have been catastrophic. Even if you kept numerous backups across externally stored container data, finding the correct images and getting those inserted in the proper places manually would have required tremendous effort.
Fortunately, we have versioning enabled on our default deployments. It was relatively easy to create a script in FileMaker to loop through affected records, retrieve the previous version of an object, and upload it once again to make it the current version. SideCar provided the functionality to make that easy.
By having Object Versioning enabled, integrating with our SideCar infrastructure enabled this client to have the functionality easily recovered from what might otherwise be catastrophic data loss. It was as simple as creating a script to list the versions of an object, going to the second listed version, and getting it. Once that was done, we simply overwrote the current object with the images we had just downloaded.
Other Considerations in AWS S3
A foundational service like S3 provides solid, dependable basic functionality. Additionally, S3 has other useful features. While our standard configuration for deploying an S3 bucket with our SideCar infrastructure meets most of our client’s needs, we can easily fine-tune a deployment to add important functionality and save on costs.
Cross-Region Redundancy
For example, a client may require additional compliance for storing objects in another region. This is called Cross Region Redundancy. You can configure this to automatically make a copy of each object, including all stored versions, in an entirely different AWS region. Once configured, if you were to upload a PDF, for example, in a bucket deployed in North Virginia, that object (the PDF) would automatically get copied to a Bucket in Oregon.
You can do the same across AWS Accounts. Say a client wants a copy of all images stored in their own AWS account. We can configure a bucket with cross-account redundancy to make a copy of any object stored in our bucket to be replicated to an S3 bucket in their account. This gives our clients complete control over backups.
Tier Selection Strategy
Costs are low for storing objects S3. You can further reduce costs by leveraging different types of tiers for object storage that AWS offers. For most, the standard tier provides low cost without any need to change. However, tiers Infrequent Access can lower costs further. These can come with additional requirements like storing for a minimum number of days, so research carefully.
LifeCycle Policy
We can also create a LifeCycle Policy on a bucket to automate actions on objects stored. You can use this to move objects to a different price tier or even expire objects entirely, depending on the need and use case. Consider a situation in which you use a bucket (or folder within a bucket) as temporary storage. Here, objects are placed to be available to a third party for a limited time, then are removed. In this case, we do not need any versions stored. We can create a LifeCycle Policy to remove any objects, including their versions, and delete them after 30 days. 
Launching Your FileMaker Application’s Storage Strategy
Soliant Consulting is both a Claris Platinum Partner and an AWS Advanced Tier Partner. Our team of experts can help you optimize your FileMaker solution from both the platform side and the cloud side. Contact us to learn more about SideCar for S3 and how to implement it for your business application today.
