When FileMaker 17 came out, Claris also released a new tool called the FileMaker Data Migration Tool (DMT). My colleague Matt Hintz wrote a nice overview of how to use it. He explained how it made deploying FileMaker solutions a lot easier than with previous methods. More recently, Mislav Kos posted about how running concurrent copies of DMT can speed up the migration of a multi-file solution. In this post, I’ll provide a brief overview of some of the tools that make DMT easier to use. I also want to introduce a new command-line tool we developed.
Existing DMT Tools
Over the last two years, there have been some wonderful tools to help developers use the DMT. This is not an in-depth review of these tools but rather a quick overview of the current landscape.
The first class of tools I want to talk about are the FM Data Migration Assistant by Productive Computing, FMDataMigration by Lesterius, FM Data Migration Companion by DB Services, and the FileMaker Data Migration Tool Demo that was created by Soliant’s own Wim Decorte. Each tool is a FileMaker file that provides a very easy GUI to migrate your database files. They are by far the easiest way to get up and running with DMT. They’re a great starting point for anyone who wants to experiment with DMT without having to use the command line.
If you are comfortable with command-line tools, or if you don’t want to run FileMaker Pro to convert the files, you might want to write a bash or batch script to run DMT. If you frequently migrate the same files, this might be a good solution for you as it does not require you to install or buy any extra software like FileMaker Pro or one of the tools I talk about next.
The next class of tools I would like to cover includes 360Deploy by 360Works, Otto by Geist Interactive, and base72 by 27solutions. These tools are more than just something to help you run DMT as they also help with creating clones on your development server and with moving those clones to the production server. These tools require some setup, but their goal is to make deployments as simple as pushing a button.
Multi-File Deployments
As Mislav posted, one of our clients wanted to have deployments run as quickly as possible, because due to the 24×7 nature of their business, all FileMaker outages, including those caused by deployments, had a serious business impact. In addition to speed, security was a major concern – they didn’t want the full access password stored in either clear text or a reversible manner as is required by all other tools (except a batch file that prompts for and then stores the password in a variable). A sufficiently sophisticated system script could accomplish all of these requirements, but the development effort requires a specialized skillset.
We ended up creating a simple application that we are releasing here. We have since tested it on a few other projects and wanted to share it with the wider FileMaker community.
YADMT (Yet Another Data Migration Tool)
YADMT (Yet Another Data Migration Tool) is a command-line tool written in .NET. It essentially acts as a DMT wrapper and is intended for running DMT on more than one database file (although it will work fine if used with just one file). The tool has been mainly tested on Windows and minimally tested on macOS using Mono.
YADMT is intended for situations where there are at least two files to migrate, the overall size of the solution files is large, and the migration time needs to be made as short as possible. You can select to have YADMT run anywhere from 1 to 9 concurrent DMT processes.
YADMT will process files from largest to smallest to run the parallel jobs as efficiently as possible. When you run the tool, it will prompt you with several questions; however, these inputs can also be specified using an ini file. YADMT is dependent on the context you run it from as the clone files, source files, ini, and logs are all based on the current command line directory.
Setting Up Folder Structure with Our FileMaker Data Migration Tool
Below is an example of how you can set up your folder structure for use with YADMT. In this example, we organized our deployment using a timestamped folder (2020-01-01) and “clone” and “source” subfolders.
C:\DEPLOYMENTS
│ YADMT.exe
│
├───2020-01-01
│ │ YADMT.ini
│ │
│ ├───clone
│ │ DEV_Data Clone.fmp12
│ │ DEV_UI Clone.fmp12
│ │
│ ├───source
│ │ Data.fmp12
│ │ UI.fmp12
│
└───fm_data_migration_18.0.3.317_x64
DBEngine.dll
FMDataMigration.exe
libcrypto-1_1-x64.dll
libcurl.dllIn the example shown above, you would need to change the current directory to c:\deployments\2020-01-01\ before running YADMT. You would also need to tell YADMT that your clone files have a prefix of DEV_.
When you run YADMT, you will get a log file for each migrated file and another log file that shows the timing of each migration.
# Proces File Size(b). start finish. duration 1 1 Data 70021120 14:36:29.3 14:36:34.4 00:00:05.0626747 2 2 UI 24850432 14:36:29.3 14:36:31.9 00:00:02.5635413 ---- ---- ------ -------- -------- -------- -------- 2 2 - 94871552 14:36:29.3 14:36:34.5 00:00:05.1592291 ---- ---- ------ -------- -------- -------- --------
In the console, you will see the output of each concurrent DMT process combined in real-time. Each line of DMT output is prefixed with the file name and with a timestamp.
The tool will create the migrated files in the target folder (which will be automatically created if it doesn’t already exist). The target files will be named with whatever prefixes or suffixes are used by the clones.
Finally, YADMT will inform you if there is a new version of DMT or YADMT. It will do this by checking a file hosted on one of our servers and caching the versions in a small file located in %USERPROFILE%\.YADMT\ on Windows and ~/.YADMT/ on Linux and macOS. If the cached value is more then 24 hours old, YADMT will check the server again. If you are running YADMT on a computer that has a restrictive network, checking for the update should fail silently and gracefully.
This short 3-minute video shows us using YADMT to migrate two database files.
YADMT Arguments and Prompts
Let’s walk through the various arguments and prompts used by YADMT.
The original DMT uses several arguments – these are the options that can be specified, such as -src_path. YADMT, on the other hand, will accept only one argument – the path to the DMT executable. This is an optional argument. If it’s left blank, YADMT will look for the DMT exe in the current, parent, and sibling folders (based on the current directory).
Which DMT Arguments Should It Use?
So how does YADMT know which DMT arguments it should use? It can determine some of them automatically. For example, because YADMT requires that clone and source files are placed in Clone and Source subfolders, it’s able to tell DMT where those files are located.
YADMT will prompt for the remaining DMT arguments. It will ask you to specify the account name and password (the same credentials must be used with all of the files being migrated). And it will ask you if you would like to specify any extra arguments (like -v, -force, -src_key, -ignore_accounts).
How to Specify the Number of Concurrent DMT Processes to Run
In addition to these prompts, YADMT will also ask you to specify how many concurrent DMT processes should be run. Choosing the optimal number of processes will require some experimentation. See Mislav’s post for more details about that – but keep in mind that each DMT process will use about 2-3 GB of RAM.
Finally, YADMT will also ask whether or not the clone or source files have a suffix or a prefix. Some development environments set up their files in such a way. For example, the dev files might be prefixed with “DEV_.” If a prefix or a suffix is used, the target files will be created in the same way. (The prefix/suffix will always come from the clone files.)
There is no need to remember all of this detail. The arguments and prompts will be summarized for you when you run YADMT. If you want to be reminded of it without actually running the tool, you can always run the command and then hit control-c to stop YADMT.
INI file
The ini file is not required, but you may find it useful in certain situations. It is a simple key-value text file. Any line that starts with a semicolon is ignored. Below is a sample ini file that has every setting defined.[/vc_column_text]
;dmtPath can be either relative or absolute
dmtPath=C:\deployments\fm_data_migration_18.0.3.317_x64\FMDataMigration.exe
;dmtArgs will be passed through to DMT
dmtArgs=-v -force
;processCnt specifies number of concurrent DMT processes
processCnt=2
user=admin
pass=FullAccessPassword
;do clones or source files have a prefix or a suffix? p=prefix, s=suffix, n=neither
hasPrefix=p
;cloneExtra and sourceExtra are ignored if hasPrefix=n
cloneExtra=dev_
sourceExtra=Download YADMT
- Source code on GitHub
- Release
If you need help with this tool or other aspects of your FileMaker solution, please contact our team for support.
Thanks for the wonderful post.
I want to appeal, the voice on the video demonstration is very low. I could barely hear you.