Being a DBA and data professional doesn’t mean I always work with SQL Server – sometimes I’m not working with databases at all. We’ve recently acquired some new storage at work (aka Daddy Warbucks bought us a new SAN) and I’ve been charged with moving things to it. Some aspects of this are easier than others and there will be a few more posts coming about that in the future.
For now though, let’s talk about copying files. Lots of files. Server logs, audit logs, things like that. Mostly they were small in size but large in quantity. I ended up with a handful of directories with several thousand files in them that needed to move from SAN A to SAN B. Windows gives us several ways to do this:
- Copy in Windows Explorer (ewww)
- The good ol’ DOS Copy command
- Xcopy, which offers a few more features
- Robocopy, the most advanced of the MS copy utilities (though I believe it prefers to be called Murphy)
All of these will do a fine job of copying your files, though Robocopy will probably be the fastest due to its multithreading capabilities. But how do you know they all reached their destination intact? Copy and Xcopy offer the option of verification (both using the /v parameter) but sadly Robocopy does not. I’m not sure if verification is just built-in to Robocopy and can’t be disabled, or if it doesn’t exist at all. Either way I didn’t want to risk errors in moving all this data, so I decided to go the extra mile and use another tool to make sure. It didn’t take me long to find the Microsoft File Checksum Integrity Verifier (“FCIV” for short), a nifty little unsupported command-line utility that does exactly what I was looking for.
FCIV In A Nutshell
Basically, FCIV calculates MD5 or SHA-1 hash values for files and outputs them either to the screen or to an XML file. It can also compare files to those checksums saved in XML and tell you if anything differs or is missing. A demo is worth a lot of words, so let’s see it in action!
- Download Microsoft FCIV and extract the executable wherever you like – for this demo I put it in G:\
- Download the demo files and extract them. I put mine in G:\demofiles
- Use FCIV to generate checksums of all files in the folder and save to an XML file with the following syntax:
fciv.exe -add G:\demofiles -wp -sha1 -xml G:\hashdb.xml
-wp means we’re saving only the file names in the XML file, not their full path
-sha1 specifies to calculate a SHA-1 hash on each file. The default is MD5.
-xml means output the checksums to an XML file, in this case the G:\hashdb.xml that follows it.
Let’s open up that XML file and see what it contains:
As you can see it’s very simple, just the file names and a checksum for each. Now let’s make a few changes.
- Change the name of the directory the files are in. I changed mine from “demofiles” to “demofiles2”.
- Delete fileE.txt
- In fileD.txt, delete the line that says “***DELETE THIS LINE***”
Now let’s use FCIV to verify our files against the checksums we captured in the XML file. Change the current directory to demofiles2 (it won’t work unless you do this) and then run
G:\fciv.exe -v -sha1 -xml G:\hashdb.xml
-v means we’re now in verification mode, so it will verify checksums in the current directory against those in the XML file
-sha1 again specifies we’re using the SHA-1 hash
-xml is the file we’re comparing our calculated checksums against
Here’s the output it produces:
As you can see, FCIV is telling us that the contents of fileD have changed and fileE is missing. It’s really that easy!
Final Thoughts
I think FCIV is a great utility to keep in your toolbox. Some people may argue that checksum verification isn’t necessary – that Windows does it for you behind the scenes. That may be entirely true, but I wasn’t able to find any concrete documentation proving that it does. Then 10 minutes I spent finding this program online and figuring it out is a very small price to pay for some extra peace of mind in knowing that thousands of files made it to their destination intact.
Others may raise the point that both the MD5 and SHA-1 checksums both suffer from collision vulnerabilities and there are better alternatives out there that this application doesn’t support. They’re totally correct, but it’s also important to remember that we’re using these checksums to detect changes, not for cryptography or protecting secrets. Any form of verification is better than none, and for my purposes FCIV has proven to be very helpful.
