Copy files compressed with tar via ssh to a different server

Sometimes we need to transfer files to a different machine. Great tools for this tasks are of course scp and rsync, but when we want to transfer folders with many small files (for example when the folder contains node_modules …) the transferring takes a lot of time, especially with rsync, as it does some checks for every file.

The solution would be to compress the files at the source and zip it to one single file, transfer this and unzip it at the destination. For this task we can use the beloved tar with all those easy to remember options.

The naive approach would be to compress the files to a .tar.gz file, send it and decompress it on the destination. This leaves ugly temporary files that nobody need, but start to clutter the drive.

Using pipes to avoid temporary files

We can leverage the power of unix pipes to avoid those temporary files. If we would like to send the folder data to our new server newserver (for which we created an entry in ~/.ssh/config) we could use the following:

tar czf - data | ssh newserver "cd /tmp && tar -xvzf - "

This compresses the folder, sends it to newserver and decompresses it to /tmp/data. From there we can move the folder, or change initially the path, where we want to cd to.