What is the best way to transfer a large number of small files? What is the optimal transfer size?
HPSS performance is greatly improved when the transfer size is between 1 and 10 GB. For that reason, users with large numbers of relatively small files should combine those files into one or a few 1 GB to 10 GB files and then transfer the larger files. The files can be combined with tar on the HPC system, or they can be created on the fly with a command similar to tar cvf some_dir -|hsi put – : somedir.tar. This command will tar all files in the some_dir subdirectory into a file named somedir.tar on HPSS. HPSS also supports the htar command.
