opportunities to improve cpu performance?

[rjkroege]

I had the (I thought inspired) but (obvious given the context here) idea of using cpu to mount a selection of pre-built binary tools into a GCP node rather than installing them or building an archive or such not. But performance is insufficient for this use case.

While Internet bandwidth is definitely a very significant part of this, I did some benchmarking and believe that cpu itself could be more performant.

# local to remote
; time rsync -av $_h/wrks/archive/bins/linux/amd64/rclone ween:
building file list ... done
rclone

sent 71019517 bytes  received 42 bytes  2407442.68 bytes/sec
total size is 71010723  speedup is 1.00
       29.16 real         0.26 user         0.32 sys
;
; time cpu ween cp /usr/local/rjkroege/wrks/archive/bins/linux/amd64/rclone /home/rjkroege/rclone2
      127.54 real         0.78 user         1.85 sys

[patattila]

Have you tested gnu parallell?

https://www.gnu.org/software/parallel/

Use tmpfs to get the best performance.

[rjkroege]

Can you expand on your suggestion. I don't think that the local filesystem is the bottleneck here because it was getting used at both ends of the transfer both with cpu and with rsync.