Question

我正在使用gsutil rsync从Google云端存储下载大量公共数据文件。有时，一些文件的下载失败。为了确保我拥有所有请求的文件，我第二次运行gsutil并打开校验和。在第二次运行期间，gsutil报告它计算校验和的文件少于下载的文件。我在下面附上了一些示例输出。在这种情况下，它在第一个rsync期间下载了29个文件，但仅报告它在第二个rsync期间计算了16个文件的校验和。

gsutil是不是计算校验和并为某些文件执行rsync，还是只是报告它正在执行校验和？

肯

mix> gsutil -m rsync -R -P gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131230000_20170131235959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201090000_20170201095959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201130000_20170201135959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201060000_20170201065959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201100000_20170201105959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201110000_20170201115959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201120000_20170201125959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201150000_20170201155959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201080000_20170201085959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201070000_20170201075959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201140000_20170201145959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201220000_20170201225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201230000_20170201235959.tar...
- [29/29 files][387.3 MiB/387.3 MiB] 100% Done                                  
Operation completed over 29 objects/387.3 MiB.                                   
mix> gsutil -m rsync -R -P -c gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
mix>

Answer 1

gsutil rsync -c仅在源和目标的大小匹配时才计算校验和。这样可以节省时间，因为无需计算校验和来确定需要下载的大小匹配文件。

使用gsutil rsync

1 个答案: