写入S3备份时出现严重错误

时间:2015-05-06 16:17:25

标签: cassandra-2.0 datastax-enterprise opscenter

我正在使用Datastax Enterprise 4.5.1运行OpsCenter 5.1.1。它是AWS上的一个3节点集群,我正在备份到S3(仍然...)我已经开始看到一个新的错误。我认为这是一个与我发布的b4不同的错误。

node1_agent.log: SEVERE: error after writing 15736832/16777216 bytes to https://cassandra-dev-bkup.s3.amazonaws.com/snapshots/407bb4b1-5c91-43fe-9d4f-767115668037/sstables/1430904167-reporting_test-transaction_lookup-jb-288-Index.db?partNumber=2&uploadId=.MA3X4RYssg7xL_Hr7Msgze.J4exDq9zZ_0Y7qEj9gZhJ570j73kZNr5_nbxactmPMJeKf0XyZfEC0KAplWOz9lpyRCtNeeDCvCmtEXDchH8F1J2c57aq4MrxfBcyiZr java.io.IOException: Error writing request body to server at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3192) at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3175) at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53) at com.google.common.io.ByteStreams.copy(ByteStreams.java:179) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:308) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:192) at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:72) at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:95) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94) at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55) at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156) at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123) at com.sun.proxy.$Proxy48.uploadPart(Unknown Source) at org.jclouds.aws.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111) at org.jclouds.aws.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93) at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:89) at org.jclouds.blobstore2$put_blob.doInvoke(blobstore2.clj:246) at clojure.lang.RestFn.invoke(RestFn.java:494) at opsagent.backups.destinations$create_blob$fn__12007.invoke(destinations.clj:69) at opsagent.backups.destinations$create_blob.invoke(destinations.clj:64) at opsagent.backups.destinations$fn__12170.invoke(destinations.clj:192) at opsagent.backups.destinations$fn__11799$G__11792__11810.invoke(destinations.clj:24) at opsagent.backups.staging$start_staging_BANG_$fn__12338$state_machine__7576__auto____12339$fn__12344$fn__12375.invoke(staging.clj:61) at opsagent.backups.staging$start_staging_BANG_$fn__12338$state_machine__7576__auto____12339$fn__12344.invoke(staging.clj:59) at opsagent.backups.staging$start_staging_BANG_$fn__12338$state_machine__7576__auto____12339.invoke(staging.clj:56) at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940) at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944) at clojure.core.async.impl.ioc_macros$take_BANG_$fn__7592.invoke(ioc_macros.clj:953) at clojure.core.async.impl.channels.ManyToManyChannel$fn__4097.invoke(channels.clj:102) at clojure.lang.AFn.run(AFn.java:24) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

我在agent.log文件中看到了这个错误

os.system('wkhtmltopdf -q --page-width 230 --page-height 330 "%s" "%s"' % (filePath, pdfFilePath))

1 个答案:

答案 0 :(得分:1)

TL; DR -

您的SSTable是38866048字节,在您的文件系统和S3上。这意味着文件已经转移,您处于良好状态。无需担心此错误(尽管我打开了一个内部票证来处理这种异常,而不是抛出转储)。

详细信息 - 我怀疑发生的事情摘要

1)当您从sstable的16777216字节切片中达到15736832时,出现文件传输错误。

2)此时OpsCenter没有完成转移表或在s3中留下部分版本

3)稍后的另一次备份尝试移动了sstable,没有错误,并且存在有效的备份。