In our production server, we create thousands of pdf documents by wkhtmltopdf and sign them via iTextSharp using HSM. Most of the operation performed at night. But after we performed an update for multithreading we have been stack into a serious situation. After working a long time, our WCF service starts to give exception for every request:
System.IndexOutOfRangeException: Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
at System.Buffer.InternalBlockCopy(Array src, Int32 srcOffsetBytes, Array dst, Int32 dstOffsetBytes, Int32 byteCount
at System.IO.StreamWriter.Write(Char[] buffer, Int32 index, Int32 count
at System.IO.TextWriter.SyncTextWriter.WriteLine(String value
at iTextSharp.text.pdf.PdfReader..ctor(IRandomAccessSource byteSource, Boolean partialRead, Byte[] ownerPassword, X509Certificate certificate, ICipherParameters certificateKey, Boolean closeSourceOnConstructorError
at iTextSharp.text.pdf.PdfReader..ctor(RandomAccessFileOrArray raf, Byte[] ownerPassword, Boolean partial
at Turkkep.Daemons.Crypto.PDFSignerService.sign(Byte[] sourceData, Stream destination
at Turkkep.Daemons.Crypto.PDFSignerService.Sign(Byte[] sourceData) Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type
at Turkkep.Core.Signature.IPDFSigner.Sign(Byte[] sourceData
at Turkkep.Shared.Signature.PDFSigner.Sign(Byte[] sourceData)
--- End of inner exception stack trace --- at Turkkep.Shared.Signature.PDFSigner.Sign(Byte[] sourceData
at Turkkep.Shared.Daemon.InvoiceWorker.getPdfDocument(InvoiceType12 ublInvoice, Int64 organization
at Turkkep.Shared.Daemon.InvoiceWorker.processEArchive(Invoice invoice, String signedInvoice, InvoiceType12 ublInvoice
at Turkkep.Shared.Daemon.InvoiceWorker.sendInvoice12(Invoice invoice)
Our signing code for WCF service is;
private void sign(byte[] sourceData, Stream destination)
{
RandomAccessFileOrArray source = new RandomAccessFileOrArray(sourceData);
using (var reader = new PdfReader(source, null))
{
using (var stamper = PdfStamper.CreateSignature(reader, destination, '\0'))
{
var appearance = stamper.SignatureAppearance;
appearance.Reason = _reason;
appearance.Location = _location;
appearance.SetVisibleSignature(new Rectangle(0, 0, 0, 0), 1, String.Format("TurkKEP-{0}-{1}-{2}", DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day));
var externalSigner = new PDFExternalSigner("SHA-256");
ICollection<X509Certificate> chain = getChain(externalSigner.GetActiveConnection());
MakeSignature.SignDetached(appearance, externalSigner, chain, null, null, null, _estimatedSize, CryptoStandard.CMS);
}
}
}
public byte[] Sign(byte[] sourceData)
{
try
{
using (MemoryStream destination = new MemoryStream())
{
sign(sourceData, destination);
return destination.ToArray();
}
}
catch (FaultException)
{
throw;
}
catch (Exception ex)
{
_logger.Error("PDFSignerService.Sign", ex);
throw new Exception(ex.ToString());
}
}
I think locking the PdfReader can solve the problem but I don't want to weak multithread power. Getting an exception for some messages is not a problem. In this situation service never works again until we restart it.
Note: Production server is up to date. We've just updated.
Edit: Last night we have caught similar exception with different stacktrace:
19.12.2017 04:03:08
System.Exception: PDF sign failed. Crypto service communication exception occured.
---> System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: System.IndexOutOfRangeException: Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader's or TextWriter's Synchronized methods. This also applies to classes like StreamWriter and StreamReader.
at System.Buffer.InternalBlockCopy(Array src, Int32 srcOffsetBytes, Array dst, Int32 dstOffsetBytes, Int32 byteCount
at System.IO.StreamWriter.Write(Char[] buffer, Int32 index, Int32 count
at System.IO.TextWriter.SyncTextWriter.WriteLine(String value
at iTextSharp.text.pdf.PdfStamperImp.Close(PdfIndirectReference info, Int32 skipInfo
at iTextSharp.text.pdf.PdfStamperImp.Close(IDictionary`2 moreInfo
at iTextSharp.text.pdf.PdfSignatureAppearance.PreClose(Dictionary`2 exclusionSizes
at iTextSharp.text.pdf.security.MakeSignature.SignDetached(PdfSignatureAppearance sap, IExternalSignature externalSignature, ICollection`1 chain, ICollection`1 crlList, IOcspClient ocspClient, ITSAClient tsaClient, Int32 estimatedSize, CryptoStandard sigtype, SignaturePolicyIdentifier signaturePolicy
at Turkkep.Daemons.Crypto.PDFSignerService.sign(Byte[] sourceData, Stream destination
at Turkkep.Daemons.Crypto.PDFSignerService.Sign(Byte[] sourceData) Server stack trace: at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc
After this exception service throws same exception for every call.
Now we are going to change code as following for moving the PdfStamper out of the using block.
private void sign(byte[] sourceData, Stream destination)
{
RandomAccessFileOrArray source = new RandomAccessFileOrArray(sourceData);
using (var reader = new PdfReader(source, null))
{
try
{
var stamper = PdfStamper.CreateSignature(reader, destination, '\0');
var appearance = stamper.SignatureAppearance;
appearance.Reason = _reason;
appearance.Location = _location;
appearance.SetVisibleSignature(new Rectangle(0, 0, 0, 0), 1, String.Format("TurkKEP-{0}-{1}-{2}", DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day));
var externalSigner = new PDFExternalSigner("SHA-256");
ICollection<X509Certificate> chain = getChain(externalSigner.GetActiveConnection());
MakeSignature.SignDetached(appearance, externalSigner, chain, null, null, null, _estimatedSize, CryptoStandard.CMS);
stamper.Dispose();
}
catch (Exception ex)
{
_logger.Error("Document sign failed unexpectedly.", ex);
throw new FaultException("Document sign failed unexpectedly. Exception: " + ex.ToString());
}
}
}