406: not acceptable response received using LWP::UserAgent/File::Download

Question

Edit: it seems the issue was caused by a dropped cookie. There should have been a session id cookie as well.

For posterity, here's the original question

When sending a request formed as this

htmlspecialchars();

I receive the following response

GET https://<url>?<parameters>
Cache-Control: max-age=0
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: iso-8859-1,utf-8,UTF-8
Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
Accept-Language: en-US,en;q=0.5
If-None-Match: "6eb7d55abfd0546399e3245ad3a76090"
User-Agent: Mozilla/5.0 libwww-perl/6.13
Cookie: auth_token=<blah>; __cfduid=<blah>
Cookie2: $Version="1"

I'm not entirely sure why the response is response-type: text/html charset=utf-8 HTTP/1.1 406 Not Acceptable Cache-Control: no-cache Connection: keep-alive Date: Fri, 12 Feb 2016 18:34:00 GMT Server: cloudflare-nginx Content-Type: text/html; charset=utf-8 CF-RAY: 273a62969a9b288e-SJC Client-Date: Fri, 12 Feb 2016 18:34:00 GMT Client-Peer: <IP4>:443 Client-Response-Num: 10 Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limite d/CN=COMODO ECC Domain Validation Secure Server CA 2 Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domai n/CN=ssl<blah>.cloudflaressl.com Client-SSL-Cipher: <some value> Client-SSL-Socket-Class: IO::Socket::SSL Client-SSL-Warning: Peer certificate not verified Client-Transfer-Encoding: chunked Status: 406 Not Acceptable X-Runtime: 9 . When downloaded with firefox, the file in question in 996 KB (as reported by Windows 8.1's explorer). It looks like I have a partially transferred file from my perl script at 991 KB (again, windows explorer size), so it got MOST of the file before throwing the 406 Not Acceptable response. Using the same URL pattern and request style, I was able to successfully download a 36 MB file from the server with this perl library and request form, so the size of the file should not be magically past some max (chunk) size. As these files are being updated on approximately 15-minute intervals, I suppose it's possible that a write was performed on the server, invalidating the ETag before all chunks were complete on this file?

I tried adding chunked to Not Acceptable, but that's not for Accept-Encoding encoding and it appears to have no affect on the server's behavior. Additionally, as I've been able to download larger files (same format) from the same server, that alone shouldn't be the cause of my woes. LWP is supposed to be able to handle chunked data returned by a response to Transfer (as per this newsgroup post).

The server in question is running GET with nginx. The particular server configuration (which I in no way control), throws Rack::Lint errors on its own attempts to send 500. This caused me to write a workaround for File::Download (sub lintWorkAround here), so I'm not above putting blame on the server in this instance also, if warranted. I don't believe I buggered up the chunk-handling code from File::Download 0.3 (see diff), but I suppose that's also possible. Is it possible to request a particular chunk size from the server?

I'm using 304: not modified and LWP versions 6.13 in libwww 5.18.2. perl version is my own 0.4_050601.

So, what else could the 406 error mean? Is there a way to request that the server temporarily cache/version control the entire file so that I can download a given ETag'd file once the transfer begins?