paramiko挂起后自己加载20 MB的文件

时间：2017-08-26 03:05:30

标签： python-3.x ssh paramiko

我需要python sftp客户端从sftp服务器下载文件。我开始使用Paramiko。 KB中的小文件运行良好但是当我尝试下载600 MB的文件时，它在下载20 MB文件后无限期挂起。无法弄清问题是什么。增加窗口大小也没有解决。任何帮助将不胜感激！

host = config.getsafe(section, "host")
username = config.getsafe(section, "username")
port = config.getsafe(section, "port")
remote_dir = config.getsafe(section, "remote_dir")
download_dir = config.getsafe(section, "download_dir")
archive_dir = config.getsafe(section, "archive_dir") if config.has_option(section, "archive_dir") else \
    None
password = config.getsafe(section, "password") if config.has_option(section, "password") else None
file_pattern = config.getsafe(section, "file_pattern") if config.has_option(section, "file_pattern") \
    else "*"
passphrase = config.getsafe(section, "passphrase") if config.has_option(section, "passphrase") else None
gnupg_home = config.getsafe(section, "gnupg_home") if config.has_option(section, "gnupg_home") else None

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, port=int(port), username=username, password=password)

sftp = ssh.open_sftp()
sftp.sshclient = ssh

sftp.get("/SFTP/PL_DEV/test.dat", "C:/import/download/test.dat")

3 个答案:

答案 0 :(得分：2)

我做了两件事来解决类似的问题：

  1. increase window size - 你说你也试过了;对我来说，这有助于从几十MB增加到半GB但没有进一步

  2. effectively disable rekeying - 这可能会带来安全隐患，但帮助我从一个奇怪的Windows sftp服务器上获取GB文件

    with paramiko.Transport((_SFTP['host'], 22)) as transport:
    # SFTP FIXES
    transport.default_window_size=paramiko.common.MAX_WINDOW_SIZE
    transport.packetizer.REKEY_BYTES = pow(2, 40)  # 1TB max, this is a security degradation!
    transport.packetizer.REKEY_PACKETS = pow(2, 40)  # 1TB max, this is a security degradation!
    # / SFTP FIXES

    transport.connect(username=_SFTP['user'], password=_SFTP['password'])
        with paramiko.SFTPClient.from_transport(transport) as sftp:
            listdir = sftp.listdir()
            # ...
            sftp.get(remotepath=filename, localpath=localpath)

答案 1 :(得分：1)

按如下方式增加default_max_packet_size和default_window_size对我有用：

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()

client.connect(hostname, username=username, password=password, port=port)
tr = client.get_transport()
tr.default_max_packet_size = 100000000
tr.default_window_size = 100000000
sftp = client.open_sftp()
sftp.get(remote_file, local_filepath)

client.close()

答案 2 :(得分：0)

使用最新的paramiko 2.4.2，我遇到类似的问题。就我而言，我们的供应商几天前将其SFTP提供者从Globalscape（SSH-2.0-1.82_sshlib Globalscape）切换到Cerberus（SSH-2.0-CerberusFTPServer_10.0）。从那时起，paramiko一直无法下载〜450MB的文件。

这是症状： 下载速度非常慢。下载20〜30MB后，总是出现以下错误： Server connection dropped msg.

以下是日志（Globalscape）-成功下载：

"paramiko.transport", "DEBUG", "starting thread (client mode): 0x160096d8"
"paramiko.transport", "DEBUG", "Local version/idstring: SSH-2.0-paramiko_2.4.1"
"paramiko.transport", "DEBUG", "Remote version/idstring: SSH-2.0-1.82_sshlib Globalscape"
"paramiko.transport", "INFO", "Connected (version 2.0, client 1.82_sshlib)"
"paramiko.transport", "DEBUG", "kex algos:['diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['twofish256-cbc', 'twofish-cbc', 'twofish128-cbc', 'blowfish-cbc', '3des-cbc', 'arcfour', 'cast128-cbc', 'aes256-cbc', 'aes128-cbc', 'aes256-ctr', 'aes128-ctr'] server encrypt:['twofish256-cbc', 'twofish-cbc', 'twofish128-cbc', 'blowfish-cbc', '3des-cbc', 'arcfour', 'cast128-cbc', 'aes256-cbc', 'aes128-cbc', 'aes256-ctr', 'aes128-ctr'] client mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['zlib', 'none'] server compress:['zlib', 'none'] client lang:[''] server lang:[''] kex follows?False"
"paramiko.transport", "DEBUG", "HostKey agreed: ssh-rsa"
"paramiko.transport", "DEBUG", "Cipher agreed: aes128-ctr"
"paramiko.transport", "DEBUG", "MAC agreed: hmac-sha1"
"paramiko.transport", "DEBUG", "Compression agreed: none"
"paramiko.transport", "DEBUG", "Got server p (2048 bits)"
"paramiko.transport", "DEBUG", "kex engine KexGex specified hash_algo <built-in function openssl_sha1>"
"paramiko.transport", "DEBUG", "Switch to new keys ..."
"paramiko.transport", "DEBUG", "Attempting public-key auth..."
"paramiko.transport", "DEBUG", "userauth is OK"
"paramiko.transport", "INFO", "Auth banner: b'Welcome to the our Secure FTP Server'"
"paramiko.transport", "INFO", "Authentication (publickey) successful!"
"paramiko.transport", "DEBUG", "[chan 0] Max packet in: 32768 bytes"
"paramiko.transport", "DEBUG", "[chan 0] Max packet out: 35840 bytes"
"paramiko.transport", "DEBUG", "Secsh channel 0 opened."
"paramiko.transport", "DEBUG", "[chan 0] Sesch channel 0 request ok"
"paramiko.transport.sftp", "INFO", "[chan 0] Opened sftp connection (server version 3)"
"paramiko.transport.sftp", "DEBUG", "[chan 0] stat(b'data.csv')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb') -> 31"
"paramiko.transport.sftp", "DEBUG", "[chan 0] close(31)"
"paramiko.transport.sftp", "INFO", "[chan 0] sftp session closed."
"paramiko.transport", "DEBUG", "[chan 0] EOF sent (0)"
"paramiko.transport", "DEBUG", "EOF in transport thread"

以下是日志（Cerberus）-下载失败：

"paramiko.transport", "DEBUG", "starting thread (client mode): 0x119706d8"
"paramiko.transport", "DEBUG", "Local version/idstring: SSH-2.0-paramiko_2.4.1"
"paramiko.transport", "DEBUG", "Remote version/idstring: SSH-2.0-CerberusFTPServer_10.0"
"paramiko.transport", "INFO", "Connected (version 2.0, client CerberusFTPServer_10.0)"
"paramiko.transport", "DEBUG", "kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-ctr', 'aes128-cbc', 'aes192-ctr', 'aes192-cbc', 'aes256-ctr', 'aes256-cbc', '3des-cbc'] server encrypt:['aes128-ctr', 'aes128-cbc', 'aes192-ctr', 'aes192-cbc', 'aes256-ctr', 'aes256-cbc', '3des-cbc'] client mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-md5'] server mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:['en-US'] server lang:['en-US'] kex follows?False"
"paramiko.transport", "DEBUG", "Kex agreed: ecdh-sha2-nistp256"
"paramiko.transport", "DEBUG", "HostKey agreed: ssh-rsa"
"paramiko.transport", "DEBUG", "Cipher agreed: aes128-ctr"
"paramiko.transport", "DEBUG", "MAC agreed: hmac-sha2-256"
"paramiko.transport", "DEBUG", "Compression agreed: none"
"paramiko.transport", "DEBUG", "kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>"
"paramiko.transport", "DEBUG", "Switch to new keys ..."
"paramiko.transport", "DEBUG", "Attempting public-key auth..."
"paramiko.transport", "DEBUG", "userauth is OK"
"paramiko.transport", "INFO", "Authentication (publickey) successful!"
"paramiko.transport", "DEBUG", "[chan 0] Max packet in: 32768 bytes"
"paramiko.transport", "DEBUG", "[chan 0] Max packet out: 32768 bytes"
"paramiko.transport", "DEBUG", "Secsh channel 0 opened."
"paramiko.transport", "DEBUG", "[chan 0] Sesch channel 0 request ok"
"paramiko.transport.sftp", "INFO", "[chan 0] Opened sftp connection (server version 3)"
"paramiko.transport.sftp", "DEBUG", "[chan 0] stat(b'data.csv')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb') -> 7b45394343333830462d383832352d343436342d393831302d4444373838314237303433367d"
"paramiko.transport", "DEBUG", "EOF in transport thread"

添加

transport.default_window_size = paramiko.common.MAX_WINDOW_SIZE

为我工作（至少目前如此）。不知道如果文件大小从〜450MB增加到>> 0.5GB，会发生什么情况。

相关问题
最新问题