Question

我正在使用Python的 libtaxii 库从 Hailataxii 服务获取提要。它有9个供稿，我已经成功获取了6个供稿，但是其中3个供稿给了我 Memory Error （内存错误）。

我已经在具有15GB RAM的Linux上运行了该程序，跟踪了6个小时，并且内存使用率超过了12小时，当我再次检查进度后，它显示了 Memory Error 。

import libtaxii as t
import libtaxii.messages_11 as tm11
import libtaxii.clients as tc
from libtaxii.common import generate_message_id
from libtaxii.constants import *

RESULTING_FOLDER = 'results'

FEEDS = [
    'Abuse_ch',
    'CyberCrime_Tracker',
    'EmergingThreats_rules',
    'Lehigh_edu',
    'MalwareDomainList_Hostlist',
    'blutmagie_de_torExits',
    'dataForLast_7daysOnly',
    'dshield_BlockList',
    'phishtank_com'
]

USERNAME = 'guest'

PASSWORD = 'guest'


def main(username, password, feeds, result_folder):
    client = tc.HttpClient()
    client.set_auth_type(tc.HttpClient.AUTH_BASIC)
    client.set_auth_credentials({'username': username, 'password': password})
    client.set_use_https(False)

    for feed in feeds:
        print('Start extracting `{}` feed ({}/{})'.format(feed, feeds.index(feed)+1, len(feeds)))
        #print('Start extracting `'+str(feed)+'` feed ('+str(feeds.index(feed)+1)+'/'+str(len(feeds))+')')
        poll_request = tm11.PollRequest(generate_message_id(),
                                        collection_name=feed,
                                        subscription_id=generate_message_id())
        poll_xml = poll_request.to_xml(pretty_print=True)
        http_resp = client.call_taxii_service2('hailataxii.com', '/taxii-data', VID_TAXII_XML_11, poll_xml)
        taxii_message = t.get_message_from_http_response(http_resp, poll_request.message_id)

我希望在结果目录中创建XML文件，因为 get_message_from_http_response（）函数之后的代码会将响应以XML格式写入。除blutmagie_de_torExits，dataForLast_7daysOnly和phishtank_com外，所有提要均已成功提取。控制台错误是

Traceback (most recent call last): 
 File "hailataxii_feeds_saver.py", line 63, in <module> 
    feeds.feeds_arg, result_folder.result_folder_arg) 
 File "hailataxii_feeds_saver.py", line 31, in main 
    http_resp, poll_request.message_id) 
 File "/home/bscs8/.local/lib/python2.7/site-packages/libtaxii/ init .py", li ne 47, in get_message_from_http_response 
    return get_message_from_urllib_addinfourl(http_response, in_response_to) 
 File "/home/bscs8/.local/lib/python2.7/site-packages/libtaxii/ init .py", li ne 109, in get_message_from_urllib_addinfourl 
    return tmll.get_message_from_xml(response_message, encoding) 
 File "/home/bscs8/.local/lib/python2.7/site-packages/libtaxii/messages_11.py", line 75, in get_message_from_xml 
    xml_string = xml_string.decode(encoding, 'replace') 
 File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode 
    return codecs.utf_8_decode(input, errors, True) 
MemoryError

从hailataxii服务获取提要时如何解决“ Memory Error”？

0 个答案: