使用python从JSON文件中删除一些特定对象,然后将该文件推送到Elasticsearch Index

时间:2015-09-21 08:24:44

标签: python elasticsearch boto amazon-sqs

我有一个包含许多对象的大型json文件。我试图从这个提取的文件中删除一定数量的对象,因为它们包含公钥和东西..只是对于一个ref我有类似这样的名字download.json:

import os
import json
import uuid
import time
import boto.sqs
import boto
from boto.sqs.connection import SQSConnection
from boto.sqs.message import Message
from boto.sqs.message import RawMessage

sqs = boto.sqs.connect_to_region("ap-southeast-1")
q = sqs.get_queue("Nishantqueue")

#text_file = open('download.json', 'w')
m = q.read(visibility_timeout=15)
if m == None:
  print "No message!"
else:
  with open('download.json', 'w') as json_data:
    print m.get_body()
    json_data.write(m.get_body())

#    clean_data = json.load(json_data)  ##
#    for element in clean_data:         ##
#      del element['sshdsakey','sshrsakey'] 
#      json_data.write(clean_data) ## 

    json_data.close()
    q.delete_message(m)
    print "Push To ES"
    os.system('./push_to_ES.sh')
    print "Cleaning the temporary json file"
    os.remove('download.json')
    print "++++++ SUCCESSFUL RUN +++++++"

我的python脚本从SQS队列下载上面的json文件,并按如下方式将其推送到ES服务器:

IOError: File not open for reading

现在您可以在注释部分中看到,我正在尝试删除我不想要的对象,然后将其写入download.json文件并将其写入ES。这个评论的部分没有像我想的那样工作,并抛出类似的东西: $sqldelreq = "DELETE FROM `requests` WHERE `tablecode` = 1 and `type` = 'order'";

非常感谢任何帮助

1 个答案:

答案 0 :(得分:-2)

以读写模式打开文件。

变化: -

with open('download.json', 'w') as json_data:

with open('download.json', 'r+') as json_data:

------------- EDIT -----------------

为什么不做以下事情: -

clean_data = json.loads(m.get_body())
for element in clean_data:
  del element['sshdsakey','sshrsakey'] 

#write clean_data to download.json  
json_data.write(clean_data)

q.delete_message(m)
print "Push To ES"
os.system('./push_to_ES.sh')
print "Cleaning the temporary json file"
os.remove('download.json')
print "++++++ SUCCESSFUL RUN +++++++"