Question

我有一个bash脚本，它使用curl命令检索网站上的一些JSON信息。

我正在尝试从这些JSON结果中选择一些特定信息，并将它们填充到CSV文件中。以下就我所知。

首先是我用来从网站上提取JSON数据的bash脚本。（出于安全原因，我删除了会话ID并更改了信息）

#!/bin/bash
#this script takes in the id of any customer and gives back the customer details in json format

mdid=("abcdefg" "hijklmno")
result=$(
for i in "${mdid[@]}"
do
#curl command to connect to the session and use $i as the variable for the mdid above.
curl "https://www.test.com/api/customer/$i" -H "Cookie: xxx";  -H "X-Requested-With: XMLHttpRequest" -w "\n" -H "Connection: keep-alive" --compressed
done;
)
#printing to test.txt in same folder as bash script
echo "$result" > test.json

这导致以下条目进入test.json，两行分开

{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null}
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}

然后我使用我在另一个问题（Converting JSON to .csv）上找到的python脚本来处理与此类似的问题，并将其用作以下内容

import csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))

for i in data:
    ma = data[i]["merchantApplication"]
    array = ma["email"]
    for j in array:
        f.writerow(j)

json_data.close()

当我尝试在终端中运行此python脚本时，我收到以下错误：

追踪（最近一次通话）：文件“merch_dets.py”，第5行，in data = json.load（json_data）文件“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”，第290行，载入中 ** KW）文件“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”，第338行，载入中 return _default_decoder.decode（s）文件“/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”，第369行，解码提出ValueError（errmsg（“额外数据”，s，结束，len（s））） ValueError：额外数据：第2行第1行 - 第95行第1列（字符2887 - 250855）

我知道我呈现json行/文件的方式有问题，因为当我在在线格式化程序中尝试一行时，它工作正常，但两行都没有。我在使用JSON和cURL方面经验很少。

Answer 1

您的json文件无效JSON，因为您有多个JSON根元素。

您可以使用在线验证器进行检查。例如：https://jsonformatter.curiousconcept.com/

将有效的json写入文件后，请再试一次。

例如，要使其有效：

而不是：

{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null}
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}

使用：

[
 {"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"Studio","ERROR_LIST":[],"MERCHANT_CODE":"218331","VALID":true,"_mId":"T712484","_status":"INPROCESS","email":"test7@gmail.com","name":"Studio","valid":true},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[{"accountMode":"Live","label":"Virtual Terminal","publishableKey":"niunibiubniunijknkjknj","source":"VIRTUAL_TERMINAL"},{"accountMode":"Live","label":"Default","publishableKey":"iiuhiuhiu","source":"ECOMMERCE"}],"appLogoUrl":null,"applicationId":"541d75e0-7db8b343a31f","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":null,"email":"test1@gmail.com","id":"ddddeff","invitationCode":null,"locale":"en_IE","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"54 My St, 1","businessAddress2":null,"businessCity":"Abbey","businessCountry":"IRL","businessPhone":null,"businessState":"DUBLIN","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"UAB \"Studio\"","maxTransactionAmount":null,"mccCode":"5712","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"tlana","ownerLastName":"nava","phone":"37647","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"9104d65i08d071","routingNumber":null,"singleTransactionAmount":null,"statementName":"UAB \"Studio\"","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":"http://www.design.lt"},"merchantId":"12484","merchantPromotionCode":null,"mposEnabled":true,"name":"Studio","netonfiguration":null,"onboardedDate":1505513232485,"onboardingMethod":null,"onboardingStatus":"INPROCESS","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[{"key":"MERCHANT_DETAILS","value":"{\"zip\":\"Wicklow\",\"phone\":\"342647\",\"email\":\"suppoor@outlook.com\",\"address\":\"Bck 6\",\"state\":\"Ireland\",\"addressLine2\":\"Unit 8, Bl Par\",\"city\":\"Wicklow\"}"},{"key":"VAT_NUMBER","value":"/evzaqen/"}],"timezone":"Europe/Dublin","tinStatus":null},
{"accountMode":"Live","acquirer":"TEST","acquirerConstraints":{"cardTypes":["MASTERCARD","MAESTRO","VISA"],"cvcRegexp":"^[0-9]{3}$","cvcRequired":true,"maxAmount":500000,"minAmount":50},"acquirerDetails":{"TEST":"test","ERROR_LIST":[],"MERCHANT_CODE":"594920","MID_ASSIGNED":true,"VALID":true,"_mId":"103558","_status":"APPROVED","acquiringMid":"1036598","descriptor":"test 8885551212","email":"test@gmail.com","gatewayMid":"SIMP337","id":"SIMP337","level4Mid":"76576576","name":"test","status":"APPROVED","transactionCurrency":"USD;EUR;GBP","valid":true,"paymentGatewayKey":"ytfytfytfyt"},"acquirerValidations":null,"allowedCurrencies":["EUR","USD","GBP"],"apiKeyPairs":[],"appLogoUrl":null,"applicationId":"949bdde5-07-d8d58f4c3d01","authorizationCode":"","closedDate":null,"closureReason":null,"declineAvsAddressFailure":false,"declineAvsZipFailure":false,"declineCvcFailure":false,"defaultCurrency":"EUR","descriptor":"test85551212","email":"test@gmail.com","id":"9f3a7d7","invitationCode":null,"locale":"en_US","merchantApplication":{"accountNumber":null,"acquirer":"TEST","annualAmount":null,"annualVolume":null,"applicationType":"APPROVAL","bankName":"UNKNOWN","brand":null,"businessAddress":"123 test","businessAddress2":null,"businessCity":"Atlanta","businessCountry":"IRL","businessPhone":null,"businessState":"CARLOW","businessZip":null,"data":null,"email":"test@gmail.com","escalationPhone":null,"fax":null,"legalName":"stest","maxTransactionAmount":null,"mccCode":"521","merchantPromotionCode":null,"mobile":null,"monthlyAmount":null,"monthlyVolume":null,"ownerFirstName":"moto","ownerLastName":"test","phone":"3141212","GuideAccepted":null,"privacyAccepted":true,"privacyVersion":"1a","referenceId":"2920","routingNumber":null,"singleTransactionAmount":null,"statementName":"test","taxId":null,"termsAccepted":true,"termsVersion":"1a","url":null},"merchantId":"1036558","merchantPromotionCode":null,"mposEnabled":true,"name":"test","netonfiguration":null,"onboardedDate":1456846054925,"onboardingMethod":null,"onboardingStatus":"CLOSED","partner":null,"saqCompliant":false,"saqExpires":null,"settings":[],"timezone":"Europe/Dublin","tinStatus":"InCompliance"}
]

Answer 2

除了changes to the JSON format mentioned by Bret Weinraub之外，还需要更改迭代JSON元素的代码。现在你正在使用新数组，好像它是一本字典。

此外，您在一个名为"email"的变量中存储字段array的字符串值（在给定示例中始终只包含一个字符串），并遍历此变量以输出电子邮件地址。这将以不同的行输出电子邮件地址的所有字符。

根据Bret Weinraub的建议更改您的JSON格式并更改您的迭代（我更自由地重命名您的变量：

import csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))
for entry in data:
    if "merchantApplication" in entry:
        ma = entry["merchantApplication"]
        if "email" in ma:
            f.writerow([ma["email"]])

json_data.close()

请注意，writerow也将数组作为参数，数组中的每个条目都对应于生成的CSV文件的不同列。 See the Python csv documentation for details.由于您的示例代码似乎只创建了一个列，因此我将ma["email"]直接插入到我在此处传递给writerow的数组中。

entry是你的词典（你的新JSON列表中每行有一个）。

我在我的代码示例中包含了检查，如果它们格式错误/不包含您想要的数据，它将跳过JSON条目。此外，这个解决方案非常强大，可以很好地处理换行符，如果它们没有破坏JSON格式 - 您的JSON词典不必分别在一行中。

更新：Unicode

正如您在评论中提到的，您遇到了Unicode编码错误。这是因为您的输入是Unicode编码，输出是Ascii代码。一个总是有点烦人的问题。 Python CSV包不能很好地支持Unicode，但有一个drop-in replacement called unicodecsv，它可以很轻松地解决您的问题。您只需将导入交换为unicodecsv即可。示例代码如下：

import unicodecsv as csv
import json

json_data = open("test.json")
data = json.load(json_data)

f = csv.writer(open("results.csv","wb+"))
for entry in data:
    if "merchantApplication" in entry:
        ma = entry["merchantApplication"]
        if "email" in ma:
            f.writerow([ma["email"]])

json_data.close()

或者，您可以手动编码所有数据 - 我通常会尝试避免这种情况，并且完全使用Unicode字符串。

Answer 3

恕我直言，最简单的方法是先逐行处理文件并将每一行（看作一个字符串）转换为字典：

...
json_data = open("test.json")
for line in json_data:
    data = json.loads(line)
    ma = data["merchantApplication"]
    array = ma["email"]
    f.writerow(array)
...

请注意，上面的代码将创建一个包含单个字段的csv文件...

从bash脚本

3 个答案:

更新：Unicode