Question

我有一个300 MB的CSV，其中包含来自Geonames.org的300万行城市信息。我正在尝试将此CSV转换为JSON以使用mongoimport导入MongoDB。我想要JSON的原因是它允许我将“loc”字段指定为数组而不是用于地理空间索引的字符串。 CSV以UTF-8编码。

我的CSV片段如下所示：

"geonameid","name","asciiname","alternatenames","loc","feature_class","feature_code","country_code","cc2","admin1_code","admin2_code","admin3_code","admin4_code"
3,"Zamīn Sūkhteh","Zamin Sukhteh","Zamin Sukhteh,Zamīn Sūkhteh","[48.91667,32.48333]","P","PPL","IR",,"15",,,
5,"Yekāhī","Yekahi","Yekahi,Yekāhī","[48.9,32.5]","P","PPL","IR",,"15",,,
7,"Tarvīḩ ‘Adāī","Tarvih `Adai","Tarvih `Adai,Tarvīḩ ‘Adāī","[48.2,32.1]","P","PPL","IR",,"15",,,

与mongoimport一起使用的所需JSON输出（charset除外）如下：

{"geonameid":3,"name":"Zamin Sukhteh","asciiname":"Zamin Sukhteh","alternatenames":"Zamin Sukhteh,Zamin Sukhteh","loc":[48.91667,32.48333] ,"feature_class":"P","feature_code":"PPL","country_code":"IR","cc2":null,"admin1_code":15,"admin2_code":null,"admin3_code":null,"admin4_code":null}
{"geonameid":5,"name":"Yekahi","asciiname":"Yekahi","alternatenames":"Yekahi,Yekahi","loc":[48.9,32.5] ,"feature_class":"P","feature_code":"PPL","country_code":"IR","cc2":null,"admin1_code":15,"admin2_code":null,"admin3_code":null,"admin4_code":null}
{"geonameid":7,"name":"Tarvi? ‘Adai","asciiname":"Tarvih `Adai","alternatenames":"Tarvih `Adai,Tarvi? ‘Adai","loc":[48.2,32.1] ,"feature_class":"P","feature_code":"PPL","country_code":"IR","cc2":null,"admin1_code":15,"admin2_code":null,"admin3_code":null,"admin4_code":null}

我已经尝试了所有可用的在线CSV-JSON转换器，但由于文件大小，它们无法正常工作。我得到的最接近的是Mr Data Converter（如上图所示），在删除文档之间的开始和结束括号和逗号后将导入MongoDb。不幸的是，该工具无法使用300 MB的文件。

上面的JSON设置为以UTF-8编码，但仍然存在字符集问题，很可能是由于转换错误？

我花了最后三天学习Python，尝试使用Python CSVKIT，尝试堆栈溢出中的所有CSV-JSON脚本，将CSV导入MongoDB并将“loc”字符串更改为数组（这不幸地保留了引号）甚至尝试一次手动复制和粘贴30,000条记录。很多逆向工程，反复试验等等。

有没有人知道如何在保持编码正确的同时实现上述JSON，就像上面的CSV一样？我完全停顿了。

Answer 1

Python标准库（加上用于十进制编码支持的simplejson）可满足您的所有需求：

import csv, simplejson, decimal, codecs

data = open("in.csv")
reader = csv.DictReader(data, delimiter=",", quotechar='"')

with codecs.open("out.json", "w", encoding="utf-8") as out:
   for r in reader:
      for k, v in r.items():
         # make sure nulls are generated
         if not v:
            r[k] = None
         # parse and generate decimal arrays
         elif k == "loc":
            r[k] = [decimal.Decimal(n) for n in v.strip("[]").split(",")]
         # generate a number
         elif k == "geonameid":
            r[k] = int(v)
      out.write(simplejson.dumps(r, ensure_ascii=False, use_decimal=True)+"\n")

“in.csv”包含您的大csv文件。以上代码经过测试，可用于Python 2.6＆amp; 2.7，带有大约100MB的csv文件，生成一个正确编码的UTF-8文件。没有包围括号，数组引用或逗号分隔符，如请求。

还值得注意的是，传递ensure_ascii和use_decimal参数是编码正常工作所必需的（在本例中）。

最后，作为based on simplejson，python stdlib json 包迟早会获得十进制编码支持。所以最终只需要stdlib。

Answer 2

也许您可以尝试使用

将csv直接导入mongodb

mongoimport -d <dB> -c <collection> --type csv --file location.csv --headerline

使用Python将CSV转换为mongoimport友好的JSON

2 个答案: