使用scrapy管道写入文件

时间:2019-06-13 18:52:36

标签: python scrapy

我正在尝试使用刮擦的管线文件。py,using System; namespace My.Example { public static class GetPaths { private static readonly string MyPATH = Environment.GetEnvironmentVariable("PATH"); private static readonly string MySpecialPath = Environment.GetEnvironmentVariable("PREFIX_SpecialPath"); ... } } 已正确解析,并且在运行时显示在终端中。 这是我的pipleines.py

item

错误消息:

import datetime,csv

class AmazonfullPipeline(object):
    keys = ["Product_Name","Price","Amazon_Stock","rating","ASIN","Rank1","Rank1_category","Rank2","Rank2_category",
    "UPC","Item_Model_Number"]

    def __init__(self):
        now = datetime.datetime.now()
        current_date = now.strftime("%d%b")
        file_name = "TestFile"
        infile = open("{}_{}.csv".format(current_date,file_name),"w").close()
        dict_writer = csv.DictWriter(infile, self.keys)
        dict_writer.writeheader()
    def process_item(self, item, spider):
        self.dict_writer.writerow(item)

1 个答案:

答案 0 :(得分:2)

您有几个问题:

  1. 您在使用前关闭文件描述符;
  2. 您未设置类变量。在self.dict_writer中使用dict_writer,而不是__init__

验证码:

import datetime,csv

class AmazonfullPipeline(object):
    keys = ["Product_Name","Price","Amazon_Stock","rating","ASIN","Rank1","Rank1_category","Rank2","Rank2_category",
    "UPC","Item_Model_Number"]

    def __init__(self):
        now = datetime.datetime.now()
        current_date = now.strftime("%d%b")
        file_name = "TestFile"
        infile = open("{}_{}.csv".format(current_date,file_name),"w")  # <- remove close() here
        self.dict_writer = csv.DictWriter(infile, self.keys)  # <- add self. here
        self.dict_writer.writeheader()  # <- add self. here

    def process_item(self, item, spider):
        self.dict_writer.writerow(item)