Question

所以我正在尝试使用load data local infile命令直接将.csv文件导入到mysql中，但是我正在运行的字段是残留的excel公式字段，我不知道如何摆脱它们。以下示例中第一个字段的内容前面有=。

表格结构设置为允许第一个字段为VARCHAR(100)，但如果可能，我想将其设为INT。以下是正在上传的csv内容示例。

"MID","DBA Name","Partner ID","Partner Name","Sub Partner ID","Sub Partner Name","Active Months","Bonus Amount","Bonus Applied Date","Partner Percentage","Partner Share","Total Payment"
="0008788014065741","company2","7968","me,"11839","Joe Blow","0","$50.00","","","","$350.64"
="0008788014065756","company2","7968","you","11839","Joe Blow","0","$50.00","","","","$294.60"

这是我用来导入数据的mysql load命令：

sql = """
    LOAD DATA LOCAL INFILE '%(upload)s' IGNORE INTO TABLE `%(table)s`
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' 
    LINES TERMINATED BY '\\r\\n' 
    IGNORE 1 LINES ;
      """ % {"upload": file, "table": report}
self.db.query( sql )

在正则表达式或其他方面，是否可以对导入进行某些操作？我不知道，我只是抓住稻草......

感谢您的投入！

Answer 1

您可以使用LOAD DATA INFILE两种方式完成此任务。

首先按原样读取第一个字段值，然后在=子句中删除等号SET和双引号。此外，您最有可能在加载数据时进行其他转换：

在字段为空时设置实际NULL
从货币价值中剥离美元符号
您可能需要转换日期值（但您的示例数据没有它们，因此没有可以推断出的信息）

LOAD DATA LOCAL INFILE '/path/to/your/file.csv' 
IGNORE INTO TABLE table_name
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY '\r\n'
    IGNORE 1 LINES 
(@MID, DBAName, PartnerID, PartnerName, SubPartnerID, SubPartnerName, ActiveMonths,
 @BonusAmount, @BonusAppliedDate, @PartnerPercentage, @PartnerShare, @TotalPayment)
SET MID = TRIM(BOTH '"' FROM SUBSTR(@MID, 2)), -- here we get rid of equal sign and double quotes
    BonusAmount  = TRIM(LEADING '$' FROM NULLIF(@BonusAmount, '')),
    BonusAppliedDate = NULLIF(@BonusAppliedDate, ''),
    PartnerPercentage = NULLIF(@PartnerPercentage, ''),
    PartnerShare = TRIM(LEADING '$' FROM NULLIF(@PartnerShare, '')),
    TotalPayment = TRIM(LEADING '$' FROM NULLIF(@TotalPayment, ''))

第二种方法是利用LINES STARTING BY子句

LOAD DATA LOCAL INFILE '/path/to/your/file.csv' 
IGNORE INTO TABLE table_name
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY '\r\n' STARTING BY '='
    IGNORE 1 LINES 
(MID, DBAName, PartnerID, PartnerName, SubPartnerID, SubPartnerName, ActiveMonths, @BonusAmount, BonusAppliedDate, PartnerPercentage, PartnerShare, @TotalPayment)
SET BonusAmount  = TRIM(LEADING '$' FROM NULLIF(@BonusAmount, '')),
    BonusAppliedDate = NULLIF(@BonusAppliedDate, ''),
    PartnerPercentage = NULLIF(@PartnerPercentage, ''),
    PartnerShare = TRIM(LEADING '$' FROM NULLIF(@PartnerShare, '')),
    TotalPayment = TRIM(LEADING '$' FROM NULLIF(@TotalPayment, ''))

现在，如果您的目标表架构看起来像

CREATE TABLE table_name 
(
    MID BIGINT, 
    DBAName           VARCHAR(100),
    PartnerID         INT,
    PartnerName       VARCHAR(100),
    SubPartnerID      INT,
    SubPartnerName    VARCHAR(100),
    ActiveMonths      INT,
    BonusAmount       DECIMAL(19, 2),
    BonusAppliedDate  DATE,
    PartnerPercentage DECIMAL(3, 2),
    PartnerShare      DECIMAL(19, 2),
    TotalPayment      DECIMAL(19, 2)
);

然后在加载任何一种方法后，我们在表格中得到了

mysql> select * from table_name;
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
| MID           | DBAName  | PartnerID | PartnerName | SubPartnerID | SubPartnerName | ActiveMonths | BonusAmount | BonusAppliedDate | PartnerPercentage | PartnerShare | TotalPayment |
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
| 8788014065741 | company2 |      7968 | me          |        11839 | Joe Blow       |            0 |       50.00 | NULL             |              NULL |         NULL |       350.64 |
| 8788014065756 | company2 |      7968 | you         |        11839 | Joe Blow       |            0 |       50.00 | NULL             |              NULL |         NULL |       294.60 |
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
2 rows in set (0.00 sec)

在mysql加载数据infile导入期间删除CSV公式值

1 个答案: