Question

硬盘，
默认的PostgreSQL配置文件
没有其他连接。

我有一张桌子：

CREATE TABLE "public"."ParamValueBlock" (
  "ParameterId" int2 NOT NULL,
  "DeviceId" int2 NOT NULL,
  "CompressedData" bytea,
  "StartDate" int4 NOT NULL,
  "UncompressedDataBits" int4 NOT NULL
)

这是我用于批量复制到数据库的代码：

connectionString := fmt.Sprintf("host=%s port=%d user=%s "+
        "password=%s dbname=%s sslmode=disable",
        host, port, user, password, dbname)
db, err := sql.Open("postgres", connectionString)
if err != nil {
    return err
}
defer db.Close()

tx, err := db.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelReadCommitted, ReadOnly: false})
if err != nil {
    return err
}

stmt, err := tx.Prepare(`COPY "ParamValueBlock" ("ParameterId", "DeviceId", "CompressedData", "StartDate", "UncompressedDataBits") FROM STDIN;`)
if err != nil {
    return err
}

for _, item := range items{
    _, err := stmt.Exec(
        int16(item.paramID),
        1,
        item.dataBuffer,
        item.secondsSince2015,
        int32(item.uncompressedBitsSize))
    if err != nil {
        tx.Rollback()
        return err
    }
}

err = stmt.Close()
if err != nil {
    tx.Rollback()
    return err
}

err = tx.Commit()
if err != nil {
    return err
}
return nil

我正在写1000项（1项= 28KB），这是7秒。

为什么这么慢，我如何优化呢？

如果来自文件：

COPY "ParamValueBlock" FROM 'C:\Temp\x.txt' (FORMAT text);

时间是0.7秒

个人资料cpu：

      flat  flat%   sum%        cum   cum%
     1.82s 31.06% 31.06%      1.83s 31.23%  runtime.cgocall
     0.35s  5.97% 37.03%      0.36s  6.14%  ...pq.appendEscapedText
     0.30s  5.12% 42.15%      0.76s 12.97%  fmt.(*fmt).fmtInteger
     0.30s  5.12% 47.27%      1.53s 26.11%  fmt.(*pp).doPrintf
     0.14s  2.39% 67.92%      2.98s 50.85%  fmt.Sprintf
     0.11s  1.88% 76.11%      0.97s 16.55%  fmt.(*pp).printArg
     0.10s  1.71% 77.82%      0.19s  3.24%  fmt.(*buffer).Write (inline)
     0.09s  1.54% 79.35%      0.85s 14.51%  fmt.(*pp).fmtInteger
     0.09s  1.54% 80.89%      3.26s 55.63%  github.com/lib/pq.encodeBytea
     0.01s  0.17% 94.37%      5.20s 88.74%  github.com/lib/pq.(*copyin).Exec

Answer 1

这是pg驱动程序错误：https://github.com/lib/pq/pull/784

我将github.com/lib/pq替换为github.com/jackc/pgx/stdlib，并在postgresql.conf中设置了此服务器配置：

autovacuum_vacuum_scale_factor = 0.0  
autovacuum_vacuum_threshold = 5000
autovacuum_analyze_scale_factor = 0.0  
autovacuum_analyze_threshold = 5000  
synchronous_commit = off
wal_level = 'minimal'
archive_mode = off
max_wal_senders = 0

这将插入速度提高了60％

其他信息

将PREPARE与INSERT或COPY一起使用
Stmt必须在交易后关闭

仅通过本机（无stdlib）接口支持
pgx COPY。您可以使用AcquireConn和ReleaseConn从数据库/ SQL池。

我的用于插入许多行的示例：

*pgx.Conn

COPY命令运行缓慢

1 个答案: