Question

我正在探索使用Elixir快速导入混合类型（CSV，JSON）的Postgres数据。作为Elixir的新手，我遵循youtube视频“使用Elixir和Postgrex快速导入和导出-Elixir Hex软件包展示”（https://www.youtube.com/watch?v=YQyKRXCtq4s）中给出的示例。基本的混合应用程序可以一直工作到引入Poolboy为止，即Postgrex使用单个连接成功将记录加载到数据库中。

当我尝试遵循Poolboy配置并通过运行

对其进行测试时

FastIoWithPostgrex.import（“ ./ data_with_ids.txt”）

在iex或命令行中，出现以下错误，我无法确定原因（删除用户名和密码）：

** (UndefinedFunctionError) function DBConnection.Poolboy.child_spec/1 is 
undefined (module DBConnection.Poolboy is not available)
DBConnection.Poolboy.child_spec({Postgrex.Protocol, [types: 
Postgrex.DefaultTypes, name: :pg, pool: DBConnection.Poolboy, pool_size: 4, 
hostname: "localhost", port: 9000, username: "XXXX", password: 
"XXXX", database: "ASDDataAnalytics-DEV"]})
(db_connection) lib/db_connection.ex:383: DBConnection.start_link/2
(fast_io_with_postgrex) lib/fast_io_with_postgrex.ex:8: 
FastIoWithPostgrex.import/1

我正在Windows 10上运行此程序，并通过本地SSH隧道连接到PostgreSQL 10.x服务器。这是lib / fast_io_with_postgrex.ex文件：

defmodule FastIoWithPostgrex do
  @moduledoc """
  Documentation for FastIoWithPostgrex.
  """

  def import(filepath) do

    {:ok, pid} = Postgrex.start_link(name: :pg,
      pool: DBConnection.Poolboy,
      pool_size: 4,
      hostname: "localhost",
      port: 9000,
      username: "XXXX", password: "XXXX", database: "ASDDataAnalytics-DEV")

    File.stream!(filepath)
    |> Stream.map(fn line ->
        [id_str, word] = line |> String.trim |> String.split("\t", trim: true, parts: 2)
        {id, ""} = Integer.parse(id_str)
        [id, word]
    end)
    |> Stream.chunk_every(10_000, 10_000, [])
    |> Task.async_stream(fn word_rows ->
      Enum.each(word_rows, fn word_sql_params ->
        Postgrex.transaction(:pg, fn conn ->
          IO.inspect Postgrex.query!(conn, "INSERT INTO asdda_dataload.words (id, word) VALUES ($1, $2)", word_sql_params)
#        IO.inspect Postgrex.query!(pid, "INSERT INTO asdda_dataload.words (id, word) VALUES ($1, $2)", word_sql_params)    
        end , pool: DBConnection.Poolboy, pool_timeout: :infinity, timeout: :infinity) 
      end)
    end, timeout: :infinity)
    |> Stream.run

  end # def import(file)
end

这是mix.exs文件：

defmodule FastIoWithPostgrex.MixProject do
  use Mix.Project

  def project do
    [
      app: :fast_io_with_postgrex,
      version: "0.1.0",
      elixir: "~> 1.7",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger, :poolboy, :connection]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", 
tag: "0.1.0"},

      {:postgrex, "~>0.14.1"},
      {:poolboy, "~>1.5.1"}
    ]
  end
end

这是config / config.exs文件：

# This file is responsible for configuring your application
# and its dependencies with the aid of the Mix.Config module.

use Mix.Config

config :fast_io_with_postgrex, :postgrex,
  database: "ASDDataAnalytics-DEV",
  username: "XXXX",
  password: "XXXX",
  name: :pg,
  pool: DBConnection.Poolboy,
  pool_size: 4

# This configuration is loaded before any dependency and is restricted
# to this project. If another project depends on this project, this
# file won't be loaded nor affect the parent project. For this reason,
# if you want to provide default values for your application for
# 3rd-party users, it should be done in your "mix.exs" file.

# You can configure your application as:
#
#     config :fast_io_with_postgrex, key: :value
#
# and access this configuration in your application as:
#
#     Application.get_env(:fast_io_with_postgrex, :key)
#
# You can also configure a 3rd-party app:
#
#     config :logger, level: :info
#

# It is also possible to import configuration files, relative to this
# directory. For example, you can emulate configuration per environment
# by uncommenting the line below and defining dev.exs, test.exs and such.
# Configuration from the imported file will override the ones defined
# here (which is why it is important to import them last).
#
#     import_config "#{Mix.env()}.exs"

在发现此错误原因方面的任何帮助将不胜感激！

Answer 1

我不想深入研究它是如何工作的，但是该示例有些古老，并且您被poolboy 1.5.1吸引的deps.get来自{{3} } ..并且该示例使用了elixir 1.4

此外，如果看到Postgrex的mix.exs部门，您会注意到您刚安装的lib（1.14）取决于elixir_ecto/db_connection 2.x

2015使用Postgres 1.13.x，它取决于{:db_connection, "~> 1.1"}。所以我会期望不兼容。

如果要查看它的工作原理，我将使用示例代码mix.lock文件中看到的lib的版本，这是一种长生不老药的版本。

也许在那个时候先尝试将Postgrex版本降低到某个水平（可能在0.12.2和示例的锁定版本之间）。

此外，长生不老药的版本可能在这里起作用，请检查code you are referring

问候！

玩得开心

编辑：

您可以使用DBConnection.ConnectionPool代替poolboy，并使用最新的postgrex和Elixir版本，虽然不确定性能差异，但是可以比较，只需执行以下操作即可：

在config/config.exs上（检查是否需要密码等。）

config :fast_io_with_postgrex, :postgrex,
  database: "fp",
  name: :pg,
  pool: DBConnection.ConnectionPool,
  pool_size: 4

然后在lib/fast_io_with.....ex中将Postgrex.start_link(...的两行都替换为：

{:ok, pid} = Application.get_env(:fast_io_with_postgrex, :postgrex)
          |> Postgrex.start_link

那给了我

mix run -e 'FastIoWithPostgrex.import("./data_with_ids.txt")'
1.76s user 0.69s system 106% cpu 2.294 total

在Postgrex 0.14.1和Elixir 1.7.3上

Answer 2

谢谢您，根据您的建议，我通过降级mix.exs文件中的依赖项版本并将依赖项添加到较早版本的db_connection来工作，从而得到了原始示例：

   # Run "mix help deps" to learn about dependencies.
   defp deps do
     [
       # {:dep_from_hexpm, "~> 0.3.0"},
       # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"},

      {:postgrex, "0.13.5"},
      {:db_connection, "1.1.3"},
      {:poolboy, "~>1.5.1"}
 ]
 end

我还将尝试建议您更改代码，以在更高版本的db_connection中用新的池管理器替换Poolboy，以查看其是否同样有效。

我敢肯定，架构更改已引起很多思考，但是就Poolboy为何如此受欢迎而言，我不得不说几乎没有，但是在最新版本的db_connection中，甚至没有支持作为连接类型。

Windows上带有poolboy的Elixir postgrex示例由于“模块DBConnection.Poolboy不可用”而失败

2 个答案: