Question

我是elixir的新手。我正在学习elixir bcz它是并发和快速的。我在MongoDB中拥有大量数据。我想将一些数据导出为CSV或逗号分隔文件。

我在elixir中使用以下MongoDB驱动程序。 https://github.com/ankhers/mongodb

我创建了新的混合应用程序起诉mix new mongo_elixir应用程序。在里面我添加了以下代码。

def export() do
    import Supervisor.Spec

    children = [
      worker(Mongo, [[name: :mongo, database: "test_table", pool: DBConnection.Poolboy]])
    ]

    opts = [strategy: :one_for_one, name: MongodbExport.Supervisor]
    Supervisor.start_link(children, opts)

    file = File.stream!("a.txt")
    cursor = Mongo.find(:mongo, "ssd", %{}, limit: 100000, pool: DBConnection.Poolboy)
    cursor
    |> Enum.map(fn record -> Map.get(record, "status") end)
    |> Enum.into(file)


  end

如果我使用这个命令运行上面的代码`time mix run -e'MongodbExport.export（）'它的输出是

Compiling 1 file (.ex)

real    0m8.134s
user    0m7.160s
sys 0m2.718s

需要8秒才能获得100_000并写入文本文件。这只是简单的.txt文件。

但如果我使用以下csv模块 https://github.com/beatrichartz/csv

并将上述数据写入文件，大约需要24秒。意味着3倍的时间然后是.txt文件。我不知道为什么。

我可以使用elixir和mongoDb在5秒到8秒内创建100_000条记录的CSV文件吗？是的，如果我只是想用逗号分隔的行创建.txt文件，从我从mongo查询获得的数据中选择列。我正在记录2条记录。

  [%{
"_id" => #BSON.ObjectId<59df8a37a971f13e2caea76e>,
"business" => #BSON.ObjectId<58e362130368e8f94f503f48>,
"causeAggregator" => "SERVICE_DOWN",
"causeId" => "3425156827238253899",
"corporate" => nil,
"createdAt" => 1507822135626.0,
"loopedTimes" => 0,
"optin" => false,
"phoneNumber" => "9876543210",
"scheduledAt" => nil,
"senderId" => nil,
"sentAt" => 1507822139920.0,
"sentSms" => #BSON.ObjectId<59df8a37a971f13e2caea4ac>,
"smsCreditsUsed" => 0,
"smsRef" => #BSON.ObjectId<59df8a37a971f13e2caea4c5>,
"status" => "failed",
"statusAggregator" => "fail",
"transactionId" => nil,
"updatedAt" => 1507836544930.0
},
  %{
    "_id" => #BSON.ObjectId<59df8a37a971f13e2caea76f>,
    "business" => #BSON.ObjectId<58e362130368e8f94f503f48>,
    "causeAggregator" => "SUCCESS",
    "causeId" => "3425156827238253899",
    "corporate" => nil,
    "createdAt" => 1507822135626.0,
    "dndStatusExtracted" => true,
    "loopedTimes" => 0,
    "optin" => false,
    "phoneNumber" => "9876543210",
    "scheduledAt" => nil,
    "senderId" => nil,
    "sentAt" => 1507822139920.0,
    "sentSms" => #BSON.ObjectId<59df8a37a971f13e2caea4ac>,
    "smsCreditsUsed" => 3,
    "smsRef" => #BSON.ObjectId<59df8a37a971f13e2caea4c6>,
    "status" => "completed",
    "statusAggregator" => "success",
    "transactionId" => nil,
    "updatedAt" => 1510627097023.0
  }]

`

Elixir MongoDB将数据导出到文件

0 个答案: