Question

我正在尝试将一个csv文件导入到orientdb 3.0中，该文件已创建并测试了json文件，并且它可用于较小的数据集。但是我要导入的数据集大约有十亿行（六列）

以下是我用于通过oetl导入的user.json文件

{
    "source": { "file": { "path": "d1.csv" } },
    "extractor": { "csv": {} },
    "transformers": [
      { "vertex": { "class": "User" } }
    ],
    "loader": {
      "orientdb": {
         "dbURL": "plocal:/databases/magriwebdoc",
         "dbType": "graph",
         "classes": [
           {"name": "User", "extends": "V"}
         ], "indexes": [
           {"class":"User", "fields":["id:string"], "type":"UNIQUE" }
         ]
      }
    }
  }

这是oetl命令的控制台输出：

2019-05-22 14:31:15:484 INFO  Windows OS is detected, 262144 limit of open files will be set for the disk cache. [ONative]
2019-05-22 14:31:15:647 INFO  8261029888 B/7878 MB/7 GB of physical memory were detected on machine [ONative]
2019-05-22 14:31:15:647 INFO  Detected memory limit for current process is 8261029888 B/7878 MB/7 GB [ONative]
2019-05-22 14:31:15:649 INFO  JVM can use maximum 455MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:649 INFO  Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:650 INFO  OrientDB auto-config DISKCACHE=6,477MB (heap=455MB os=7,878MB) [orientechnologies]
2019-05-22 14:31:15:652 INFO  System is started under an effective user : `lenovo` [OEngineLocalPaginated]
2019-05-22 14:31:15:670 INFO  WAL maximum segment size is set to 6,144 MB [OrientDBEmbedded]
2019-05-22 14:31:15:701 INFO  BEGIN ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:15:703 INFO  [file] Reading from file d1.csv with encoding UTF-8 [OETLFileSource]
2019-05-22 14:31:15:703 INFO  Started execution with 1 worker threads [OETLProcessor]
2019-05-22 14:31:16:008 INFO  Page size for WAL located in D:\databases\magriwebdoc is set to 4096 bytes. [OCASDiskWriteAheadLog]
2019-05-22 14:31:16:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 1001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:16:770 INFO  Storage 'plocal:D:\databases/magriwebdoc' is opened under OrientDB distribution : 3.0.18 - Veloce (build 747595e790a081371496f3bb9c57cec395644d82, branch 3.0.x) [OLocalPaginatedStorage]
2019-05-22 14:31:17:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:17:954 SEVER ETL process has problem:  [OETLProcessor]
2019-05-22 14:31:17:956 INFO  END ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:17:957 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2255ms [0 warnings, 0 errors] [OETLProcessor]D:\orientserver\bin>

我知道代码是正确的，但是我认为这更多是内存问题！

请告知我该怎么办。

Answer 1

您是否尝试过根据要处理的数据大小来改进内存设置？

从文档中，您可以自定义以下属性：

Configuration Environmental Variables（请参见$ORIENTDB_OPTS_MEMORY参数）
Performance-Tuning - Memory Settings

也许可以帮助您

Answer 2

您的json脚本似乎没问题，但是您可以尝试删除索引部分。由于索引错误，我也遇到了同样的问题。可能是因为UNIQUE索引约束。您可以尝试：

删除json脚本的索引部分。
如果需要此索引，请确保在导入数据集之前清除数据库。

为什么OrientDB OETL导入给我这个错误

2 个答案: