Question

我需要你的帮助来解决一个大问题。我必须索引来自EMC xDB源的一些数据。我将xDB配置为像RDBMS一样被请求。然后我配置Squirrel来请求xDB。它起作用，例如，当我尝试：

 Select * from books

我使用与用于squirrel相同的驱动程序和相同的连接字符串配置solr。

这是我的data-config.xml：

<dataConfig>
<dataSource type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>
<document name="books">
    <entity name="book" query="select * from books">
        <field column="author" name="author" />
        <field column="price" name="price" />
        <field column="name" name="name" />
        <field column="description" name="description" />
        <field column="title" name="title" />
        <field column="publish_date" name="publish_date" />         
    </entity>
</document>
</dataConfig>

solrconfig.xml中

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
    <lst name="defaults">
        <str name="config">C:\solr\solr-5.2.1\server\solr\jcg\conf\data-config.xml</str>
    </lst>
</requestHandler>

我的xDB BOOKS-MetaData.xml：

<?xml version="1.0" encoding="UTF-16"?>
<TABLEMETADATA TABLENAME="books">
  <RECORD-COUNT>2</RECORD-COUNT>
  <ROOT-ELEMENT>books</ROOT-ELEMENT>
  <ROW-ELEMENT>book</ROW-ELEMENT>
  <DATATYPES>
     <BOOKS-TYPE>
      <NAME>author</NAME>
      <COLNO>1</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>title</NAME>
      <COLNO>2</COLNO>
      <DATATYPE>VARCHAR2(75)</DATATYPE>
    </BOOKS-TYPE>
    <BOOKS-TYPE>
      <NAME>genre</NAME>
      <COLNO>3</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>price</NAME>
      <COLNO>4</COLNO>
      <DATATYPE>VARCHAR2(25)</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>publish_date</NAME>
      <COLNO>5</COLNO>
      <DATATYPE>DATETIME</DATATYPE>
   </BOOKS-TYPE>
   <BOOKS-TYPE>
      <NAME>description</NAME>
      <COLNO>6</COLNO>
      <DATATYPE>VARCHAR2(350)</DATATYPE>
    </BOOKS-TYPE>
  </DATATYPES>
</TABLEMETADATA>

solr响应的一部分：

"verbose-output": [
"entity:book",
[
  "document#1",
  [
    "query",
    "select * from books",
    "time-taken",
    "0:0:0.114",
    "EXCEPTION",
    "org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1

当我检查solr日志时，我有3个例外。 1号：

getNext() failed for query 'select * from books':org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

2号：

Exception while processing: book document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

最后一个：

Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)
at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:185)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
... 29 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:70)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:398)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:296)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:336)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(JdbcDataSource.java:328)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 31 more
Caused by: java.sql.SQLFeatureNotSupportedException: Method com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject-columnLabel is not yet implemented.
at com.emc.ia.xdbjdbc.Driver.notImplemented(Driver.java:745)
at com.emc.ia.xdbjdbc.xdb.XdbResultSet.getObject(XdbResultSet.java:659)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(JdbcDataSource.java:358)
... 39 more

提前感谢您的帮助，因为我真的不明白为什么它适用于松鼠而不适用于solr。

问题编辑以回复@Cheffe

由于getObject（）方法，它确实是一个xDB驱动程序问题。我查看了松鼠代码并且它有效，因为它使用了列的指示而不是它的名称。

int idx = _columnIndices != null ? _columnIndices[i] : i + 1;
        try
        {
            final int columnType = _rsmd.getColumnType(idx);
            //final String columnClassName = _rsmd.getColumnClassName(idx);
            switch (columnType)
            {
                case Types.NULL:
                    row[i] = null;
                    break;

                // TODO: When JDK1.4 is the earliest JDK supported
                // by Squirrel then remove the hardcoding of the
                // boolean data type.
                case Types.BIT:
                case 16:

// case Types.BOOLEAN： row [i] = _rs.getObject（idx）; 。。

无论如何它适用于

convertType="true"

Answer 1

检查Solr源后：看来该数据库的驱动程序可能不适合与Solr的DIH一起运行。不幸的是，数据库是专有的，因此我无法尝试。

从供应商的驱动程序中抛出异常。所有三个根本原因都在那里居住。当您向下滚动到堆栈跟踪的最底部时，您可以看到。所有这些堆栈跟踪都告诉你

方法xxx.getObject-columnLabel尚未实现。

这意味着：您的数据库供应商EMC未在其JDBC驱动程序中实现此方法。如果您不想亲自动手，则应向他们提交您需要实施此功能的错误报告或功能请求。

你可能有的一点是，堆栈跟踪中指出Solr源的最后一行导致the internal getARow method

private Map<String, Object> getARow() {
  if (resultSet == null)
    return null;

  Map<String, Object> result = new HashMap<>();
  for (String colName : colNames) {
    try {
      if (!convertType) {
      Object value = resultSet.getObject(colName); // <-- here the exception takes of
      if (value instanceof BigDecimal || value instanceof BigInteger) {
        result.put(colName, value.toString());
      } else {
        result.put(colName, value);
      }
      continue;
    }
// more code there, but left out

您可以通过设置dataimporthandler的配置参数来绕过resultSet.getObject的调用。即这是

convertType （默认值：false） - 将数据库返回的字段类型的其他转换应用于schema.xml中定义的字段类型。默认值似乎更安全，因为它不会导致额外的神奇转换。但是，在特殊情况下（例如BLOB字段），转换是解决问题的方法之一。 [Reference]

您需要在 data-config.xml

中执行此操作

<dataConfig>
<dataSource convertType="true" type="JdbcDataSource" driver="com.emc.ia.xdbjdbc.Driver" url="jdbc:xdbeas://localhost:1235/MaDatabase" user="*****" password="*****"/>

但是，我无法告诉您这是否与您的数据库及其驱动程序成功。可能这只会导致另一个例外。在这种情况下，您需要回退到解决方案

要求供应商这样做
编写自定义数据导入例程

Solr导入命令不起作用

1 个答案: