Question

我有一个Spark应用程序（版本2.1）将数据写入Ignite服务器缓存（版本2.2）。这是我用来从Spark作业中的IgniteContext创建缓存的代码：

object Spark_Streaming_Processing {

 case class Custom_Class(
                  @(QuerySqlField @field)(index = true) a: String,
                  @(QuerySqlField @field)(index = true) b: String,
                  @(QuerySqlField @field)(index = true) c: String,
                  @(QuerySqlField @field)(index = true) d: String,
                  @(QuerySqlField @field)(index = true) e: String,
                  @(QuerySqlField @field)(index = true) f: String,
                  @(QuerySqlField @field)(index = true) g: String,
                  @(QuerySqlField @field)(index = true) h: String

                )

//START IGNITE CONTEXT

val addresses=new util.ArrayList[String]()
addresses.add("127.0.0.1:48500..48520")

val igniteContext:IgniteContext=new IgniteContext(sqlContext.sparkContext,()=>
new IgniteConfiguration().setDiscoverySpi(new TcpDiscoverySpi().setIpFinder(new TcpDiscoveryVmIpFinder().setAddresses(addresses))
  ).setCacheConfiguration(new CacheConfiguration[String,Custom_Class]()
  .setName("Spark_Ignite").setBackups(1).setIndexedTypes(classOf[String],classOf[Custom_Class]))
,true)


println(igniteContext.ignite().cacheNames())

val ignite_cache_rdd:IgniteRDD[String,Custom_Class] =igniteContext.fromCache[String,Custom_Class]("Spark_Ignite")

val processed_Pair:RDD[(String,Custom_Class)]=(...)// rdd with data, which as you can see has the correct datatypes as parameters

ignite_cache_rdd.savePairs(processed_PairRDD)

}
  }

一切正常，但昨天我决定销毁Spark_Ignite缓存并重启Ignite服务器。但是，再次运行Spark应用程序，我现在收到以下错误

javax.cache.CacheException: Failed to find data nodes for cache: Spark_Ignite
    at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.stableDataNodes(GridReduceQueryExecutor.java:447)
    at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:591)
    at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1160)

如果我转到Ignite遮阳板，我可以看到缓存是在Ignite服务器端创建的，我可以看到Ignite服务器检测到Spark应用程序的Ignite客户端：

 [12:17:01] Topology snapshot [ver=4, servers=1, clients=1, CPUs=12, heap=1.9GB]

但是，Spark应用程序中的Ignite客户端似乎无法在启动时检测到服务器节点，即使已创建缓存：

18/04/13 12:17:01 INFO GridDiscoveryManager: Topology snapshot [ver=4, servers=0, clients=1, CPUs=12, heap=0.89GB]
18/04/13 12:17:07 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 2)
javax.cache.CacheException: Failed to find data nodes for cache: Spark_Ignite

我应该补充一点，如果Ignite服务器未启动，Ignite客户端将无法启动，它会在给定的地址中查找并挂起。当我连接服务器时，会创建缓存，然后我才会收到此错误。

这里的问题可能是什么？它之前是如何运作的呢？

谢谢。

更新

以下是我用于单个Ignite服务器的xml配置文件：

<?xml version="1.0" encoding="UTF-8"?>

 <beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:util="http://www.springframework.org/schema/util"
   xsi:schemaLocation="
    http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
    http://www.springframework.org/schema/util
    http://www.springframework.org/schema/util/spring-util-2.0.xsd">

<bean id="ignite_cluster.cfg" 
 class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="igniteInstanceName" value="ignite_node1" />
    <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
     <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                    <!-- Initial local port to listen to. -->
                    <property name="localPort" value="48510"/>
                    <!-- Changing local port range. This is an optional action. -->
                    <property name="localPortRange" value="20"/>
                    <!-- Setting up IP finder for this cluster -->
                    <property name="ipFinder">
                            <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                                    <property name="addresses">
                                            <list>
                                                    <!-- Addresses and port range of the nodes from the first cluster.
          127.0.0.1 can be replaced with actual IP addresses or host names.
          Port range is optional. -->

           <value>127.0.0.1:48500..48520</value>
                                            </list>
                                    </property>
                            </bean>
                    </property>
            </bean>
    </property>
    <property name="communicationSpi">
            <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
                    <property name="localPort" value="48100"/>
            </bean>
    </property>
</bean>
 <bean class="org.apache.ignite.configuration.FileSystemConfiguration">
    <property name="secondaryFileSystem">
 <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
  <property name="fileSystemFactory">
    <bean class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
      <property name="uri" value="hdfs://localhost:9000/"/>
      </bean>
     </property>
    </bean>
  </property>
</bean>
<bean class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
 <property name="uri" value="hdfs://localhost:9000/"/>
 <property name="configPaths">
    <list>
      <value>/etc/hadoop-2.9.0/etc/hadoop/core-site.xml</value>
    </list>
  </property>
 </bean>

 </beans>

Answer 1

我遇到了类似的问题，错误消息完全相同：

javax.cache.CacheException: Failed to find data nodes for cache: <cache name>
    at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.stableDataNodes(GridReduceQueryExecutor.java:447)
    at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:591)
    at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1160)

做了以下事情：

通过以下方式检查了 ignite 拓扑：./control.sh --baseline
如果有任何处于离线状态的基线节点，要么启动该基线节点，要么通过 ./control.sh --baseline remove <ConsistentID> 将其删除

重新启动您的客户端应用程序

在Apache Ignite

1 个答案: