Question

尝试使用ADAM和Zeppelin进行基因组分析。我不确定我是否正确行事，但遇到了以下问题。

%dep
z.reset()
z.addRepo("Spark Packages Repo").url("http://dl.bintray.com/spark-packages/maven")
z.load("com.databricks:spark-csv_2.10:1.2.0")   
z.load("mysql:mysql-connector-java:5.1.35")  
z.load("org.bdgenomics.adam:adam-core_2.10:0.20.0")
z.load("org.bdgenomics.adam:adam-cli_2.10:0.20.0")
z.load("org.bdgenomics.adam:adam-apis_2.10:0.20.0")

%spark

import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.projections.{ AlignmentRecordField, Projection }
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.projections.Projection
import org.bdgenomics.adam.projections.AlignmentRecordField
import scala.io.Source
import org.apache.spark.rdd.RDD
import org.bdgenomics.formats.avro.Genotype
import scala.collection.JavaConverters._
import org.bdgenomics.formats.avro._
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.{ Vector => MLVector, Vectors }
import org.apache.spark.mllib.clustering.{ KMeans, KMeansModel }

val ac = new ADAMContext(sc)

我得到以下输出错误

import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.projections.{AlignmentRecordField, Projection}
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.bdgenomics.adam.rdd.ADAMContext
import org.bdgenomics.adam.rdd.ADAMContext._
import org.bdgenomics.adam.projections.Projection
import org.bdgenomics.adam.projections.AlignmentRecordField
import scala.io.Source
import org.apache.spark.rdd.RDD
import org.bdgenomics.formats.avro.Genotype
import scala.collection.JavaConverters._
import org.bdgenomics.formats.avro._
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.{Vector=>MLVector, Vectors}
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
res7: org.apache.spark.SparkContext = org.apache.spark.SparkContext@62ec8142
<console>:188: error: constructor ADAMContext in class ADAMContext cannot be accessed in class $iwC
              new ADAMContext(sc)

知道在哪里看？我错过了任何依赖吗？ ^

Answer 1

根据您使用的版本中的文件ADAMContext.scala。构造函数是私有的。

class ADAMContext private (@transient val sc: SparkContext) 
    extends Serializable with Logging {
    ...
}

你可以这样使用。

import org.bdgenomics.adam.rdd.ADAMContext._

val adamContext: ADAMContext = z.sc

它将使用对象ADAMContext

中的隐式转换

object ADAMContext {
    implicit def sparkContextToADAMContext(sc: SparkContext): ADAMContext = 
        new ADAMContext(sc)
}

Answer 2

没有使用Z参考确实有效!!

val ac:ADAMContext  = sc
val genotypes: RDD[Genotype] = ac.loadGenotypes("/tmp/ADAM2").rdd

输出

ac: org.bdgenomics.adam.rdd.ADAMContext = org.bdgenomics.adam.rdd.ADAMContext@2c60ef7e

genotypes: 
org.apache.spark.rdd.RDD[org.bdgenomics.formats.avro.Genotype] = MapPartitionsRDD[3] at map at ADAMContext.scala:207

我曾尝试在adam-shell提示符下执行此操作，但我不记得必须使用隐式转换。它使用的是0.19版本的ADAM。

Spark，ADAM和Zeppelin

2 个答案: