spark - 未定义架构,并且未找到Parquet数据文件或摘要文件

时间:2015-08-24 04:53:23

标签: python jdbc apache-spark

首先我开始

  imageSlider = [[UIScrollView alloc]initWithFrame:CGRectMake(5, 60, 700, 700)];
    imageSlider.pagingEnabled = YES;
    imageSlider.showsHorizontalScrollIndicator =NO;

    for (int i = 0; i < [myObject count]; i++) {
        CGRect frame;
        frame.origin.x = imageSlider.frame.size.width * i;
        frame.origin.y = 0;
        frame.size = imageSlider.frame.size;
        NSManagedObject *con = [myObject objectAtIndex:i] ;
        UIImage *Image = [UIImage imageWithData:[con valueForKey:@"image"]];
        UIImageView *imgView = [[UIImageView alloc] initWithFrame:CGRectMake(10, 10, 700, 700)];
        imgView.image = Image;
        imgView.frame = frame;
        [imageSlider addSubview:imgView];
    }


     imageSlider.contentSize = CGSizeMake(imageSlider.frame.size.width*([myObject count]), 600);
     imageSlider.delegate = self;
     [self.view addSubview:imageSlider];

并编写此代码

Image<uint8_t> orig_uint = Tools::load_image("../foo.ppm");

Var x, y, c;
Func orig("orig"), orig_lum("orig_lum"), m45("m45"), m135("m135"), f45("f45"), f135("f135"), f4x4_horiz("f4x4_horiz"), f4x4("f4x4"), diag("diag");

Func orig_clamped = BoundaryConditions::repeat_edge(orig_uint);

const float wta = 1.0f, wtb = 3.0f, wt0 = wta * wta, wt1 = wta * wtb, wt2 = wtb * wtb;

orig(x, y, c) = cast<float_t>(orig_clamped(x, y, c));

orig_lum(x, y) = 0.299f * orig(x, y, 0) + 0.587f * orig(x, y, 1) + 0.114f * orig(x, y, 2);

m45(x, y) = abs(orig_lum(x - 1, y - 1) - orig_lum(x, y)) + abs(orig_lum(x, y) - orig_lum(x + 1, y + 1)) + abs(orig_lum(x + 1, y + 1) - orig_lum(x + 2, y + 2));

m135(x, y) = abs(orig_lum(x + 2, y - 1) - orig_lum(x + 1, y)) + abs(orig_lum(x + 1, y) - orig_lum(x, y + 1)) + abs(orig_lum(x, y + 1) - orig_lum(x - 1, y + 2));

f45(x, y, c) = wta * (orig(x - 1, y - 1, c) + orig(x + 2, y + 2, c)) + wtb * (orig(x, y, c) + orig(x + 1, y + 1, c));

f135(x, y, c) = wta * (orig(x - 1, y + 2, c) + orig(x + 2, y - 1, c)) + wtb * (orig(x, y + 1, c) + orig(x + 1, y, c));

f4x4_horiz(x, y, c) = wta * (orig(x - 1, y, c) + orig(x + 2, y, c)) + wtb * (orig(x, y, c) + orig(x + 1, y, c));

f4x4(x, y, c) = wta * (f4x4_horiz(x, y - 1, c) + f4x4_horiz(x, y + 2, c)) + wtb * (f4x4_horiz(x, y, c) + f4x4_horiz(x, y + 1, c));

diag(x, y, c) = select(m135(x, y) > m45(x, y), f45(x, y, c), select(m45(x, y) > m135(x, y), f135(x, y, c), f4x4(x, y, c)));

// schedule
orig_lum.compute_root();
m45.compute_root().bound(x, 0, orig_uint.width()).bound(y, 0, orig_uint.height());
m135.compute_root().bound(x, 0, orig_uint.width()).bound(y, 0, orig_uint.height());
f45.compute_at(diag, x);
f135.compute_at(diag, x);
f4x4.compute_at(diag, x);
diag.compute_root();

// compile so we can take a look at the code
diag.compile_to_lowered_stmt("diag.html", {}, HTML);    // stack oflo here

当我只写dbtable =&#34; test_db&#34;时,错误是相同的。

发生此错误后,

  

py4j.protocol.Py4JJavaError:调用o66.load时发生错误。   :java.lang.AssertionError:断言失败:未定义架构,并且未找到Parquet数据文件或摘要文件。   在scala.Predef $ .assert(Predef.scala:179)   在org.apache.spark.sql.parquet.ParquetRelation2 $ MetadataCache.org $ apache $ spark $ sql $ parquet $ ParquetRelation2 $ MetadataCache $$ readSchema(newParquet.scala:429)   .....

为什么会出现这个错误? 我想知道并解决这个问题。

谢谢。

1 个答案:

答案 0 :(得分:0)

我不知道这个错误的原因,但我偶然发现了它,然后找到了一种方法来使同样的事情发挥作用。

试试这个:

df = sqlContext.read.format("jdbc").options(url="jdbc:mysql://server/table?user=usr&password=secret", dbtable="table_name").load()

我认为.load语法不再有效,或者对jdbc不起作用。希望它有效!

顺便说一下,我用这个命令启动了控制台:

SPARK_CLASSPATH=~/progs/postgresql-9.4-1205.jdbc42.jar pyspark

我的数据库是postgres,因此我使用jdbc下载了jar,并按照文档中的建议将其添加到我的类路径中。 http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases