火花检索20多条记录

时间：2015-07-17 14:54:19

标签： scala apache-spark-sql

我有一个名为students.json的文件，如下所示：

{"uid":1,"name":"Michael","sid":1}
{"uid":2,"name":"Andy", "age":30,"sid":1}
{"uid":3,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":4,"name":"Andasxsay", "age":30,"sid":2}
{"uid":5,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":6,"name":"Anwereddy", "age":30,"sid":3}
{"uid":7,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":8,"name":"Andwedewy", "age":30,"sid":1}
{"uid":9,"name":"Justidedwn", "age":19,"sid":1}
{"uid":10,"name":"Anddwdey", "age":30,"sid":3}
{"uid":11,"name":"Michael","sid":1}
{"uid":12,"name":"Andy", "age":30,"sid":1}
{"uid":13,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":14,"name":"Andasxsay", "age":30,"sid":2}
{"uid":15,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":16,"name":"Anwereddy", "age":30,"sid":3}
{"uid":17,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":18,"name":"Andwe2fr3fdewy", "age":30,"sid":1}
{"uid":19,"name":"Justide4y45y54dwn", "age":19,"sid":1}
{"uid":20,"name":"Anddwd45y45yey", "age":30,"sid":3}
{"uid":21,"name":"Justdw45y45yedin", "age":19,"sid":1}
{"uid":22,"name":"An45y45ydy", "age":30,"sid":1}
{"uid":23,"name":"Jsaxsus4y54ytin", "age":19,"sid":1}
{"uid":24,"name":"Andas45y4y5xsay", "age":30,"sid":2}
{"uid":25,"name":"Jusewe4y5dtin", "age":19,"sid":1}
{"uid":26,"name":"Anwere45y45yddy", "age":30,"sid":3}
{"uid":27,"name":"Justdwe4y4y5dwein", "age":19,"sid":2}
{"uid":28,"name":"Andwede45ywy", "age":30,"sid":1}
{"uid":29,"name":"Justided45y45wn", "age":19,"sid":1}
{"uid":30,"name":"Anddwde4t4y", "age":30,"sid":3}
{"uid":31,"name":"Mich4y554ael","sid":1}
{"uid":32,"name":"An45ydy", "age":30,"sid":1}
{"uid":33,"name":"Jsaxsudfsstin", "age":19,"sid":1}
{"uid":34,"name":"Andasxssdfdsay", "age":30,"sid":2}
{"uid":35,"name":"Jusewedtsdfdsin", "age":19,"sid":1}
{"uid":36,"name":"Anweredsfdsdy", "age":30,"sid":3}
{"uid":37,"name":"Justdwedfsdwein", "age":19,"sid":2}
{"uid":38,"name":"Andwedewy", "age":30,"sid":1}
{"uid":39,"name":"Jdsfdsfustidedwn", "age":19,"sid":1}
{"uid":40,"name":"Ansdfdsey", "age":30,"sid":3}
{"uid":41,"name":"Jussdsdtdwedin", "age":19,"sid":1}

然后我使用sparks导入数据并使用sqlContext对其进行sql查询。要检索我正在使用以下scala代码的信息（几乎从scala上的spark示例中取出）：

val sqlContext = new org.apache.spark.sql.SQLContext(sc);
import sqlContext.implicits._
val students = sqlContext.read.json("../myTests/students.json");
students.registerTempTable("students");
sqlContext.sql("SELECT * FROM students").show;

这不是返回列表中的所有项目，而是返回20条记录（前20条记录）... 如果在最后一行而不是

sqlContext.sql("SELECT * FROM students").show;

我把

sqlContext.sql("SELECT * FROM students where uid>10").show;
它从10到30给了我20条记录。 总之，我希望看到超过20条记录，我该怎么做？可能吗？我查看了文档并且没有阅读任何有关此内容的内容....我知道它可能有点傻但我在进行更严肃的事情之前正在做一些测试...我下载了火花并且我正在运行它只是为了看scala而独立......

我的火花初始化显示了这一点（仅错误，我更改了配置以便它不会显示信息）

15/07/17 11:37:55 ERROR Shell: Failed to locate the winutils binary in the hadoo
p binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha
doop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
        at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)

        at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group
s.java:280)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI
nformation.java:271)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use
rGroupInformation.java:248)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(
UserGroupInformation.java:763)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou
pInformation.java:748)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr
oupInformation.java:621)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
        at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:212)
        at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
        at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkIL
oop.scala:187)
        at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:2
17)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:949)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClass
Loader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pr
ocess(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSub
mit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:16
9)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.

scala> studentsstudentsstudentsstudents

由于

2 个答案:

答案 0 :(得分：6)

假设你想看到100行，你也可以使用，

df.show(100)

sqlContext.sql("SELECT * FROM students where uid>10").show(100)

链接到method

答案 1 :(得分：5)

sqlContext.sql（“SELECT * FROM students”）返回一个DataFrame实例，请参阅DataFrame API。要检索不同的行数，可以调用“.take（n）”：

sqlContext.sql("SELECT * FROM students where uid>10").take(30).foreach(println)
相关问题
最新问题