问题:
我一直在寻找有关如何正确设置程序以将参数传递给GUI应用程序对象ViewDF的信息。
我有一个名为:ViewDF的主对象,其中包含一个带有 参数列表。
如果args列表包含关键字--debug, 然后分配debugBoolean = true
如果debugBoolean = true 然后执行GUI处理。
在对象ViewDf的主要功能中: 我从类(UI)创建一个用户界面(ui)对象。
在类(UI)中,我处理源数据帧: 并从源数据帧创建rowData变量。 并从源数据帧创建coldata变量。
a。调试参数。 (false或true)
false - do not execute the GUI processing.
true - when: the debug option is true.
Then: 1.0 Perform the GUI processing.
1.1 Pass the debug option to the application.
1.2 Pass the DataFrame object to the application.
1.3 Pass the Spark Session object to the application.
1.4. create the rowData from the source Data Frame.
1.5 create the colData from the source Data Frame.
1.6 create the table object with the rowData and colData
as input to the table.
b。 Spark Session Object参数。 (spark = spark)
This parameter is specified inside my main program
where the viewDF object is created.
c。 Spark数据框对象名称。 (srcDF = DF)
This parameter is specified inside my main program
where the viewDF object is created.
1.0在运行我的主要SCALA程序并通过SpaRK-Submit调用时:================================ =================================
1.1在我的主要Scala程序中,我将调用ViewDF 创建火花数据框架(DF)之后。
EXAMPLE:
1. CREATE THE SPARK DATA FRAME.
var DF:DataFrame=spark.read
.format("com.databricks.spark.csv")
.option("header","true")
.option("inferschema","true")
.option("delimiter",",")
.load("FILE:///home/vneto/test.csv")
2. DISPLAY A DATA FRAME TO A GUI TABLE
ONLY WHEN ( debug=true && **--deploy-mode CLIENT** )
val viewRC = new ViewDF( debug=debug,spark=spark,srcDF=DF)
2.0以SPARK-SUBMIT模式运行时: ================================
2.1将关键字<--debug>传递给Main Scala程序 从Spark-Submit调用中。
This statement is placed after the application jar file.
When: --debug keyword is found in the args list,
Then: assign the variable debugBoolean = true.
2.2
See Item: 2.4
2.3应在主内部指定
See item: 2.4)
2.4 ViewDF对象是在创建数据框后创建的。
val viewRC = new ViewDF( debug=debug,spark=spark,srcDF=DF)
3.0在火花壳模式下运行时:
===============================
3.1导入SCALA模块。
import org.apache.spark.sql.DataFrame
import com.databricks.spark.csv
import com.company.viewDF
3.2创建读取CSV文件的空间数据帧。
var DF:DataFrame=spark.read
.format("com.databricks.spark.csv")
.option("header","true")
.option("inferschema","true")
.option("delimiter",",")
.load("FILE:///home/vneto/test.csv")
3.3在spark.read语句之后放置(viewDF)语句。
val viewDF = new ViewDF( debug=debug, spark=spark, srcDF=DF)
SCALA代码:
import org.apache.hadoop.fs.DF
import org.apache.spark.sql.{DataFrame, SparkSession}
import scala.swing._
//===================================================================
// OBJECT: ViewDF - View a scala spark Data Frame.
//===================================================================
object ViewDF {
// ================================================================
// function: main
// ================================================================
def main ( args : Array[String] ) {
// ------------------------------------------------------------
// step 01. INITIALIZE debug-Boolean VARIABLE AS: false.
// ------------------------------------------------------------
var debugBoolean = false
// ------------------------------------------------------------
// step 02. GET SPARK-SUBMIT COMMAND LINE OPTION: --debug
// ------------------------------------------------------------
for ( s <- args ) {
if ( s == "--debug" ) {
debugBoolean = true
}
}
// -------------------------------------------------------------
// step 03. CREATE TOP MAIN FRAME WINDOW OBJECT.
//
// 1. pass the debug option to the interface.
// 2. pass the spark session object to the interface.
// 3. pass the data frame object to the interface.
// -------------------------------------------------------------
val top = new UI (
debug = debugBoolean
)
// -------------------------------------------------------------
// step 04. DISPLAY TOP MAIN FRAME WINDOW to USER.
// -------------------------------------------------------------
top.visible = true
// -------------------------------------------------------------
// step 05. DISPLAY MESSAGE AT THE END OF THE MAIN FUNCTION.
// -------------------------------------------------------------
println("end of main function")
} // END OF: main
} // END OF: viewDF
//===================================================================
//
// Define class: UI - USER INTERFACE
//
// PURPOSE: 1. This UI class defines the User Interface.
//
// ------------------------------------------------------------------
//
// Attributes: 1. spark = Passes a Spark Session to the interface.
//
// Default: ???
//
// 2. debug = Passes a debug switch to the interface.
//
// Default: false - turn interface off.
// true - turn interface on.
//
// 3. srcDF = Passes a source data frame to the
// interface.
// Default: ???
//
// ===================================================================
class UI (
spark : SparkSession = ??? ,
srcDF : DataFrame = ??? ,
debug : Boolean = false
) extends MainFrame {
// -------------------------------------------------------------------
// step 200. if debug = true then display user interface.
// -------------------------------------------------------------------
if ( debug ) {
// ---------------------------------------------------------------
// s-201. ASSIGN A TITLE TO THE TOP WINDOW.
// ---------------------------------------------------------------
title = "GUI 1 Data Frame Viewer"
// ---------------------------------------------------------------
// s-202. ASSIGN THE PREFERRED SIZE OF THE TOP WINDOW.
// ---------------------------------------------------------------
preferredSize = new Dimension(400,400)
// ---------------------------------------------------------------
// s-203. ASSIGN SOURCE DATA FRAME TO A MUTABLE OUTPUT DATA FRAME.
// ---------------------------------------------------------------
var outDF = srcDF
// ---------------------------------------------------------------
// s-204. GET THE DATA FRAME SCHEMA FROM THE SOURCE DATA FRAME.
//
// attributes: name - dataType
// ---------------------------------------------------------------
val DFS = srcDF.schema
// ---------------------------------------------------------------
// s-205. Converts all elements of the outDF Data Frame
// into a string data type format
// looping over the columns of the data frame schema (DFS).
// ---------------------------------------------------------------
DFS.foreach( row => {
if ( row.dataType != "StringType" ) {
import spark.implicits._ ;
outDF = outDF.withColumn(
s"${row.name}",
$"${row.name}".cast("String")
);
} // end of outDF String Conversion.
println( row.name + " " + row.dataType + " " );
}) // end of DFS Schema Column Name For Loop.
// ----------------------------------------------------------------
// s-206. prints the schema of the output data frame
// to the log after the string conversion.
// ----------------------------------------------------------------
outDF.printSchema ;
// ----------------------------------------------------------------
// s-207. Converts the outDF Data Frame into an Array of Arrays.
//
// The rowData variable is passed to the scala swing table.
// ----------------------------------------------------------------
val rowData = outDF.collect().map( row => {
var a = Array[Any]() ;
for ( i <- 0 to row.size -1 ) { a = a :+ row.getString(i); };
a ;
}) // END OF CONVERSION OF THE DATA FRAME INTO AN ARRAY OF ARRAYS.
// ----------------------------------------------------------------
// s-208. CREATES THE HEADER COLUMN NAME SEQUENCE ARRAY.
// * list of column names.
// ----------------------------------------------------------------
val colData = srcDF.columns
// ----------------------------------------------------------------
// s-209. CREATES THE USER INTERFACE OBJECT( ui ).
//
// NOTE: 1. Creates a new Vertical Box Panel as:
// ui = User Interface Object.
//
// 2. Creates a new Table object
// with input rowData and colData objects.
//
// 3. Creates a new Scroll Pane with the table added.
// ----------------------------------------------------------------
lazy val ui = new BoxPanel (Orientation.Vertical) {
val table = new Table( rowData , colData ) ;
contents += new ScrollPane( table ) ;
} // End of User Interface - Box Panel object.
// ----------------------------------------------------------------
// s-201. LOADS THE USER INTERFACE OBJECT INTO THE CONTENTS OBJECT.
// ----------------------------------------------------------------
contents = ui
// ----------------------------------------------------------------
// s-211. Turn on visibility of the top window
// to the user on the client machine.
// ----------------------------------------------------------------
visible = true
} // END OF BLOCK: If debug == true Then Display User Interface.
} //课堂结束:用户界面