Question

我在这里关注了博客（下面提到的）并下载了包裹并根据需要放置。如果安装了任何人和步骤，请告诉我。

foo.stats

/opt/cloudera/csd/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658-el5.parcel

但服务cloudera-scm-server restart没有执行。要使用Cloudera Express（免费），请运行：

(https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html)

这需要至少8 GB的RAM和至少2个虚拟CPU。

Answer 1

尝试此命令

sudo /home/cloudera/cloudera-manager --express --force

Answer 2

SPARK 2.2 Installation Setup on Cloudera VM

Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids. 


Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.


Step 3: Please open the terminal and switch to root user as:
         su root
         password: cloudera

Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

(b). Switch to /usr/java/ directory with “cd /usr/java/” command.

(c). cp the java download tar file to the /usr/java/ directory.

(d). Untar the directory with “tar –zxvf jdk-8u31-linux-x64.tar.gz”

(e). Open the profile file with the command “vi ~/.bash_profile” 

(f). export JAVA_HOME to the new java directory.
       “export JAVA_HOME=/usr/java/jdk1.8.0_131”

       Save and Exit.


(g). In order to reflect the above change, following command needs to be executed on the shell:
       source ~/.bash_profile

Step 5:  The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.

(a). Switch to /opt/  directory with the command:
“cd /opt/”

(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

(c). Untar the spark tar with the following command:
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz

(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
Save and exit

(e).    We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home : 
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/

(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive

(g). Try “spark-shell”, it should work.

Answer 3

请按照下面的视频操作，该视频具有在Clouedra VM中安装Sprak2所需的所有必要步骤。

youtubue链接-https://www.youtube.com/watch?v=lQxlO3coMxM

另外，要启动Cloudera Express（免费），您的VM应该至少分配8GB RAM，或者如果您分配了默认的4GB RAM，则可以强制完全启动以下命令，然后按照上述视频进行操作。

sudo /home/cloudera/cloudera-manager --force --express

Answer 4

我放弃了这一点，对于宗地和非宗地安装都无法正常工作。

cloudera express一经启动，便会出现许多错误，并且Java 7取代了Java 8。

我安装了带有Spark 2.x的Mapr VM。没有问题。第一次工作。

那很好。这是我的建议1。

如果您想要KUDU，那么我将安装centos并自行安装。这是第2条建议。好的，您可能会错过Impala，但是如果仅出于研发目的，那么问题就不多了。

Answer 5

通过以下两个命令，我的spark2.2自动更新为spark 2.4：

（i）sudo yum更新

可能是您的Java主目录路径被拧紧了，在这种情况下，请将Java主目录路径导出到bash文件中。

（a）vi〜/ .bash_profile （b）（c）来源〜/ .bash_profile

Answer 6

只需下载正确的Spark版本，您需要说“ spark-2.2.0-bin-hadoop2.6”

通过vi编辑器打开bashrc_profile vi〜/ .bash_profile。粘贴以下两行

SPARK_HOME = / home / cloudera / Downloads / spark-2.2.0-bin-hadoop2.6 PATH = $ PATH：$ HOME / bin：$ SPARK_HOME / bin

保存然后运行命令：source〜/ .bash_profile

现在启动spark-shell。注意：确保已安装JDK 1.8

Answer 7

  SnPARK 2.2 Installation Setup on Cloudera VM

    Step 1: Download a quickstart_vm from the link:
    Prefer a vmware platform as it is easy to use, anyways all the options are viable.
    Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids. 


    Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.


    Step 3: Please open the terminal and switch to root user as:
             su root
             password: cloudera

    Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
    (a). Downloading Java:
    wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz

    (b). Switch to /usr/java/ directory with “cd /usr/java/” command.

    (c). cp the java download tar file to the /usr/java/ directory.

    (d). Untar the directory with “tar –xvzf jdk-8u31-linux-x64.tar.gz”

    (e). Open the profile file with the command “vi ~/.bash_profile” 

    (f). export JAVA_HOME to the new java directory.
           “export JAVA_HOME=/usr/java/jdk1.8.0_131”

           Save and Exit.


    (g). In order to reflect the above change, following command needs to be executed on the shell:
           source ~/.bash_profile

    Step 5:  The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.

    (a). Switch to /opt/  directory with the command:
    “cd /opt/”

    (b). Download spark with the command:
    wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

    (c). Untar the spark tar with the following command:
    tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz

    (d). We need to define some environment variables as default settings:
    Please open a file with the following command:
    vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
    Paste the following configurations in the file:
    SPARK_MASTER_IP=192.168.50.1
    SPARK_EXECUTOR_MEMORY=512m
    SPARK_DRIVER_MEMORY=512m
    SPARK_WORKER_MEMORY=512m
    SPARK_DAEMON_MEMORY=512m
    SPARK_LOCAL_IP=127.0.0.1
    Save and exit

    (e).    We need to start spark with the following command:
    /opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
    Export spark_home : 
    export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/

    (f). Change the permissions of the directory:
    chmod 777 -R /tmp/hive

    (g). Try “spark-shell”, it should work.

与 swapnil shashank 相同，但在下面进行了小修改

SPARK_LOCAL_IP=127.0.0.1
tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz

无法在Cloudera Quickstart VM（5.10）中安装spark 2.2

7 个答案: