我正在使用pyspark（使用Spark DataFrame PCA库）使用spark模型降低ml的维度，如下所示：
pca = PCA(k=3, inputCol="features", outputCol="pca_features")
model = pca.fit(data)
其中data为Spark DataFrame，其中一列为features，其中DenseVector为3维：
data.take(1)
Row(features=DenseVector([0.4536,-0.43218, 0.9876]), label=u'class1')
拟合后，我转换数据：
transformed = model.transform(data)
transformed.first()
Row(features=DenseVector([0.4536,-0.43218, 0.9876]), label=u'class1', pca_features=DenseVector([-0.33256, 0.8668, 0.625]))
我的问题是：如何提取此PCA的特征向量？如何计算他们解释的方差？
答案 0 :(得分：28)
[更新：从Spark 2.2开始，PCA和SVD都在PySpark中可用 - 请参阅JIRA票证SPARK-6227和PCA＆amp; Spark ML 2.2的PCAModel;以下原始答案仍适用于较旧的Spark版本。]
嗯，这似乎令人难以置信，但实际上没有办法从PCA分解中提取这些信息（至少从Spark 1.5开始）。但同样，有许多类似的“投诉” - 例如，请参阅here，因为无法从CrossValidatorModel中提取最佳参数。
幸运的是，几个月前，我参加了AMPLab（伯克利）的'Scalable Machine Learning' MOOC＆amp; Databricks，即Spark的创建者，我们在那里实施了一个完整的PCA管道“手工”作为家庭作业的一部分。我已经从后面修改了我的函数（请放心，我得到了充分的信任:-)，以便使用与您的格式相同的数据框作为输入（而不是RDD）（即包含DenseVectors的行数字特征）。
我们首先需要定义一个中间函数estimatedCovariance，如下所示：
import numpy as np
def estimateCovariance(df):
"""Compute the covariance matrix for a given dataframe.
Note:
The multi-dimensional covariance array should be calculated using outer products. Don't
forget to normalize the data by first subtracting the mean.
Args:
df: A Spark dataframe with a column named 'features', which (column) consists of DenseVectors.
Returns:
np.ndarray: A multi-dimensional array where the number of rows and columns both equal the
length of the arrays in the input dataframe.
"""
m = df.select(df['features']).map(lambda x: x[0]).mean()
dfZeroMean = df.select(df['features']).map(lambda x: x[0]).map(lambda x: x-m) # subtract the mean
return dfZeroMean.map(lambda x: np.outer(x,x)).sum()/df.count()
然后，我们可以编写一个主pca函数，如下所示：
from numpy.linalg import eigh
def pca(df, k=2):
"""Computes the top `k` principal components, corresponding scores, and all eigenvalues.
Note:
All eigenvalues should be returned in sorted order (largest to smallest). `eigh` returns
each eigenvectors as a column. This function should also return eigenvectors as columns.
Args:
df: A Spark dataframe with a 'features' column, which (column) consists of DenseVectors.
k (int): The number of principal components to return.
Returns:
tuple of (np.ndarray, RDD of np.ndarray, np.ndarray): A tuple of (eigenvectors, `RDD` of
scores, eigenvalues). Eigenvectors is a multi-dimensional array where the number of
rows equals the length of the arrays in the input `RDD` and the number of columns equals
`k`. The `RDD` of scores has the same number of rows as `data` and consists of arrays
of length `k`. Eigenvalues is an array of length d (the number of features).
"""
cov = estimateCovariance(df)
col = cov.shape[1]
eigVals, eigVecs = eigh(cov)
inds = np.argsort(eigVals)
eigVecs = eigVecs.T[inds[-1:-(col+1):-1]]
components = eigVecs[0:k]
eigVals = eigVals[inds[-1:-(col+1):-1]] # sort eigenvals
score = df.select(df['features']).map(lambda x: x[0]).map(lambda x: np.dot(x, components.T) )
# Return the `k` principal components, `k` scores, and all eigenvalues
return components.T, score, eigVals
<强>测试
让我们首先使用现有方法查看结果，使用Spark ML PCA documentation中的示例数据（将其修改为全部DenseVectors）：
from pyspark.ml.feature import *
from pyspark.mllib.linalg import Vectors
data = [(Vectors.dense([0.0, 1.0, 0.0, 7.0, 0.0]),),
(Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
(Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
df = sqlContext.createDataFrame(data,["features"])
pca_extracted = PCA(k=2, inputCol="features", outputCol="pca_features")
model = pca_extracted.fit(df)
model.transform(df).collect()
[Row(features=DenseVector([0.0, 1.0, 0.0, 7.0, 0.0]), pca_features=DenseVector([1.6486, -4.0133])),
Row(features=DenseVector([2.0, 0.0, 3.0, 4.0, 5.0]), pca_features=DenseVector([-4.6451, -1.1168])),
Row(features=DenseVector([4.0, 0.0, 0.0, 6.0, 7.0]), pca_features=DenseVector([-6.4289, -5.338]))]
然后，用我们的方法：
comp, score, eigVals = pca(df)
score.collect()
[array([ 1.64857282, 4.0132827 ]),
array([-4.64510433, 1.11679727]),
array([-6.42888054, 5.33795143])]
让我强调，我们不在我们定义的函数中使用任何collect()方法 - score是RDD，因为它应该是
请注意，我们第二列的符号与现有方法的符号完全相反;但这不是一个问题：根据（免费下载）An Introduction to Statistical Learning，由Hastie＆amp; Tibshirani，p。 382
每个主要组件加载向量都是唯一的，直到符号翻转。这个 意味着两个不同的软件包将产生相同的主体 组件加载向量，尽管那些加载向量的符号 可能有所不同标志可能不同，因为每个主要组件加载 vector指定p维空间中的方向：翻转符号没有 效果方向不会改变。 [...]同样，分数向量是唯一的 直至符号翻转，因为Z的方差与-Z的方差相同。
最后，既然我们已经有了特征值，那么为解释的方差百分比写一个函数是微不足道的：
def varianceExplained(df, k=1):
"""Calculate the fraction of variance explained by the top `k` eigenvectors.
Args:
df: A Spark dataframe with a 'features' column, which (column) consists of DenseVectors.
k: The number of principal components to consider.
Returns:
float: A number between 0 and 1 representing the percentage of variance explained
by the top `k` eigenvectors.
"""
components, scores, eigenvalues = pca(df, k)
return sum(eigenvalues[0:k])/sum(eigenvalues)
varianceExplained(df,1)
# 0.79439325322305299
作为测试，我们还检查我们的示例数据中解释的方差是否为1.0，对于k = 5（因为原始数据是5维的）：
varianceExplained(df,5)
# 1.0
这应该有效 ;随时提出您可能需要的任何澄清。
[已开发＆amp;用Spark 1.5.0＆amp;测试1.5.1]
答案 1 :(得分：14)
编辑：
$('#contact-select').change(function () {
var email = $(this).children('option:selected').data('email');
var phone = $(this).children('option:selected').data('phone');
$('#phone-href').setAttribute('href', 'tel:' + phone);
$('#mail-href').setAttribute("href", "mailto:" + email);
// set phone-href to phone - mind phone:
// set email-href to email - mind email:
});
和PCA最终都可以在 pyspark 中根据此已解决的JIRA票证SPARK-6227启动 spark 2.2.0
原始回答：
@desertnaut给出的答案从理论上来说实际上非常出色，但我想提出另一种方法来研究如何计算SVD并提取当时的特征向量。
SVD这定义了我们的SVD对象。我们现在可以使用Java Wrapper定义我们的computeSVD方法。
from pyspark.mllib.common import callMLlibFunc, JavaModelWrapper
from pyspark.mllib.linalg.distributed import RowMatrix
class SVD(JavaModelWrapper):
"""Wrapper around the SVD scala case class"""
@property
def U(self):
""" Returns a RowMatrix whose columns are the left singular vectors of the SVD if computeU was set to be True."""
u = self.call("U")
if u is not None:
return RowMatrix(u)
@property
def s(self):
"""Returns a DenseVector with singular values in descending order."""
return self.call("s")
@property
def V(self):
""" Returns a DenseMatrix whose columns are the right singular vectors of the SVD."""
return self.call("V")
现在，让我们将其应用于一个例子：
def computeSVD(row_matrix, k, computeU=False, rCond=1e-9):
"""
Computes the singular value decomposition of the RowMatrix.
The given row matrix A of dimension (m X n) is decomposed into U * s * V'T where
* s: DenseVector consisting of square root of the eigenvalues (singular values) in descending order.
* U: (m X k) (left singular vectors) is a RowMatrix whose columns are the eigenvectors of (A X A')
* v: (n X k) (right singular vectors) is a Matrix whose columns are the eigenvectors of (A' X A)
:param k: number of singular values to keep. We might return less than k if there are numerically zero singular values.
:param computeU: Whether of not to compute U. If set to be True, then U is computed by A * V * sigma^-1
:param rCond: the reciprocal condition number. All singular values smaller than rCond * sigma(0) are treated as zero, where sigma(0) is the largest singular value.
:returns: SVD object
"""
java_model = row_matrix._java_matrix_wrapper.call("computeSVD", int(k), computeU, float(rCond))
return SVD(java_model)
答案 2 :(得分：5)
在spark 2.2+中，您现在可以轻松地将解释的方差理解为：
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=<columns of your original dataframe>, outputCol="features")
df = assembler.transform(<your original dataframe>).select("features")
from pyspark.ml.feature import PCA
pca = PCA(k=10, inputCol="features", outputCol="pcaFeatures")
model = pca.fit(df)
sum(model.explainedVariance)
答案 3 :(得分：2)
您问题的最简单答案是在模型中输入单位矩阵。
identity_input = [(Vectors.dense([1.0, .0, 0.0, .0, 0.0]),),(Vectors.dense([.0, 1.0, .0, .0, .0]),), \
(Vectors.dense([.0, 0.0, 1.0, .0, .0]),),(Vectors.dense([.0, 0.0, .0, 1.0, .0]),),
(Vectors.dense([.0, 0.0, .0, .0, 1.0]),)]
df_identity = sqlContext.createDataFrame(identity_input,["features"])
identity_features = model.transform(df_identity)
这应该为您提供主要组成部分。
我认为eliasah在Spark框架方面的答案更好，因为Desertnaut通过使用numpy的函数而不是Spark的动作来解决问题。但是，eliasah的答案是缺少规范化数据。所以，我在eliasah的回答中添加了以下几行：
from pyspark.ml.feature import StandardScaler
standardizer = StandardScaler(withMean=True, withStd=False,
inputCol='features',
outputCol='std_features')
model = standardizer.fit(df)
output = model.transform(df)
pca_features = output.select("std_features").rdd.map(lambda row : row[0])
mat = RowMatrix(pca_features)
svd = computeSVD(mat,5,True)
渐渐地，svd.V和identity_features.select（＆＃34; pca_features＆＃34;）。collect（）应该具有相同的值。
编辑：我在此here
中总结了PCA及其在Spark和sklearn中的用法