我正在计算两个大向量集(具有相同特征)之间的余弦相似度。每组向量均表示为稀疏的CSR稀疏矩阵A和B。我要计算A x B ^ T ,它不会稀疏。但是,我只需要跟踪超过某个阈值的值,例如0.8。我正在尝试使用带有普通RDD的Pyspark来实现此目标,并想到使用对Scipy CSR矩阵实施的快速矢量运算。
A和B的行被标准化,因此要计算余弦相似度,我只需要找到A的每一行与B的每一行的点积。 A的尺寸为5,000,000 x 5,000。 B的尺寸为2,000,000 x 5,000。
假设A和B太大而无法作为广播变量放入我的工作程序节点上的内存中。我应该如何以最佳方式并行处理A和B?
编辑 发布解决方案后,我一直在探索其他更清晰,更优化的方法,尤其是为Spark MLlib IndexedRowMatrix对象实现的columnSimilarities()函数。 (Which pyspark abstraction is appropriate for my large matrix multiplication?)
答案 0 :(得分:0)
我能够在此框架中实现解决方案。
欢迎了解为什么这种解决方案很慢-是自定义序列化吗?
def csr_mult_helper(pair):
threshold=0.8
A_row = pair[0][0] # keep track of the row offset
B_col = pair[1][0] # offset for B (this will be a column index, after the transpose op)
A = sparse.csr_matrix(pair[0][1], pair[0][2]) # non-zero entires, size data
B = sparse.csr_matrix(pair[1][1], pair[1][2])
C = A * B.T # scipy sparse mat mul
for row_idx, row in enumerate(C): # I think it would be better to use a filter Transformation instead
col_indices = row.indices # but I had trouble with the row and column index book keeping
col_values = row.data
for col_idx, val in zip(col_indices, col_values):
if val > threshold:
yield (A_row + row_idx, B_col + col_idx, val) # source vector, target vector, cosine score
def parallelize_sparse_csr(M, rows_per_chunk=1):
[rows, cols] = M.shape
i_row = 0
submatrices = []
while i_row < rows:
current_chunk_size = min(rows_per_chunk, rows - i_row)
submat = M[i_row:(i_row + current_chunk_size)]
submatrices.append( (i_row, # offset
(submat.data, submat.indices, submat.indptr), # sparse matrix data
(current_chunk_size, cols)) ) # sparse matrix shape
i_row += current_chunk_size
return sc.parallelize(submatrices)
########## generate test data ###########
K,L,M,N = 5,2000,3,2000 # matrix dimensions (toy example)
A_ = sparse.rand(K,L, density=0.1, format='csr')
B_ = sparse.rand(M,N, density=0.1, format='csr')
print("benchmark: {} \n".format((A_ * B_.T).todense())) # benchmark solution for comparison
########## parallelize, multiply, and filter #########
t_start = time.time()
A = parallelize_sparse_csr(A_, rows_per_chunk=10)
B = parallelize_sparse_csr(B_, rows_per_chunk=10) # number of elements per partition, from B
# warning: this code breaks if the B_ matrix rows_per_chunk parameter != 1
# although I don't understand why yet
print("custom pyspark solution: ")
result = A.cartesian(B).flatMap(csr_mult_helper).collect()
print(results)
print("\n {} s elapsed".format(time.time() - t_start))