Question

我尝试用scipy.sparse从json文件创建一个矩阵。

我通过这种方式拥有json文件

{"reviewerID": "A10000012B7CGYKOMPQ4L", "asin": "000100039X", "reviewerName": "Adam", "helpful": [0, 0], "reviewText": "Spiritually and mentally inspiring! A book that allows you to question your morals and will help you discover who you really are!", "overall": 5.0, "summary": "Wonderful!", "unixReviewTime": 1355616000, "reviewTime": "12 16, 2012"}

这是我的Json格式...更多类似的元素（基于Amazon Review文件）

并希望对其进行稀疏

    count            
object       a   b   c   d
id                   
him       NaN   1 NaN   1
me          1 NaN NaN   1
you         1 NaN   1 NaN

我正在尝试这样做

i

mport numpy as np
import pandas as pd
from scipy.sparse import csr_matrix

df= pd.read_json('C:\\Users\\anto-\\Desktop\\university\\Big Data computing\\Ex. Resource\\test2.json',lines=True)


a= df['reviewerID']
b= df['asin']
data= df.groupby(["reviewerID"]).size()



row = df.reviewerID.astype('category', categories=a).cat.codes
col = df.asin.astype('category', categories=b).cat.codes
sparse_matrix = csr_matrix((data, (row, col)), shape=(len(a), len(b)))

从这个旧示例中阅读

Efficiently create sparse pivot tables in pandas?

我在代码中弃用元素时遇到一些错误，但是我不了解如何构造此矩阵。

这是错误日志：

 FutureWarning: specifying 'categories' or 'ordered' in .astype() is deprecated; pass a CategoricalDtype instead
  from ipykernel import kernelapp as app

我有点困惑。有人可以给我一些建议或类似的例子吗？

Answer 1

产生一个看起来像

的稀疏矩阵

    count            
object       a   b   c   d
id                   
him       NaN   1 NaN   1
me          1 NaN NaN   1
you         1 NaN   1 NaN

您需要生成3个数组，例如：

In [215]: from scipy import sparse
In [216]: data = np.array([1,1,1,1,1,1])
In [217]: row = np.array([1,2,0,2,0,1])
In [218]: col = np.array([0,0,1,2,3,3])
In [219]: M = sparse.csr_matrix((data, (row, col)), shape=(3,4))
In [220]: M
Out[220]: 
<3x4 sparse matrix of type '<class 'numpy.int64'>'
    with 6 stored elements in Compressed Sparse Row format>
In [221]: M.A
Out[221]: 
array([[0, 1, 0, 1],
       [1, 0, 0, 1],
       [1, 0, 1, 0]], dtype=int64)

诸如“他”，“我”，“你”之类的类别必须映射到唯一索引，例如0,1,2。同样对于“ a”，“ b”，“ c”，“ d”。

json文件中的稀疏

1 个答案: