Question

我有一个数据集，其中有NaN个数据。我正在使用熊猫从文件中提取数据，并使用numpy对其进行处理。这是我读取数据的代码：

import pandas as pd
import numpy as np

def makeArray(band):
    """
    Takes as argument a string as the name of a wavelength band.
    Converts the list of magnitudes in that band into a numpy array,
    replacing invalid values (where invalid == -999) with NaNs.
    Returns the array.
    """
    array_name = band + '_mag'
    array = np.array(df[array_name])
    array[array==-999]=np.nan
    return array

#   Read data file
fields = ['no', 'NED', 'z', 'obj_type','S_21', 'power', 'SI_flag', 
          'U_mag', 'B_mag', 'V_mag', 'R_mag', 'K_mag', 'W1_mag',
          'W2_mag', 'W3_mag', 'W4_mag', 'L_UV', 'Q', 'flag_uv']

magnitudes = ['U_mag', 'B_mag', 'V_mag', 'R_mag', 'K_mag', 'W1_mag',
          'W2_mag', 'W3_mag', 'W4_mag']

df = pd.read_csv('todo.dat', sep = ' ',
                   names = fields, index_col = False)

#   Define axes for processing
redshifts = np.array(df['z'])
y = np.log(makeArray('K'))
mask = np.isnan(y)

我想一个最小的工作示例是：

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

randomNumberGenerator = np.random.RandomState(1000)
x = 4 * randomNumberGenerator.rand(100)
y = 4 * x - 1+ randomNumberGenerator.randn(100)
y[50] = np.nan

slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
fit = slope*x + intercept

plt.scatter(x, y)
plt.plot(x, fit)
plt.show()

注释MWE中的y[50] = np.nan行会生成一个漂亮的图形，但是包含它会产生与我的实际数据相同的错误消息：

C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
  return (self.a < x) & (x < self.b)
C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
  return (self.a < x) & (x < self.b)
C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:1818: RuntimeWarning: invalid value encountered in less_equal
  cond2 = cond0 & (x <= self.a)

实际数据帧的摘要：

no  NED z   obj_type    S_21    power   SI_flag U_mag   B_mag   V_mag   R_mag   K_mag   W1_mag  W2_mag  W3_mag  W4_mag  L_UV    Q   flag_uv
1   SDSSJ000005.95+145310.1 2.499   *   0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
4   SDSSJ000009.27+020621.9 1.432   UvS 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
5   SDSSJ000009.38+135618.4 2.239   QSO 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
6   SDSSJ000011.37+150335.7 2.18    *   0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
11  SDSSJ000030.64-064100.0 2.606   QSO 0.0 0.0     -999.0  -999.0  -999.0  -999.0  15.46   -999.0  -999.0  -999.0  -999.0  23.342  56.211000000000006  UV
15  SDSSJ000033.05+114049.6 0.73    UvS 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
27  LBQS2358+0038   0.95    QSO 0.0 0.0     17.342  18.483  18.203  17.825  -999.0  -999.0  -999.0  -999.0  -999.0  23.301  56.571999999999996  UV

我正在针对_mag绘制每个z列，并且试图计算并绘制一个线性回归，不包括NaN。

我已经尝试过numpy.linalg，numpy.poly，scipy.stats.linregress和statsmodels.api，但似乎他们中的任何一个都不能轻易处理{{1} } s。我在SE上发现的其他问题正在引导我转圈。

如何像MWE所示那样在数据顶部绘制OLS回归拟合？

Answer 1

您可以使用df.dropna()查看以下链接：pandas.DataFrame.dropna

Answer 2

您必须将数据转换为数据框，以删除包含至少一个NAN值的整列。这样，您将不会收到前面收到的警告。试试这个，

import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
import pandas as pd

randomNumberGenerator = np.random.RandomState(1000)
x = 4 * randomNumberGenerator.rand(100)
y = 4 * x - 1+ randomNumberGenerator.randn(100)
y[50] = np.nan

df1 = pd.DataFrame({'x': x})
df1['y'] = y
df1 = df1.dropna()
x = df1.x
y = df1.y

slope, intercept, r_value, p_value, std_err = stats.linregress(x, y)
fit = slope*x + intercept

plt.scatter(x, y)
plt.plot(x, fit)
plt.show()

在线性回归中处理NaN-科学吗？

2 个答案: