Question

我正在使用患者信息数据集，并尝试使用MATLAB从数据中计算倾向得分。删除具有很多缺失值的特征后，我仍然剩下几个缺失（NaN）值。

当我尝试使用以下Matlab代码（来自Andrew Ng的Coursera Machine Learning类）执行逻辑回归时，由于这些缺失的值，我的成本函数和梯度向量的值变为NaN，因此我会出错：

[m, n] = size(X);
X = [ones(m, 1) X];    
initial_theta = ones(n+1, 1);
[cost, grad] = costFunction(initial_theta, X, y);
options = optimset('GradObj', 'on', 'MaxIter', 400);

[theta, cost] = ...
    fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);

注意：sigmoid和costfunction是我为总体上易于使用而创建的工作函数。

如果我将所有NaN值替换为1或0，则计算可以顺利进行。但是，我不确定这是否是解决此问题的最佳方法，而且我还想知道应该选择哪种替换值（在一般情况下），以在缺失数据的情况下进行逻辑回归以获得最佳结果。 使用特定数字（0或1或其他）代替我的数据中所说的缺失值是否有好处/缺点？

注意：我也将所有特征值标准化为0-1。

任何对此问题的见解将受到高度赞赏。谢谢

Answer 1

如前所述，无论编程平台如何，这都是人们要解决的普遍问题。它被称为“缺少数据插补”。

将所有缺失值强制为一个特定数字肯定有缺点。取决于数据的分布，这可能会很麻烦，例如，将零个数多于零的二进制稀疏数据中的所有缺失值都设置为1。

幸运的是，MATLAB具有一个名为knnimpute的函数，该函数通过其最邻近的邻居来估计丢失的数据点。

根据我的经验，我经常发现knnimpute有用。但是，当您的数据中缺少太多站点时，它可能会不足。丢失站点的邻居也可能不完整，从而导致估算不准确。下面，我想出了一个解决方案。它从插入最少不完整的列开始（可选）为邻居施加安全的预定义距离。我希望这会有所帮助。

function data = dnnimpute(data,distCutoff,option,distMetric)
% data = dnnimpute(data,distCutoff,option,distMetric)
%
%   Distance-based nearest neighbor imputation that impose a distance
%     cutoff to determine nearest neighbors, i.e., avoids those samples 
%     that are more distant than the distCutoff argument.
%
%   Imputes missing data coded by "NaN" starting from the covarites 
%     (columns) with the least number of missing data. Then it continues by 
%     including more (complete) covariates in the calculation of pair-wise 
%     distances.
%
%   option, 
%       'median'      - Median of the nearest neighboring values
%       'weighted'    - Weighted average of the nearest neighboring values
%       'default'     - Unweighted average of the nearest neighboring values
%
%   distMetric,
%       'euclidean'   - Euclidean distance (default)
%       'seuclidean'  - Standardized Euclidean distance. Each coordinate
%                       difference between rows in X is scaled by dividing
%                       by the corresponding element of the standard
%                       deviation S=NANSTD(X). To specify another value for
%                       S, use D=pdist(X,'seuclidean',S).
%       'cityblock'   - City Block distance
%       'minkowski'   - Minkowski distance. The default exponent is 2. To
%                       specify a different exponent, use
%                       D = pdist(X,'minkowski',P), where the exponent P is
%                       a scalar positive value.
%       'chebychev'   - Chebychev distance (maximum coordinate difference)
%       'mahalanobis' - Mahalanobis distance, using the sample covariance
%                       of X as computed by NANCOV. To compute the distance
%                       with a different covariance, use
%                       D =  pdist(X,'mahalanobis',C), where the matrix C
%                       is symmetric and positive definite.
%       'cosine'      - One minus the cosine of the included angle
%                       between observations (treated as vectors)
%       'correlation' - One minus the sample linear correlation between
%                       observations (treated as sequences of values).
%       'spearman'    - One minus the sample Spearman's rank correlation
%                       between observations (treated as sequences of values).
%       'hamming'     - Hamming distance, percentage of coordinates
%                       that differ
%       'jaccard'     - One minus the Jaccard coefficient, the
%                       percentage of nonzero coordinates that differ
%       function      - A distance function specified using @, for
%                       example @DISTFUN.
%  
if nargin < 3
    option = 'mean';
end
if nargin < 4
    distMetric = 'euclidean';
end

nanVals = isnan(data);
nanValsPerCov = sum(nanVals,1);
noNansCov = nanValsPerCov == 0;
if isempty(find(noNansCov, 1))
    [~,leastNans] = min(nanValsPerCov);
    noNansCov(leastNans) = true;
    first = data(nanVals(:,noNansCov),:);
    nanRows = find(nanVals(:,noNansCov)==true); i = 1;
    for row = first'
        data(nanRows(i),noNansCov) = mean(row(~isnan(row)));
        i = i+1;
    end
end

nSamples = size(data,1);
if nargin < 2
    dataNoNans = data(:,noNansCov);
    distances = pdist(dataNoNans);
    distCutoff = min(distances);
end
[stdCovMissDat,idxCovMissDat] = sort(nanValsPerCov,'ascend');
imputeCols = idxCovMissDat(stdCovMissDat>0);
% Impute starting from the cols (covariates) with the least number of 
% missing data. 
for c = reshape(imputeCols,1,length(imputeCols))    
    imputeRows = 1:nSamples;
    imputeRows = imputeRows(nanVals(:,c));   
    for r = reshape(imputeRows,1,length(imputeRows))
        % Calculate distances
        distR = inf(nSamples,1);
        %
        noNansCov_r = find(isnan(data(r,:))==0);
        noNansCov_r = noNansCov_r(sum(isnan(data(nanVals(:,c)'==false,~isnan(data(r,:)))),1)==0);
        %
        for i = find(nanVals(:,c)'==false)
            distR(i) = pdist([data(r,noNansCov_r); data(i,noNansCov_r)],distMetric);            
        end
        tmp = min(distR(distR>0));        
        % Impute the missing data at sample r of covariate c
        switch option
            case 'weighted'
                data(r,c) = (1./distR(distR<=max(distCutoff,tmp)))' * data(distR<=max(distCutoff,tmp),c) / sum(1./distR(distR<=max(distCutoff,tmp)));
            case 'median'
                data(r,c) = median(data(distR<=max(distCutoff,tmp),c),1);
            case 'mean'
                data(r,c) = mean(data(distR<=max(distCutoff,tmp),c),1);
        end
        % The missing data in sample r is imputed. Update the sample 
        % indices of c which are imputed. 
        nanVals(r,c) = false;  
    end    
    fprintf('%u/%u of the covariates are imputed.\n',find(c==imputeCols),length(imputeCols));
end

Answer 2

要处理丢失的数据，可以使用以下三个选项之一：

如果缺少很多缺少值的实例，则可以删除那些缺少值的实例。
如果您有很多功能并且可以承受丢失某些信息的负担，请删除缺少值的整个功能。
最好的方法是填充一些值（均值，中位数）来代替缺失值。您可以为该功能计算其余训练示例的平均值，然后用平均值填充所有缺失的值。平均值保持在数据分布中，效果很好。

注意：用均值替换缺失值时，仅使用训练集计算均值。另外，存储该值并使用它来更改测试集中的缺失值。

如果使用0或1替换所有丢失的值，则数据可能会倾斜，因此最好用所有其他值的平均值替换丢失的值。

处理Logistic回归的NaN（缺失）值-最佳做法？

2 个答案: