Question

对于项目，我需要复制当前存在于Stata输出文件（.dta）中的一些结果，并且是从较旧的Stata脚本计算出来的。新版本的项目需要用Python编写。

我遇到困难的具体部分是基于Stata xtile command的加权版本匹配分位数断点计算。请注意，数据点之间的关系与权重无关，而我使用的权重来自连续数量，因此关系极不可能（并且我的测试数据集中没有关系）。因此，由于关系的错误分类不是它。

我已经阅读了Wikipedia article on weighted percentiles和this Cross-Validated post，它描述了一个应该复制R的7型分位数的替代算法。

我已经实现了两种加权算法（底部的代码），但是我仍然不能很好地匹配Stata输出中的计算分位数。

有没有人知道Stata例程使用的具体算法？文档没有清楚地描述这一点。它说的是在CDF的平坦部分采取平均值来反转它，但这几乎不能描述实际的算法，并且对于它是否正在进行任何其他插值都不明确。

请注意，numpy.percentile和scipy.stats.mstats.mquantiles不接受权重，不能执行加权分位数，只是常规的等加权分位数。我的问题的关键在于需要使用权重。

注意：我已经在下面调试了这两种方法，但是如果你看到一个，可以随意在评论中提出错误。我已经在较小的数据集上测试了这两种方法，结果很好，并且在我可以保证R使用什么方法的情况下也匹配R的输出。代码不是那么优雅，而且在两种类型之间复制的太多了，但是当我相信输出是我需要的时候，所有这些都将被修复。

问题在于我不知道Stata xtile使用的方法，并且我希望在同一数据集上运行时减少下面的代码与Stata xtile之间的不匹配。

我尝试过的算法：

import numpy as np

def mark_weighted_percentiles(a, labels, weights, type):
# a is an input array of values.
# weights is an input array of weights, so weights[i] goes with a[i]
# labels are the names you want to give to the xtiles
# type refers to which weighted algorithm. 
#      1 for wikipedia, 2 for the stackexchange post.

# The code outputs an array the same shape as 'a', but with
# labels[i] inserted into spot j if a[j] falls in x-tile i.
# The number of xtiles requested is inferred from the length of 'labels'.


# First type, "vanilla" weights from Wikipedia article.
if type == 1:

    # Sort the values and apply the same sort to the weights.
    N = len(a)
    sort_indx = np.argsort(a)
    tmp_a = a[sort_indx].copy()
    tmp_weights = weights[sort_indx].copy()

    # 'labels' stores the name of the x-tiles the user wants,
    # and it is assumed to be linearly spaced between 0 and 1
    # so 5 labels implies quintiles, for example.
    num_categories = len(labels)
    breaks = np.linspace(0, 1, num_categories+1)

    # Compute the percentile values at each explicit data point in a.
    cu_weights = np.cumsum(tmp_weights)
    p_vals = (1.0/cu_weights[-1])*(cu_weights - 0.5*tmp_weights)

    # Set up the output array.
    ret = np.repeat(0, len(a))
    if(len(a)<num_categories):
        return ret

    # Set up the array for the values at the breakpoints.
    quantiles = []


    # Find the two indices that bracket the breakpoint percentiles.
    # then do interpolation on the two a_vals for those indices, using
    # interp-weights that involve the cumulative sum of weights.
    for brk in breaks:
        if brk <= p_vals[0]: 
            i_low = 0; i_high = 0;
        elif brk >= p_vals[-1]:
            i_low = N-1; i_high = N-1;
        else:
            for ii in range(N-1):
                if (p_vals[ii] <= brk) and (brk < p_vals[ii+1]):
                    i_low  = ii
                    i_high = ii + 1       

        if i_low == i_high:
            v = tmp_a[i_low]
        else:
            # If there are two brackets, then apply the formula as per Wikipedia.
            v = tmp_a[i_low] + ((brk-p_vals[i_low])/(p_vals[i_high]-p_vals[i_low]))*(tmp_a[i_high]-tmp_a[i_low])

        # Append the result.
        quantiles.append(v)

    # Now that the weighted breakpoints are set, just categorize
    # the elements of a with logical indexing.
    for i in range(0, len(quantiles)-1):
        lower = quantiles[i]
        upper = quantiles[i+1]
        ret[ np.logical_and(a>=lower, a<upper) ] = labels[i] 

    #make sure upper and lower indices are marked
    ret[a<=quantiles[0]] = labels[0]
    ret[a>=quantiles[-1]] = labels[-1]

    return ret

# The stats.stackexchange suggestion.
elif type == 2:

    N = len(a)
    sort_indx = np.argsort(a)
    tmp_a = a[sort_indx].copy()
    tmp_weights = weights[sort_indx].copy()


    num_categories = len(labels)
    breaks = np.linspace(0, 1, num_categories+1)

    cu_weights = np.cumsum(tmp_weights)

    # Formula from stats.stackexchange.com post.
    s_vals = [0.0];
    for ii in range(1,N):
        s_vals.append( ii*tmp_weights[ii] + (N-1)*cu_weights[ii-1])
    s_vals = np.asarray(s_vals)

    # Normalized s_vals for comapring with the breakpoint.
    norm_s_vals = (1.0/s_vals[-1])*s_vals 

    # Set up the output variable.
    ret = np.repeat(0, N)
    if(N < num_categories):
        return ret

    # Set up space for the values at the breakpoints.
    quantiles = []


    # Find the two indices that bracket the breakpoint percentiles.
    # then do interpolation on the two a_vals for those indices, using
    # interp-weights that involve the cumulative sum of weights.
    for brk in breaks:
        if brk <= norm_s_vals[0]: 
            i_low = 0; i_high = 0;
        elif brk >= norm_s_vals[-1]:
            i_low = N-1; i_high = N-1;
        else:
            for ii in range(N-1):
                if (norm_s_vals[ii] <= brk) and (brk < norm_s_vals[ii+1]):
                    i_low  = ii
                    i_high = ii + 1   

        if i_low == i_high:
            v = tmp_a[i_low]
        else:
            # Interpolate as in the type 1 method, but using the s_vals instead.
            v = tmp_a[i_low] + (( (brk*s_vals[-1])-s_vals[i_low])/(s_vals[i_high]-s_vals[i_low]))*(tmp_a[i_high]-tmp_a[i_low])
        quantiles.append(v)

    # Now that the weighted breakpoints are set, just categorize
    # the elements of a as usual. 
    for i in range(0, len(quantiles)-1):
        lower = quantiles[i]
        upper = quantiles[i+1]
        ret[ np.logical_and( a >= lower, a < upper ) ] = labels[i] 

    #make sure upper and lower indices are marked
    ret[a<=quantiles[0]] = labels[0]
    ret[a>=quantiles[-1]] = labels[-1]

    return ret

Answer 1

以下是Stata 12手册中公式的截图（StataCorp.2011。Stata Statistical Software：Release 12. College Station，TX：StataCorp LP，p.501-502）。如果这没有帮助，您可以在Statalist上提出这个问题，或直接联系Philip Ryan（原始代码的作者）。

enter image description here

Answer 2

您知道吗，您可以阅读Stata的代码吗？

. ssc install adoedit
. adoedit xtile

使用Python匹配Stata加权xtile命令的明确方法？

2 个答案: