我试图在Julia中实施以下公式来计算工资分配的Gini coefficient:
其中
这是我使用的代码的简化版本:
# Takes a array where first column is value of wages
# (y_i in formula), and second column is probability
# of wage value (f(y_i) in formula).
function gini(wagedistarray)
# First calculate S values in formula
for i in 1:length(wagedistarray[:,1])
for j in 1:i
Swages[i]+=wagedistarray[j,2]*wagedistarray[j,1]
end
end
# Now calculate value to subtract from 1 in gini formula
Gwages = Swages[1]*wagedistarray[1,2]
for i in 2:length(Swages)
Gwages += wagedistarray[i,2]*(Swages[i]+Swages[i-1])
end
# Final step of gini calculation
return giniwages=1-(Gwages/Swages[length(Swages)])
end
wagedistarray=zeros(10000,2)
Swages=zeros(length(wagedistarray[:,1]))
for i in 1:length(wagedistarray[:,1])
wagedistarray[i,1]=1
wagedistarray[i,2]=1/10000
end
@time result=gini(wagedistarray)
它给出接近零的值,这是您对完全相等的工资分配的期望。但是,它需要相当长的时间:6.796秒。
有任何改进的想法吗?
答案 0 :(得分:13)
试试这个:
function gini(wagedistarray)
nrows = size(wagedistarray,1)
Swages = zeros(nrows)
for i in 1:nrows
for j in 1:i
Swages[i] += wagedistarray[j,2]*wagedistarray[j,1]
end
end
Gwages=Swages[1]*wagedistarray[1,2]
for i in 2:nrows
Gwages+=wagedistarray[i,2]*(Swages[i]+Swages[i-1])
end
return 1-(Gwages/Swages[length(Swages)])
end
wagedistarray=zeros(10000,2)
for i in 1:size(wagedistarray,1)
wagedistarray[i,1]=1
wagedistarray[i,2]=1/10000
end
@time result=gini(wagedistarray)
5.913907256 seconds (4000481676 bytes allocated, 25.37% gc time)
0.134799301 seconds (507260 bytes allocated)
elapsed time: 0.123665107 seconds (80112 bytes allocated)
主要问题是Swages
是一个全局变量(不是生活在函数中),这不是一个好的编码实践,但更重要的是performance killer。我注意到的另一件事是length(wagedistarray[:,1])
,它制作了该列的副本然后询问它的长度 - 这会产生一些额外的“垃圾”。第二次运行速度更快,因为在第一次运行函数时有一些编译时间。
使用@inbounds
,即
function gini(wagedistarray)
nrows = size(wagedistarray,1)
Swages = zeros(nrows)
@inbounds for i in 1:nrows
for j in 1:i
Swages[i] += wagedistarray[j,2]*wagedistarray[j,1]
end
end
Gwages=Swages[1]*wagedistarray[1,2]
@inbounds for i in 2:nrows
Gwages+=wagedistarray[i,2]*(Swages[i]+Swages[i-1])
end
return 1-(Gwages/Swages[length(Swages)])
end
给了我elapsed time: 0.042070662 seconds (80112 bytes allocated)
最后,查看这个版本,实际上比所有版本都快,也是我认为最准确的版本:
function gini2(wagedistarray)
Swages = cumsum(wagedistarray[:,1].*wagedistarray[:,2])
Gwages = Swages[1]*wagedistarray[1,2] +
sum(wagedistarray[2:end,2] .*
(Swages[2:end]+Swages[1:end-1]))
return 1 - Gwages/Swages[end]
end
哪个有elapsed time: 0.00041119 seconds (721664 bytes allocated)
。主要好处是从O(n ^ 2)双循环变为O(n)cumsum
。
答案 1 :(得分:4)
IainDunning已经为代码提供了一个很好的答案,该代码足够实用(函数gini2
)。如果一个人喜欢性能调整,可以通过避免临时数组(gini3
)来获得20倍的额外速度提升。请参阅以下代码,以比较两种实现的性能:
using TimeIt
wagedistarray=zeros(10000,2)
for i in 1:size(wagedistarray,1)
wagedistarray[i,1]=1
wagedistarray[i,2]=1/10000
end
wages = wagedistarray[:,1]
wagefrequencies = wagedistarray[:,2];
# original code
function gini2(wagedistarray)
Swages = cumsum(wagedistarray[:,1].*wagedistarray[:,2])
Gwages = Swages[1]*wagedistarray[1,2] +
sum(wagedistarray[2:end,2] .*
(Swages[2:end]+Swages[1:end-1]))
return 1 - Gwages/Swages[end]
end
# new code
function gini3(wages, wagefrequencies)
Swages_previous = wages[1]*wagefrequencies[1]
Gwages = Swages_previous*wagefrequencies[1]
@inbounds for i = 2:length(wages)
freq = wagefrequencies[i]
Swages_current = Swages_previous + wages[i]*freq
Gwages += freq * (Swages_current+Swages_previous)
Swages_previous = Swages_current
end
return 1.0 - Gwages/Swages_previous
end
result=gini2(wagedistarray) # warming up JIT
println("result with gini2: $result, time:")
@timeit result=gini2(wagedistarray)
result=gini3(wages, wagefrequencies) # warming up JIT
println("result with gini3: $result, time:")
@timeit result=gini3(wages, wagefrequencies)
输出结果为:
result with gini2: 0.0, time:
1000 loops, best of 3: 321.57 µs per loop
result with gini3: -1.4210854715202004e-14, time:
10000 loops, best of 3: 16.24 µs per loop
gini3
由于顺序求和而不如gini2
准确,因此必须使用pairwise summation的变体来提高准确性。