在Clojure

时间:2017-11-12 22:31:11

标签: clojure

我想使用自定义Comparator对原始Java数组进行排序,但是我遇到了类型错误。我认为comparator函数正在创建Comparator<java.lang.Object>而不是Comparator<Long>,但我无法弄清楚如何解决这个问题。

这是一个最小的例子:

x.core=> (def x (double-array [4 3 5 6 7]))
#'x.core/x
x.core=> (java.util.Arrays/sort x (comparator #(> %1 %2)))

ClassCastException [D cannot be cast to [Ljava.lang.Object;  x.core/eval1524 (form-init5588058267991397340.clj:1)

我已尝试在比较器功能中添加不同类型的提示,但坦率地说,我对该语言相对较新,基本上只是投掷飞镖。

我故意简化上面的例子,专注于关键问题,这是一个类型错误。在下面的部分中,我尝试提供一些更详细的信息来激发问题并说明我使用自定义Comparator的原因。

动机

我要做的是复制R&#39 order函数,其功能如下:

> x = c(7, 2, 5, 3, 1, 4, 6)
> order(x)
[1] 5 2 4 6 3 7 1
> x[order(x)]
[1] 1 2 3 4 5 6 7

正如您所看到的,它返回将对其输入向量进行排序的索引的排列。

这是Clojure中的一个有效解决方案:

(defn order
  "Permutation of indices sorted by x"
  [x]
  (let [v (vec x)]
    (sort-by #(v %) (range (count v)))))

x.core=> (order [7 2 5 3 1 4 6])
(4 1 3 5 2 6 0)

(注意,当Clojure为0索引时,R是1索引的。)诀窍是将一个向量(即x [0, 1, ..., (count x)]的索引除以向量x本身。

R vs. Clojure Performance

不幸的是,我对此解决方案的性能感到困扰。 R解决方案很多更快:

> x = runif(1000000)
> system.time({ y = order(x) })
   user  system elapsed
  0.041   0.004   0.046

对应的Clojure代码:

x.core=> (def x (repeatedly 1000000 rand))
#'x.core/x
x.core=> (time (def y (order x)))
"Elapsed time: 2857.216452 msecs"
#'x.core/y

Primitive Arrays the Solution?

我发现原始数组倾向于在与R:

相当的时间内排序
> x = runif(1000000)
> system.time({ y = sort(x) })
   user  system elapsed
  0.061   0.005   0.069

VS。

x.core=> (def x (double-array (repeatedly 1000000 rand)))
#'x.core/x
x.core=> (time (java.util.Arrays/sort x))
"Elapsed time: 86.827277 msecs"
nil

这是我尝试将自定义Comparator与java.util.Arrays类一起使用的动机。我希望速度可与R相媲美。

我应该补充一点,我可以使用带有ArrayList的自定义Comparator,如下所示,但性能并不比我的启动功能好:

(defn order2
  [x]
  (let [v (vec x)
        compx (comparator (fn [i j] (< (v i) (v j))))
        ix (java.util.ArrayList. (range (count v)))]
    (java.util.Collections/sort ix compx)
    (vec ix)))

即使您只想提供一些一般的Clojure建议,我们也将不胜感激。我还在学习这门语言,并且很有乐趣。 : - )

修改

Per Carcigenicate的答案如下,

(defn order
  [x]
  (let [ix (int-array (range (count x)))]
    (vec (-> (java.util.Arrays/stream ix)
             (.boxed)
             (.sorted (fn [i j] (< (aget x i) (aget x j))))
             (.mapToInt
               (proxy [java.util.function.ToIntFunction] []
                 (applyAsInt [^long d] d)))
             (.toArray)))))

将起作用:

x.core=> (def x (double-array [5 3 1 3.14 -10]))
#'x.core/x
x.core=> (order x)
[4 2 1 3 0]
x.core=> (map #(aget x %) (order x))
(-10.0 1.0 3.0 3.14 5.0)

不幸的是,它超级慢。我猜原始人可能不是答案。

1 个答案:

答案 0 :(得分:1)

这是使用带有随机枢轴的快速排序的order函数的Clojure实现。它与R相当接近:使用你的100万双打基准,我的定时时间大多在520-530毫秒范围内,而R一般在500毫秒左右徘徊。

更新:使用非常基本的双线程版本(2x快速排序,然后是合并步骤,生成输出向量)我的时间明显改善 - 最差基准平均值为415 ms,否则我倾向于在325-365 ms范围内得到结果。有关双线程版本,请参阅此消息的结尾,或者如果您更喜欢gist形式的版本,则它们是 - two-threadedsingle-threaded

请注意,它将输入作为中间步骤倒入一个双精度数组中,最终返回long的向量。在我的盒子上将一百万个双打倒入一个向量似乎只需要超过30毫秒,所以如果你对数组结果感到满意,你可以不用这一步。

主要并发症是invokePrim - 从Clojure 1.9.0-RC1开始,在该位置的常规函数​​调用将导致拳击。其他方法也是可行的,但这很有效,而且看起来很简单。

有关基准测试结果,请参阅此消息的结尾。第一次运行的较低分位数结果实际上是报告的最佳结果

(defn order2 [xs]
  (let [rnd (java.util.Random.)
        a1 (double-array xs)
        a2 (long-array (alength a1))]
    (dotimes [i (alength a2)]
      (aset a2 i i))
    (letfn [(quicksort [^long l ^long h]
              (if (< l h)
                (let [p (.invokePrim ^clojure.lang.IFn$LLL partition l h)]
                  (quicksort l (dec p))
                  (quicksort (inc p) h))))
            (partition ^long [^long l ^long h]
              (let [pidx (+ l (.nextInt rnd (- h l)))
                    pivot (aget a1 pidx)]
                (swap1 a1 pidx h)
                (swap2 a2 pidx h)
                (loop [i (dec l)
                       j l]
                  (if (< j h)
                    (if (< (aget a1 j) pivot)
                      (let [i (inc i)]
                        (swap1 a1 i j)
                        (swap2 a2 i j)
                        (recur i (inc j)))
                      (recur i (inc j)))
                    (let [i (inc i)]
                      (when (< (aget a1 h) (aget a1 i))
                        (swap1 a1 i h)
                        (swap2 a2 i h))
                      i)))))
            (swap1 [^doubles a ^long i ^long j]
              (let [tmp (aget a i)]
                (aset a i (aget a j))
                (aset a j tmp)))
            (swap2 [^longs a ^long i ^long j]
              (let [tmp (aget a i)]
                (aset a i (aget a j))
                (aset a j tmp)))]
      (quicksort 0 (dec (alength a1)))
      (vec a2))))

基准测试结果(注意:第一次测试使用问题文本中定义的x - (def x (repeatedly 1000000 rand));它也使用c/bench,而以下运行使用c/quick-bench) :

user> (c/bench (order2 x))
Evaluation count : 120 in 60 samples of 2 calls.
             Execution time mean : 522.485408 ms
    Execution time std-deviation : 33.490530 ms
   Execution time lower quantile : 470.089782 ms ( 2.5%)
   Execution time upper quantile : 575.687990 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order2 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 527.020004 ms
    Execution time std-deviation : 14.846061 ms
   Execution time lower quantile : 507.175127 ms ( 2.5%)
   Execution time upper quantile : 543.675752 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order2 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 513.476501 ms
    Execution time std-deviation : 12.828449 ms
   Execution time lower quantile : 497.164534 ms ( 2.5%)
   Execution time upper quantile : 525.094463 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order2 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 529.826816 ms
    Execution time std-deviation : 21.454522 ms
   Execution time lower quantile : 508.547461 ms ( 2.5%)
   Execution time upper quantile : 552.592925 ms (97.5%)
                   Overhead used : 15.378363 ns
nil

来自同一个方框的一些R时间进行比较:

> system.time({ y = order(x) })
   user  system elapsed 
  0.512   0.004   0.514 
> system.time({ y = order(x) })
   user  system elapsed 
  0.496   0.000   0.496 
> system.time({ y = order(x) })
   user  system elapsed 
  0.508   0.000   0.510 
> system.time({ y = order(x) })
   user  system elapsed 
  0.508   0.000   0.513 
> system.time({ y = order(x) })
   user  system elapsed 
  0.496   0.000   0.499 
> system.time({ y = order(x) })
   user  system elapsed 
  0.500   0.000   0.502 

更新:双线程Clojure版本:

(defn order3 [xs]
  (let [rnd (java.util.Random.)
        a1 (double-array xs)
        a2 (long-array (alength a1))]
    (dotimes [i (alength a2)]
      (aset a2 i i))
    (letfn [(quicksort [^long l ^long h]
              (if (< l h)
                (let [p (.invokePrim ^clojure.lang.IFn$LLL partition l h)]
                  (quicksort l (dec p))
                  (quicksort (inc p) h))))
            (partition ^long [^long l ^long h]
              (let [pidx (+ l (.nextInt rnd (- h l)))
                    pivot (aget a1 pidx)]
                (swap1 a1 pidx h)
                (swap2 a2 pidx h)
                (loop [i (dec l)
                       j l]
                  (if (< j h)
                    (if (< (aget a1 j) pivot)
                      (let [i (inc i)]
                        (swap1 a1 i j)
                        (swap2 a2 i j)
                        (recur i (inc j)))
                      (recur i (inc j)))
                    (let [i (inc i)]
                      (when (< (aget a1 h) (aget a1 i))
                        (swap1 a1 i h)
                        (swap2 a2 i h))
                      i)))))
            (swap1 [^doubles a ^long i ^long j]
              (let [tmp (aget a i)]
                (aset a i (aget a j))
                (aset a j tmp)))
            (swap2 [^longs a ^long i ^long j]
              (let [tmp (aget a i)]
                (aset a i (aget a j))
                (aset a j tmp)))]
      (let [lim (alength a1)
            mid (quot lim 2)
            f1 (future (quicksort 0 (dec mid)))
            f2 (future (quicksort mid (dec lim)))]
        @f1
        @f2
        (loop [out (transient [])
               i 0
               j mid]
          (cond
            (== i mid)
            (persistent!
              (if (== j lim)
                out
                (reduce (fn [out j]
                          (conj! out (aget a2 j)))
                  out
                  (range j lim))))

            (== j lim)
            (persistent!
              (reduce (fn [out i]
                        (conj! out (aget a2 i)))
                out
                (range i mid)))

            :else
            (let [ie (aget a1 i)
                  je (aget a1 j)]
              (if (< ie je)
                (recur (conj! out (aget a2 i)) (inc i) j)
                (recur (conj! out (aget a2 j)) i (inc j))))))))))

这个的一些基准测试结果:

user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order3 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 325.351056 ms
    Execution time std-deviation : 3.511578 ms
   Execution time lower quantile : 321.947510 ms ( 2.5%)
   Execution time upper quantile : 330.375038 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order3 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 339.422989 ms
    Execution time std-deviation : 19.929177 ms
   Execution time lower quantile : 318.996436 ms ( 2.5%)
   Execution time upper quantile : 366.113347 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order3 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 415.171336 ms
    Execution time std-deviation : 13.624262 ms
   Execution time lower quantile : 393.242455 ms ( 2.5%)
   Execution time upper quantile : 428.881001 ms (97.5%)
                   Overhead used : 15.378363 ns

Found 1 outliers in 6 samples (16.6667 %)
    low-severe   1 (16.6667 %)
 Variance from outliers : 13.8889 % Variance is moderately inflated by outliers
nil
user> (let [x (repeatedly 1000000 rand)]
        (c/quick-bench (order3 x)))
Evaluation count : 6 in 6 samples of 1 calls.
             Execution time mean : 324.547827 ms
    Execution time std-deviation : 5.196817 ms
   Execution time lower quantile : 318.541727 ms ( 2.5%)
   Execution time upper quantile : 331.878289 ms (97.5%)
                   Overhead used : 15.378363 ns
nil
user> (c/bench (order3 x))
Evaluation count : 180 in 60 samples of 3 calls.
             Execution time mean : 361.529793 ms
    Execution time std-deviation : 45.285047 ms
   Execution time lower quantile : 307.535934 ms ( 2.5%)
   Execution time upper quantile : 446.679687 ms (97.5%)
                   Overhead used : 15.378363 ns

Found 1 outliers in 60 samples (1.6667 %)
    low-severe   1 (1.6667 %)
 Variance from outliers : 78.9377 % Variance is severely inflated by outliers
nil