np.linalg.lstsq中的“大”数组崩溃

时间:2013-04-26 06:27:43

标签: python numpy least-squares atlas large-data

我在np.linalg.lstsq()遇到错误,其中“大型”阵列导致可怕的(非pythonic)崩溃。

一些例子:

这有效:

python -c "import numpy as np; np.linalg.lstsq(np.random.randn(10000,25),np.random.randn(10000))"

这失败了,有一个相当大的追溯:

python -c "import numpy as np; np.linalg.lstsq(np.random.randn(10000,26),np.random.randn(10000))"

26处的失败和25处的成功是可重复的,并且显然可重复到python -c "import numpy as np; np.linalg.lstsq(np.random.randn(26,26),np.random.randn(26))",低于python -c "import numpy as np; np.linalg.lstsq(np.random.randn(25,26),np.random.randn(25))"和更小的所有工作。

任何人都可以帮我解释错误信息或找出出错的地方吗?

(numpy版本是1.7.1)

*** glibc detected *** python: free(): invalid next size (normal): 0x00000000048db990 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2b28f29a845f]
/lib64/libc.so.6(cfree+0x4b)[0x2b28f29a88bb]
/projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/multiarray.so[0x2b28f633287a]
python[0x4f9df9]
python(PyEval_EvalCodeEx+0x3a3)[0x49ae73]
python(PyEval_EvalFrameEx+0x5712)[0x498ce2]
python(PyEval_EvalCodeEx+0x89b)[0x49b36b]
python(PyEval_EvalCode+0x32)[0x49b3e2]
python(PyRun_StringFlags+0x115)[0x4bdd35]
python(PyRun_SimpleStringFlags+0x3e)[0x4bde2e]
python(Py_Main+0x4d7)[0x414817]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b28f2953994]
python[0x413f69]
======= Memory map: ========
00400000-0055c000 r-xp 00000000 00:2e 51464913                           /projects/ginsbura/local/bin/python2.7
0075c000-00798000 rw-p 0015c000 00:2e 51464913                           /projects/ginsbura/local/bin/python2.7
00798000-007a7000 rw-p 00798000 00:00 0
03bbd000-04985000 rw-p 03bbd000 00:00 0                                  [heap]
2b28f1e72000-2b28f1e8e000 r-xp 00000000 00:12 85470                      /lib64/ld-2.5.so
2b28f1e8e000-2b28f1f12000 rw-p 2b28f1e8e000 00:00 0
2b28f1f44000-2b28f1fc6000 rw-p 2b28f1f44000 00:00 0
2b28f208e000-2b28f208f000 r--p 0001c000 00:12 85470                      /lib64/ld-2.5.so
2b28f208f000-2b28f2090000 rw-p 0001d000 00:12 85470                      /lib64/ld-2.5.so
2b28f2090000-2b28f20a6000 r-xp 00000000 00:12 85539                      /lib64/libpthread-2.5.so
2b28f20a6000-2b28f22a5000 ---p 00016000 00:12 85539                      /lib64/libpthread-2.5.so
2b28f22a5000-2b28f22a6000 r--p 00015000 00:12 85539                      /lib64/libpthread-2.5.so
2b28f22a6000-2b28f22a7000 rw-p 00016000 00:12 85539                      /lib64/libpthread-2.5.so
2b28f22a7000-2b28f22ab000 rw-p 2b28f22a7000 00:00 0
2b28f22ab000-2b28f22ad000 r-xp 00000000 00:12 85504                      /lib64/libdl-2.5.so
2b28f22ad000-2b28f24ad000 ---p 00002000 00:12 85504                      /lib64/libdl-2.5.so
2b28f24ad000-2b28f24ae000 r--p 00002000 00:12 85504                      /lib64/libdl-2.5.so
2b28f24ae000-2b28f24af000 rw-p 00003000 00:12 85504                      /lib64/libdl-2.5.so
2b28f24af000-2b28f24b0000 rw-p 2b28f24af000 00:00 0
2b28f24b0000-2b28f24b2000 r-xp 00000000 00:12 85549                      /lib64/libutil-2.5.so
2b28f24b2000-2b28f26b1000 ---p 00002000 00:12 85549                      /lib64/libutil-2.5.so
2b28f26b1000-2b28f26b2000 r--p 00001000 00:12 85549                      /lib64/libutil-2.5.so
2b28f26b2000-2b28f26b3000 rw-p 00002000 00:12 85549                      /lib64/libutil-2.5.so
2b28f26b3000-2b28f2735000 r-xp 00000000 00:12 85525                      /lib64/libm-2.5.so
2b28f2735000-2b28f2934000 ---p 00082000 00:12 85525                      /lib64/libm-2.5.so
2b28f2934000-2b28f2935000 r--p 00081000 00:12 85525                      /lib64/libm-2.5.so
2b28f2935000-2b28f2936000 rw-p 00082000 00:12 85525                      /lib64/libm-2.5.so
2b28f2936000-2b28f2a84000 r-xp 00000000 00:12 85480                      /lib64/libc-2.5.so
2b28f2a84000-2b28f2c84000 ---p 0014e000 00:12 85480                      /lib64/libc-2.5.so
2b28f2c84000-2b28f2c88000 r--p 0014e000 00:12 85480                      /lib64/libc-2.5.so
2b28f2c88000-2b28f2c89000 rw-p 00152000 00:12 85480                      /lib64/libc-2.5.so
2b28f2a84000-2b28f2c84000 ---p 0014e000 00:12 85480                      /lib64/libc-2.5.so
2b28f2c84000-2b28f2c88000 r--p 0014e000 00:12 85480                      /lib64/libc-2.5.so
2b28f2c88000-2b28f2c89000 rw-p 00152000 00:12 85480                      /lib64/libc-2.5.so
2b28f2c89000-2b28f2c90000 rw-p 2b28f2c89000 00:00 0
2b28f2c90000-2b28f6269000 r--p 00000000 00:12 92801                      /usr/lib/locale/locale-archive
2b28f626e000-2b28f63b6000 r-xp 00000000 00:2e 20099437                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/multiarray.so
2b28f63b6000-2b28f65b5000 ---p 00148000 00:2e 20099437                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/multiarray.so
2b28f65b5000-2b28f65c3000 rw-p 00147000 00:2e 20099437                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/multiarray.so
2b28f65c3000-2b28f65c5000 rw-p 2b28f65c3000 00:00 0
2b28f65c5000-2b28f65d7000 r-xp 00000000 00:2e 6828326                    /projects/ginsbura/local/lib/python2.7/lib-dynload/datetime.so
2b28f65d7000-2b28f67d6000 ---p 00012000 00:2e 6828326                    /projects/ginsbura/local/lib/python2.7/lib-dynload/datetime.so
2b28f67d6000-2b28f67da000 rw-p 00011000 00:2e 6828326                    /projects/ginsbura/local/lib/python2.7/lib-dynload/datetime.so
2b28f67da000-2b28f6831000 r-xp 00000000 00:2e 20099438                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/umath.so
2b28f6831000-2b28f6a30000 ---p 00057000 00:2e 20099438                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/umath.so
2b28f6a30000-2b28f6a35000 rw-p 00056000 00:2e 20099438                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/umath.so
2b28f6a35000-2b28f6a37000 rw-p 2b28f6a35000 00:00 0
2b28f6a38000-2b28f6a3d000 r-xp 00000000 00:2e 46559814                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/_dotblas.so
2b28f6a3d000-2b28f6c3c000 ---p 00005000 00:2e 46559814                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/_dotblas.so
2b28f6c3c000-2b28f6c3d000 rw-p 00004000 00:2e 46559814                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/_dotblas.so
2b28f6c50000-2b28f6c6c000 r-xp 00000000 00:12 99496                      /usr/lib64/atlas/libptf77blas.so.3.0
2b28f6c6c000-2b28f6e6c000 ---p 0001c000 00:12 99496                      /usr/lib64/atlas/libptf77blas.so.3.0
2b28f6e6c000-2b28f6e6d000 rw-p 0001c000 00:12 99496                      /usr/lib64/atlas/libptf77blas.so.3.0
2b28f6e6d000-2b28f6e8b000 r-xp 00000000 00:12 99494                      /usr/lib64/atlas/libptcblas.so.3.0
2b28f6e8b000-2b28f708b000 ---p 0001e000 00:12 99494                      /usr/lib64/atlas/libptcblas.so.3.0
2b28f708b000-2b28f708c000 rw-p 0001e000 00:12 99494                      /usr/lib64/atlas/libptcblas.so.3.0
2b28f708c000-2b28f7778000 r-xp 00000000 00:12 99485                      /usr/lib64/atlas/libatlas.so.3.0
2b28f7778000-2b28f7977000 ---p 006ec000 00:12 99485                      /usr/lib64/atlas/libatlas.so.3.0
2b28f7977000-2b28f7981000 rw-p 006eb000 00:12 99485                      /usr/lib64/atlas/libatlas.so.3.0
2b28f7981000-2b28f7a17000 r-xp 00000000 00:12 99083                      /usr/lib64/libgfortran.so.1.0.0
2b28f7a17000-2b28f7c16000 ---p 00096000 00:12 99083                      /usr/lib64/libgfortran.so.1.0.0
2b28f7c16000-2b28f7c18000 rw-p 00095000 00:12 99083                      /usr/lib64/libgfortran.so.1.0.0
2b28f7c18000-2b28f7c2b000 r-xp 00000000 00:2e 38749482                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cPickle.so
2b28f7c2b000-2b28f7e2a000 ---p 00013000 00:2e 38749482                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cPickle.so
2b28f7e2a000-2b28f7e2c000 rw-p 00012000 00:2e 38749482                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cPickle.so
2b28f7e2c000-2b28f7e30000 r-xp 00000000 00:2e 38749481                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cStringIO.so
2b28f7e30000-2b28f802f000 ---p 00004000 00:2e 38749481                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cStringIO.so
2b28f802f000-2b28f8031000 rw-p 00003000 00:2e 38749481                   /projects/ginsbura/local/lib/python2.7/lib-dynload/cStringIO.so
2b28f8031000-2b28f805a000 r-xp 00000000 00:2e 20099439                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/scalarmath.so
2b28f805a000-2b28f8259000 ---p 00029000 00:2e 20099439                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/scalarmath.so
2b28f8259000-2b28f825b000 rw-p 00028000 00:2e 20099439                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/core/scalarmath.so
2b28f825c000-2b28f8262000 r-xp 00000000 00:2e 6828330                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_collections.so
2b28f8262000-2b28f8462000 ---p 00006000 00:2e 6828330                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_collections.so
2b28f8462000-2b28f8464000 rw-p 00006000 00:2e 6828330                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_collections.so
2b28f8464000-2b28f846c000 r-xp 00000000 00:2e 6828333                    /projects/ginsbura/local/lib/python2.7/lib-dynload/operator.so
2b28f846c000-2b28f866b000 ---p 00008000 00:2e 6828333                    /projects/ginsbura/local/lib/python2.7/lib-dynload/operator.so
2b28f866b000-2b28f866d000 rw-p 00007000 00:2e 6828333                    /projects/ginsbura/local/lib/python2.7/lib-dynload/operator.so
2b28f866d000-2b28f8677000 r-xp 00000000 00:2e 6828327                    /projects/ginsbura/local/lib/python2.7/lib-dynload/itertools.so
2b28f8677000-2b28f8876000 ---p 0000a000 00:2e 6828327                    /projects/ginsbura/local/lib/python2.7/lib-dynload/itertools.so
2b28f8876000-2b28f887b000 rw-p 00009000 00:2e 6828327                    /projects/ginsbura/local/lib/python2.7/lib-dynload/itertools.so
2b28f887b000-2b28f887e000 r-xp 00000000 00:2e 6828332                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_heapq.so
2b28f887e000-2b28f8a7e000 ---p 00003000 00:2e 6828332                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_heapq.so
2b28f8a7e000-2b28f8a80000 rw-p 00003000 00:2e 6828332                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_heapq.so
2b28f8a80000-2b28f8a82000 r-xp 00000000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8a82000-2b28f8c82000 ---p 00002000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8c82000-2b28f8c83000 rw-p 00002000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8c83000-2b28f8c87000 r-xp 00000000 00:2e 6828325                    /projects/ginsbura/local/lib/python2.7/lib-dynload/time.so
2b28f8a80000-2b28f8a82000 r-xp 00000000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8a82000-2b28f8c82000 ---p 00002000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8c82000-2b28f8c83000 rw-p 00002000 00:2e 6828335                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_functools.so
2b28f8c83000-2b28f8c87000 r-xp 00000000 00:2e 6828325                    /projects/ginsbura/local/lib/python2.7/lib-dynload/time.so
2b28f8c87000-2b28f8e86000 ---p 00004000 00:2e 6828325                    /projects/ginsbura/local/lib/python2.7/lib-dynload/time.so
2b28f8e86000-2b28f8e88000 rw-p 00003000 00:2e 6828325                    /projects/ginsbura/local/lib/python2.7/lib-dynload/time.so
2b28f8e88000-2b28f8e90000 r-xp 00000000 00:2e 6828323                    /projects/ginsbura/local/lib/python2.7/lib-dynload/math.so
2b28f8e90000-2b28f908f000 ---p 00008000 00:2e 6828323                    /projects/ginsbura/local/lib/python2.7/lib-dynload/math.so
2b28f908f000-2b28f9091000 rw-p 00007000 00:2e 6828323                    /projects/ginsbura/local/lib/python2.7/lib-dynload/math.so
2b28f9092000-2b28f9153000 rw-p 2b28f9092000 00:00 0
2b28f9153000-2b28f9159000 r-xp 00000000 00:2e 41780261                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/lib/_compiled_base.so
2b28f9159000-2b28f9358000 ---p 00006000 00:2e 41780261                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/lib/_compiled_base.so
2b28f9358000-2b28f9359000 rw-p 00005000 00:2e 41780261                   /projects/ginsbura/local/lib/python2.7/site-packages/numpy/lib/_compiled_base.so
2b28f9359000-2b28f935e000 r-xp 00000000 00:2e 828549                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
2b28f935e000-2b28f955e000 ---p 00005000 00:2e 828549                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
2b28f955e000-2b28f955f000 rw-p 00005000 00:2e 828549                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
2b28f9572000-2b28f9968000 r-xp 00000000 00:12 99492                      /usr/lib64/atlas/liblapack.so.3.0
2b28f9968000-2b28f9b68000 ---p 003f6000 00:12 99492                      /usr/lib64/atlas/liblapack.so.3.0
2b28f9b68000-2b28f9b6b000 rw-p 003f6000 00:12 99492                      /usr/lib64/atlas/liblapack.so.3.0
2b28f9b6b000-2b28f9c6f000 rw-p 2b28f9b6b000 00:00 0
2b28f9c6f000-2b28f9c7c000 r-xp 00000000 00:12 85512                      /lib64/libgcc_s-4.1.2-20080825.so.1
2b28f9c7c000-2b28f9e7c000 ---p 0000d000 00:12 85512                      /lib64/libgcc_s-4.1.2-20080825.so.1
2b28f9e7c000-2b28f9e7d000 rw-p 0000d000 00:12 85512                      /lib64/libgcc_s-4.1.2-20080825.so.1
2b28f9e7d000-2b28f9e99000 r-xp 00000000 00:12 99490                      /usr/lib64/atlas/libf77blas.so.3.0
2b28f9e99000-2b28fa099000 ---p 0001c000 00:12 99490                      /usr/lib64/atlas/libf77blas.so.3.0
2b28fa099000-2b28fa09a000 rw-p 0001c000 00:12 99490                      /usr/lib64/atlas/libf77blas.so.3.0
2b28fa09a000-2b28fa0b8000 r-xp 00000000 00:12 99487                      /usr/lib64/atlas/libcblas.so.3.0
2b28fa0b8000-2b28fa2b8000 ---p 0001e000 00:12 99487                      /usr/lib64/atlas/libcblas.so.3.0
2b28fa2b8000-2b28fa2b9000 rw-p 0001e000 00:12 99487                      /usr/lib64/atlas/libcblas.so.3.0
2b28fa2b9000-2b28fa2bb000 r-xp 00000000 00:2e 38749477                   /projects/ginsbura/local/lib/python2.7/lib-dynload/grp.so
2b28fa2bb000-2b28fa4ba000 ---p 00002000 00:2e 38749477                   /projects/ginsbura/local/lib/python2.7/lib-dynload/grp.so
2b28fa4ba000-2b28fa4bb000 rw-p 00001000 00:2e 38749477                   /projects/ginsbura/local/lib/python2.7/lib-dynload/grp.so
2b28fa4bb000-2b28fa4c4000 r-xp 00000000 00:2e 828542                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/fft/fftpack_lite.so
2b28fa4c4000-2b28fa6c4000 ---p 00009000 00:2e 828542                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/fft/fftpack_lite.so
2b28fa6c4000-2b28fa6c5000 rw-p 00009000 00:2e 828542                     /projects/ginsbura/local/lib/python2.7/site-packages/numpy/fft/fftpack_lite.so
2b28fa6c5000-2b28fa6ca000 r-xp 00000000 00:2e 6828324                    /projects/ginsbura/local/lib/python2.7/lib-dynload/strop.so
2b28fa6ca000-2b28fa8c9000 ---p 00005000 00:2e 6828324                    /projects/ginsbura/local/lib/python2.7/lib-dynload/strop.so
2b28fa8c9000-2b28fa8cb000 rw-p 00004000 00:2e 6828324                    /projects/ginsbura/local/lib/python2.7/lib-dynload/strop.so
2b28fa8cb000-2b28fa91c000 r-xp 00000000 00:2e 9337629                    /projects/ginsbura/local/lib/python2.7/site-packages/numpy/random/mtrand.so
2b28fa91c000-2b28fab1b000 ---p 00051000 00:2e 9337629                    /projects/ginsbura/local/lib/python2.7/site-packages/numpy/random/mtrand.so
2b28fab1b000-2b28fab51000 rw-p 00050000 00:2e 9337629                    /projects/ginsbura/local/lib/python2.7/site-packages/numpy/random/mtrand.so
2b28fab51000-2b28fab52000 rw-p 2b28fab51000 00:00 0
2b28fab52000-2b28fab75000 r-xp 00000000 00:2e 5677765                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_ctypes.so
2b28fab75000-2b28fad74000 ---p 00023000 00:2e 5677765                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_ctypes.so
2b28fad74000-2b28fad79000 rw-p 00022000 00:2e 5677765                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_ctypes.so
2b28fad79000-2b28fad80000 r-xp 00000000 00:2e 6828319                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_struct.so
2b28fad80000-2b28faf80000 ---p 00007000 00:2e 6828319                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_struct.so
2b28faf80000-2b28faf82000 rw-p 00007000 00:2e 6828319                    /projects/ginsbura/local/lib/python2.7/lib-dynload/_struct.so
2b28faf82000-2b28faf83000 r-xs 00000000 00:12 1308474                    /tmp/ffiXRKPiN (deleted)
2b28faf83000-2b28faf84000 rw-s 00000000 00:12 1308474                    /tmp/ffiXRKPiN (deleted)
2b28faf84000-2b28fb180000 rw-p 2b28faf84000 00:00 0
7fff32ce2000-7fff32cf7000 rw-p 7ffffffe9000 00:00 0                      [stack]
7fff32d7e000-7fff32d82000 r-xp 7fff32d7e000 00:00 0                      [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vsyscall]
Aborted

更多详细信息:我尝试按照此处列出的说明操作:http://docs.scipy.org/doc/numpy/user/install.html#fortran-abi-mismatch。我的ATLAS显然是使用gfortran构建的,因此我使用了python setup.py build --fcompiler=gnu95,但仍然会收到错误。

$ ldd ./build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so
    linux-vdso.so.1 =>  (0x00007fff51dfc000)
    liblapack.so.3 => /usr/lib64/atlas/liblapack.so.3 (0x00002b79a021c000)
    libptf77blas.so.3 => /usr/lib64/atlas/libptf77blas.so.3 (0x00002b79a091a000)
    libptcblas.so.3 => /usr/lib64/atlas/libptcblas.so.3 (0x00002b79a0b37000)
    libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00002b79a0d56000)
    libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x00002b79a164c000)
    libm.so.6 => /lib64/libm.so.6 (0x00002b79a18e3000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b79a1b66000)
    libc.so.6 => /lib64/libc.so.6 (0x00002b79a1d75000)
    libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00002b79a20cd000)
    libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00002b79a22ea000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b79a250a000)
    /lib64/ld-linux-x86-64.so.2 (0x00002b799fde4000)

0 个答案:

没有答案