2019-03-17 09:14:53.422958: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 17.36GiB. Current allocation summary follows.
2019-03-17 09:14:53.423036: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): Total Chunks: 27, Chunks in use: 27. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 108B client-requested in use in bin.
2019-03-17 09:14:53.423056: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423071: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-03-17 09:14:53.423086: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048): Total Chunks: 6, Chunks in use: 6. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 11.7KiB client-requested in use in bin.
2019-03-17 09:14:53.423099: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423112: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423124: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423137: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423149: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423163: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072): Total Chunks: 6, Chunks in use: 6. 1.49MiB allocated for chunks. 1.49MiB in use in bin. 1.49MiB client-requested in use in bin.
2019-03-17 09:14:53.423175: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423188: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423200: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423212: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423224: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423237: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423249: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423261: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423275: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864): Total Chunks: 11, Chunks in use: 10. 1.33GiB allocated for chunks. 1.21GiB in use in bin. 1.21GiB client-requested in use in bin.
2019-03-17 09:14:53.423290: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423304: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 8.48GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-03-17 09:14:53.423319: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 17.36GiB was 256.00MiB, Chunk State:
2019-03-17 09:14:53.423357: I tensorflow/core/common_runtime/bfc_allocator.cc:649] Size: 8.48GiB | Requested Size: 124.23MiB | in_use: 0, prev: Size: 124.23MiB | Requested Size: 124.23MiB | in_use: 1
2019-03-17 09:14:53.423370: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da00000 of size 1280
2019-03-17 09:14:53.423378: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da00500 of size 256
2019-03-17 09:14:53.423386: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da00600 of size 256
2019-03-17 09:14:53.423393: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da00700 of size 260608
2019-03-17 09:14:53.423401: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da40100 of size 2048
2019-03-17 09:14:53.423408: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da40900 of size 260608
2019-03-17 09:14:53.423415: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da80300 of size 256
2019-03-17 09:14:53.423422: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da80400 of size 256
2019-03-17 09:14:53.423430: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da80500 of size 130266112
2019-03-17 09:14:53.423437: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102156bb900 of size 130266112
2019-03-17 09:14:53.423444: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d2f6d00 of size 2048
2019-03-17 09:14:53.423451: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d2f7500 of size 130266112
2019-03-17 09:14:53.423458: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32900 of size 256
2019-03-17 09:14:53.423465: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32a00 of size 256
2019-03-17 09:14:53.423472: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32b00 of size 256
2019-03-17 09:14:53.423478: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32c00 of size 256
2019-03-17 09:14:53.423485: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32d00 of size 256
2019-03-17 09:14:53.423492: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32e00 of size 256
2019-03-17 09:14:53.423499: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f32f00 of size 256
2019-03-17 09:14:53.423506: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33000 of size 256
2019-03-17 09:14:53.423513: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33100 of size 256
2019-03-17 09:14:53.423519: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33200 of size 256
2019-03-17 09:14:53.423526: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33300 of size 256
2019-03-17 09:14:53.423532: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33400 of size 256
2019-03-17 09:14:53.423539: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33500 of size 256
2019-03-17 09:14:53.423546: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f33600 of size 260608
2019-03-17 09:14:53.423553: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73000 of size 256
2019-03-17 09:14:53.423560: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73100 of size 256
2019-03-17 09:14:53.423567: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73200 of size 2048
2019-03-17 09:14:53.423573: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73a00 of size 256
2019-03-17 09:14:53.423580: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73b00 of size 256
2019-03-17 09:14:53.423587: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73c00 of size 256
2019-03-17 09:14:53.423593: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73d00 of size 256
2019-03-17 09:14:53.423600: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73e00 of size 256
2019-03-17 09:14:53.423606: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10224f73f00 of size 256
2019-03-17 09:14:53.423613: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022cb6dd00 of size 2048
2019-03-17 09:14:53.423620: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022cb6e500 of size 260608
2019-03-17 09:14:53.423626: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022cbadf00 of size 256
2019-03-17 09:14:53.423633: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022cbae000 of size 256
2019-03-17 09:14:53.423640: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022cbae100 of size 130266112
2019-03-17 09:14:53.423646: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102347e9500 of size 130266112
2019-03-17 09:14:53.423653: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1023c424900 of size 260608
2019-03-17 09:14:53.423660: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1023c464300 of size 130266112
2019-03-17 09:14:53.423666: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024409f700 of size 130266112
2019-03-17 09:14:53.423673: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024bcdab00 of size 2048
2019-03-17 09:14:53.423680: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024bcdb300 of size 2048
2019-03-17 09:14:53.423686: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024bcdbb00 of size 260608
2019-03-17 09:14:53.423693: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024bd1b500 of size 130266112
2019-03-17 09:14:53.423700: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10253956900 of size 130266112
2019-03-17 09:14:53.423706: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025b591d00 of size 130266112
2019-03-17 09:14:53.423714: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10224f74000 of size 129998080
2019-03-17 09:14:53.423720: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x102631cd100 of size 9101723648
2019-03-17 09:14:53.423727: I tensorflow/core/common_runtime/bfc_allocator.cc:676] Summary of in-use Chunks by size:
2019-03-17 09:14:53.423736: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 27 Chunks of size 256 totalling 6.8KiB
2019-03-17 09:14:53.423744: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB
2019-03-17 09:14:53.423752: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 2048 totalling 12.0KiB
2019-03-17 09:14:53.423760: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 260608 totalling 1.49MiB
2019-03-17 09:14:53.423767: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 10 Chunks of size 130266112 totalling 1.21GiB
2019-03-17 09:14:53.423775: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 1.21GiB
2019-03-17 09:14:53.423785: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats:
Limit: 10535967129
InUse: 1304245248
MaxInUse: 1747813376
NumAllocs: 27270
MaxAllocSize: 130266112
2019-03-17 09:14:53.423801: W tensorflow/core/common_runtime/bfc_allocator.cc:277] **************______________________________________________________________________________________
2019-03-17 09:14:53.423952: W tensorflow/core/framework/op_kernel.cc:1192] Internal: Dst tensor is not initialized.
[[Node: _arg_input_R_0_0/_17 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_51__arg_input_R_0_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
Traceback (most recent call last):
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: _arg_input_R_0_0/_17 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_51__arg_input_R_0_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-b237ea7850d0>", line 1, in <module>
runfile('/home/wangdong/桌面/工程项目目录Spyter-tensorblow/第二篇论文研究/原论文数据处理方式－扩展实验-10M /变处理结构 (10M)/AutoRec.py', wdir='/home/wangdong/桌面/工程项目目录Spyter-tensorblow/第二篇论文研究/原论文数据处理方式－扩展实验-10M /变处理结构 (10M)')
File "/home/wangdong/下载/pycharm-2018.2.1/helpers/pydev/_pydev_bundle/pydev_umd.py", line 194, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/home/wangdong/下载/pycharm-2018.2.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/wangdong/桌面/工程项目目录Spyter-tensorblow/第二篇论文研究/原论文数据处理方式－扩展实验-10M /变处理结构 (10M)/AutoRec.py", line 177, in <module>
cost_gg, decoder = sess.run([cost, Decoder], feed_dict={input_R: test_R, input_mask_R: test_mask_R})
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/wangdong/anaconda3/envs/tensorflow-gpu/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: _arg_input_R_0_0/_17 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_51__arg_input_R_0_0", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]