我的glusterfs使用heketi安装在k8s中。以前效果不错，但是今天我突然无法创建该音量。错误如下：
[root@localhost grafana]# heketi-cli -s http://10.10.11.181:8080 --user admin --secret runsdata-gf-admin volume create --size=1 --replica=3 --clusters=8fecf14f0a49ea787aa5a35816237cd1 --name=runsdata-monitoring-grafana-data
Error: volume start: runsdata-monitoring-grafana-data: failed: Commit failed on 172.16.5.152. Please check log file for details.
然后我检查glusterfs pod日志，该日志的一部分如下所示：
[2020-03-12 07:04:34.004415] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2020-03-12 07:04:34.340139] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-03-12 07:04:34.340226] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-03-12 07:04:34.340265] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-03-12 07:04:34.340833] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-03-12 07:04:34.340882] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-03-12 07:04:34.368301] I [MSGID: 106618] [glusterd-svc-helper.c:901:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=runsdata-monitoring-grafana-data) to existing process with pid 198
[2020-03-12 07:04:34.368344] I [MSGID: 106498] [glusterd-svc-helper.c:747:__glusterd_send_svc_configure_req] 0-management: not connected yet
The message "I [MSGID: 106498] [glusterd-svc-helper.c:747:__glusterd_send_svc_configure_req] 0-management: not connected yet" repeated 14 times between [2020-03-12 07:04:34.368344] and [2020-03-12 07:04:48.370243]
[2020-03-12 07:04:49.370380] W [MSGID: 106617] [glusterd-svc-helper.c:932:glusterd_attach_svc] 0-glusterd: attach failed for glustershd(volume=runsdata-monitoring-grafana-data)
[2020-03-12 07:04:49.370747] E [MSGID: 106048] [glusterd-shd-svc.c:483:glusterd_shdsvc_start] 0-glusterd: Failed to attach shd svc(volume=runsdata-monitoring-grafana-data) to pid=198
[2020-03-12 07:04:49.371163] E [MSGID: 106122] [glusterd-mgmt.c:326:gd_mgmt_v3_commit_fn] 0-management: Volume start commit failed.
[2020-03-12 07:04:49.371193] E [MSGID: 106122] [glusterd-mgmt.c:1582:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Start on local node
[2020-03-12 07:04:49.371217] E [MSGID: 106122] [glusterd-mgmt.c:2336:glusterd_mgmt_v3_initiate_all_phases] 0-management: Commit Op Failed
[2020-03-12 07:04:49.623179] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0x3c63d) [0x7f64aec6e63d] -->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0xe39d5) [0x7f64aed159d5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f64b49a8025] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=runsdata-monitoring-grafana-data --last=no
[2020-03-12 07:04:49.632675] E [run.c:242:runner_log] (-->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0x3c63d) [0x7f64aec6e63d] -->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0xe3933) [0x7f64aed15933] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f64b49a8025] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=runsdata-monitoring-grafana-data --last=no
[2020-03-12 07:04:49.632877] I [MSGID: 106542] [glusterd-utils.c:8659:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 2505
[2020-03-12 07:04:50.633158] I [socket.c:864:__socket_shutdown] 0-management: intentional socket shutdown(20)
[2020-03-12 07:04:50.770813] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped
[2020-03-12 07:04:50.770958] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped
[2020-03-12 07:04:50.771054] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2020-03-12 07:04:50.771705] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped
[2020-03-12 07:04:50.771790] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped
[2020-03-12 07:04:50.804019] I [MSGID: 106619] [glusterd-svc-helper.c:954:glusterd_detach_svc] 0-management: removing svc glustershd (volume=runsdata-monitoring-grafana-data) from existing process with pid 198
我从错误日志中看不到是什么导致了故障。
其他信息： 我的gluster群集的每个节点在VG中仍然有4G空间。 （172.16.5.152）的自我修复未在线。如何在线制作？