试图摆脱heketi中的孤立卷会导致错误,没有原因

时间:2019-11-25 21:39:15

标签: kubernetes glusterfs

我正在尝试摆脱heketi中的一堆孤立卷。尝试时,出现“错误”,然后显示有关我刚尝试删除的卷的信息,序列化为JSON。没有别的了。我试图挖掘日志,但它们什么也没透露。

这是我用来尝试删除卷的命令:

heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  volume delete 22f1a960651f0f16ada20a15d68c7dd6
Error: {"size":30,"name":"vol_22f1a960651f0f16ada20a15d68c7dd6","durability":{"type":"none","replicate":{},"disperse":{}},"gid":2008,"glustervolumeoptions":["","cluster.post-op-delay-secs 0"," performance.client-io-threads off"," performance.open-behind off"," performance.readdir-ahead off"," performance.read-ahead off"," performance.stat-prefetch off"," performance.write-behind off"," performance.io-cache off"," cluster.consistent-metadata on"," performance.quick-read off"," performance.strict-o-direct on"," storage.health-check-interval 0",""],"snapshot":{"enable":true,"factor":1},"id":"22f1a960651f0f16ada20a15d68c7dd6","cluster":"e924a50aa93d9eae1132c60eb1f36310","mount":{"glusterfs":{"hosts":["<SECRET>"],"device":"<SECRET>:vol_22f1a960651f0f16ada20a15d68c7dd6","options":{"backup-volfile-servers":""}}},"blockinfo":{},"bricks":[{"id":"0f4c6d7f605e9368bfe3dc7cc117b69a","path":"/var/lib/heketi/mounts/vg_970f0faf60f8dfc6f6a0d6bd25bdea7c/brick_0f4c6d7f605e9368bfe3dc7cc117b69a/brick","device":"970f0faf60f8dfc6f6a0d6bd25bdea7c","node":"107894a855c9d2c34509b18272e6c298","volume":"22f1a960651f0f16ada20a15d68c7dd6","size":31457280}]}

请注意,第二行仅包含Error,然后是有关序列化为json的卷的信息。

该卷在gluster中不存在。我使用以下命令来验证该卷不再存在:

kubectl -n default  exec -t -i glusterfs-rgz9g bash
gluster volume info 
<shows volume i did not delete>

Kubernetes没有显示PersistentVolumeClaim或PersistentVolume:

kubectl get pvc -A
No resources found.
kubectl get pv -A
No resources found.

我尝试查看heketi日志,但它仅报告该卷的GET

kubectl -n default logs   heketi-56f678775c-nrbwd
[negroni] 2019-11-25T21:29:19Z | 200 |   1.407715ms | <SECRET>:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6
[negroni] 2019-11-25T21:29:19Z | 200 |   1.111984ms | <SECRET>:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6
[negroni] 2019-11-25T21:29:19Z | 200 |   1.540357ms | <SECRET>:8080 | GET /volumes/22f1a960651f0f16ada20a15d68c7dd6 

我尝试设置更详细的日志级别,但设置不成立:

heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  loglevel set debug
Server log level updated
heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret ${KEY}  loglevel get
info

我的CLI使用

heketi-cli -v
heketi-cli v9.0.0

Heketi服务器正在运行:

kubectl -n default  exec -t -i  heketi-56f678775c-nrbwd bash
heketi -v
Heketi v9.0.0-124-gc2e2a4ab

基于日志,我认为heketi-cli出现问题,然后再也没有实际将POST或DELETE请求发送到heketi服务器。

如何进行调试?在这一点上,我唯一的解决方法是重新创建群集,但是我想避免这种情况,特别是在类似情况再次出现时。

1 个答案:

答案 0 :(得分:0)

好像heketi-cli中有一个错误,因为如果我使用ruby和curl手动创建请求,则可以删除该卷:

TOKEN=$(ruby makeToken.rb DELETE /volumes/22f1a960651f0f16ada20a15d68c7dd6)
curl -X DELETE -H "Authorization: Bearer $TOKEN" http://10.233.21.178:8080/volumes/22f1a960651f0f16ada20a15d68c7dd6

有关如何生成jwt令牌的信息,请参见https://github.com/heketi/heketi/blob/master/docs/api/api.md#authentication-model

我手动创建了请求,期望得到更好的错误消息,命令行工具将其吞下。原来cli确实被破坏了。

用于制作jwt令牌的ruby代码(makeToken.rb)。您需要填写通行证和服务器。

#!/usr/bin/env ruby

require 'jwt'
require 'digest'

user = "admin"
pass = "<SECRET>"
server = "http://localhost:8080/"

method = "#{ARGV[0]}"
uri = "#{ARGV[1]}"

payload = {}

headers = {
  iss: 'admin',
  iat: Time.now.to_i,
  exp: Time.now.to_i + 600,
  qsh: Digest::SHA256.hexdigest("#{method}&#{uri}")
}

token = JWT.encode headers, pass, 'HS256'
print("token for\n")
print("#{method} #{uri}\n")
print("#{token}\n")