istio收到“ RBAC：访问被拒绝”，甚至已检查servicerolebinding是否被允许

时间：2020-08-07 07:19:54

标签： authentication istio rbac kubeflow openid-dex

我一直在与istio斗争...所以我在这里寻求专家的帮助！

背景

我正在尝试通过 dex 为多租户部署我的kubeflow应用程序。 用kubeflow offical document

引用manifest file from github

这是组件/版本信息的列表

  • 我正在GKE上运行kubernetes 1.15
  • Istio 1.1.6在kubeflow中用于服务方法
  • 尝试为ML部署kubeflow 1.0
  • 为身份验证部署了dex 1.0

使用清单文件，我成功在集群上部署了kubeflow。这就是我所做的。

  • 在集群上部署kubeflow应用程序
  • 使用OIDC服务部署Dex以启用对Google Oauth2.0的身份验证
  • 启用RBAC
  • 创建使节过滤器以将标头“ kubeflow-userid”追加为登录用户

这是第3步和第4步的验证 检查RBAC是否已启用，并为kubeflow-userid添加了envoyfilter 

[root@gke-client-tf leilichao]# k get clusterrbacconfigs -o yaml
apiVersion: v1
items:
- apiVersion: rbac.istio.io/v1alpha1
  kind: ClusterRbacConfig
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"rbac.istio.io/v1alpha1","kind":"ClusterRbacConfig","metadata":{"annotations":{},"name":"default"},"spec":{"mode":"ON"}}
    creationTimestamp: "2020-07-04T01:28:52Z"
    generation: 2
    name: default
    resourceVersion: "5986075"
    selfLink: /apis/rbac.istio.io/v1alpha1/clusterrbacconfigs/default
    uid: db70920e-f364-40ec-a93b-a3364f88650f
  spec:
    mode: "ON"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
[root@gke-client-tf leilichao]# k get envoyfilter -n istio-system -o yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1alpha3
  kind: EnvoyFilter
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.istio.io/v1alpha3","kind":"EnvoyFilter","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"oidc-authservice","app.kubernetes.io/instance":"oidc-authservice-v1.0.0","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"oidc-authservice","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v1.0.0"},"name":"authn-filter","namespace":"istio-system"},"spec":{"filters":[{"filterConfig":{"httpService":{"authorizationRequest":{"allowedHeaders":{"patterns":[{"exact":"cookie"},{"exact":"X-Auth-Token"}]}},"authorizationResponse":{"allowedUpstreamHeaders":{"patterns":[{"exact":"kubeflow-userid"}]}},"serverUri":{"cluster":"outbound|8080||authservice.istio-system.svc.cluster.local","failureModeAllow":false,"timeout":"10s","uri":"http://authservice.istio-system.svc.cluster.local"}},"statusOnError":{"code":"GatewayTimeout"}},"filterName":"envoy.ext_authz","filterType":"HTTP","insertPosition":{"index":"FIRST"},"listenerMatch":{"listenerType":"GATEWAY"}}],"workloadLabels":{"istio":"ingressgateway"}}}
    creationTimestamp: "2020-07-04T01:40:43Z"
    generation: 1
    labels:
      app.kubernetes.io/component: oidc-authservice
      app.kubernetes.io/instance: oidc-authservice-v1.0.0
      app.kubernetes.io/managed-by: kfctl
      app.kubernetes.io/name: oidc-authservice
      app.kubernetes.io/part-of: kubeflow
      app.kubernetes.io/version: v1.0.0
    name: authn-filter
    namespace: istio-system
    resourceVersion: "4715289"
    selfLink: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/envoyfilters/authn-filter
    uid: e599ba82-315a-4fc1-9a5d-e8e35d93ca26
  spec:
    filters:
    - filterConfig:
        httpService:
          authorizationRequest:
            allowedHeaders:
              patterns:
              - exact: cookie
              - exact: X-Auth-Token
          authorizationResponse:
            allowedUpstreamHeaders:
              patterns:
              - exact: kubeflow-userid
          serverUri:
            cluster: outbound|8080||authservice.istio-system.svc.cluster.local
            failureModeAllow: false
            timeout: 10s
            uri: http://authservice.istio-system.svc.cluster.local
        statusOnError:
          code: GatewayTimeout
      filterName: envoy.ext_authz
      filterType: HTTP
      insertPosition:
        index: FIRST
      listenerMatch:
        listenerType: GATEWAY
    workloadLabels:
      istio: ingressgateway
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

RBAC问题分析

完成部署后。我执行了以下功能测试：

  • 我可以使用google oauth用我的Google帐户登录
  • 我能够创建自己的个人资料/名称空间
  • 我能够创建一个笔记本服务器
  • 但是我不能连接到笔记本服务器

RBAC问题调查

在kubeflow上成功创建笔记本服务器并尝试连接笔记本服务器后，出现“ RBAC：访问被拒绝”错误。 我设法更新了特使日志级别，并在下面获取日志。

[2020-08-06 13:32:43.290][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:64] checking request: remoteAddress: 10.1.1.2:58012, localAddress: 10.1.2.66:8888, ssl: none, headers: ':authority', 'compliance-kf-system.ml'
':path', '/notebook/roger-l-c-lei/aug06/'
':method', 'GET'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en,zh-CN;q=0.9,zh;q=0.8'
'cookie', 'authservice_session=MTU5NjY5Njk0MXxOd3dBTkZvMldsVllVMUZPU0VaR01sSk5RVlJJV2xkRFVrRTFTVUl5V0RKV1EwdEhTMU5QVjFCVlUwTkpSVFpYUlVoT1RGVlBUa0U9fN3lPBXDDSZMT9MTJRbG8jv7AtblKTE3r84ayeCYuKOk; _xsrf=2|1e6639f2|10d3ea0a904e0ae505fd6425888453f8|1596697030'
'referer', 'http://compliance-kf-system.ml/jupyter/'
'upgrade-insecure-requests', '1'
'x-forwarded-for', '10.10.10.230'
'x-forwarded-proto', 'http'
'x-request-id', 'babbf884-4cec-93fd-aea6-2fc60d3abb83'
'kubeflow-userid', 'roger.l.c.lei@XXXX.com'
'x-istio-attributes', 'CjAKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEg8SDXJvZ2VyLWwtYy1sZWkKIwoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEgcSBWF1ZzA2Ck4KCnNvdXJjZS51aWQSQBI+a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTg5Y2Q0YmQ0Yy1kdnF3dC5pc3Rpby1zeXN0ZW0KQQoXZGVzdGluYXRpb24uc2VydmljZS51aWQSJhIkaXN0aW86Ly9yb2dlci1sLWMtbGVpL3NlcnZpY2VzL2F1ZzA2CkMKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBInEiVhdWcwNi5yb2dlci1sLWMtbGVpLnN2Yy5jbHVzdGVyLmxvY2Fs'
'x-envoy-expected-rq-timeout-ms', '300000'
'x-b3-traceid', '3bf35cca1f7b75e7a42a046b1c124b1f'
'x-b3-spanid', 'a42a046b1c124b1f'
'x-b3-sampled', '1'
'x-envoy-original-path', '/notebook/roger-l-c-lei/aug06/'
'content-length', '0'
'x-envoy-internal', 'true'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
  }
}

[2020-08-06 13:32:43.290][26][debug][rbac] [external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:108] enforced denied

从源代码看，允许的函数似乎返回false，因此给出了“ RBAC：拒绝访问”响应。

  if (engine.has_value()) {
    if (engine->allowed(*callbacks_->connection(), headers,
                        callbacks_->streamInfo().dynamicMetadata(), nullptr)) {
      ENVOY_LOG(debug, "enforced allowed");
      config_->stats().allowed_.inc();
      return Http::FilterHeadersStatus::Continue;
    } else {
      ENVOY_LOG(debug, "enforced denied");
      callbacks_->sendLocalReply(Http::Code::Forbidden, "RBAC: access denied", nullptr,
                                 absl::nullopt);
      config_->stats().denied_.inc();
      return Http::FilterHeadersStatus::StopIteration;
    }
  }

我对转储的特使进行了搜索，看来规则应该允许任何带有标头密钥的请求作为我的邮件地址。现在，我可以确认我已经在日志上方的标题中找到了它。

{
 "name": "envoy.filters.http.rbac",
 "config": {
  "rules": {
   "policies": {
    "ns-access-istio": {
     "permissions": [
      {
       "and_rules": {
        "rules": [
         {
          "any": true
         }
        ]
       }
      }
     ],
     "principals": [
      {
       "and_ids": {
        "ids": [
         {
          "header": {
           "exact_match": "roger.l.c.lei@XXXX.com"
          }
         }
        ]
       }
      }
     ]
    }
   }
  }
 }
}

了解到，用于验证RBAC身份验证的特使配置来自此配置。然后它通过混音器分发到Sidecar，日志和代码将我带到servicerolebinding的rbac.istio.io配置。

[root@gke-client-tf leilichao]# k get servicerolebinding -n roger-l-c-lei -o yaml
apiVersion: v1
items:
- apiVersion: rbac.istio.io/v1alpha1
  kind: ServiceRoleBinding
  metadata:
    annotations:
      role: admin
      user: roger.l.c.lei@XXXX.com
    creationTimestamp: "2020-07-04T01:35:30Z"
    generation: 5
    name: owner-binding-istio
    namespace: roger-l-c-lei
    ownerReferences:
    - apiVersion: kubeflow.org/v1
      blockOwnerDeletion: true
      controller: true
      kind: Profile
      name: roger-l-c-lei
      uid: 689c9f04-08a6-4c51-a1dc-944db1a66114
    resourceVersion: "23201026"
    selfLink: /apis/rbac.istio.io/v1alpha1/namespaces/roger-l-c-lei/servicerolebindings/owner-binding-istio
    uid: bbbffc28-689c-4099-837a-87a2feb5948f
  spec:
    roleRef:
      kind: ServiceRole
      name: ns-access-istio
    subjects:
    - properties:
        request.headers[]: roger.l.c.lei@XXXX.com
  status: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

我想尝试更新此ServiceRoleBinding以验证某些假设，因为我无法调试特使源代码，并且没有足够的日志显示为什么“ allow”方法确切返回false。

但是我发现自己无法更新servicerolebinding。每当我完成编辑后，它都会恢复为原始版本。

我发现有一个istio-galleyvalidatingAdmissionConfiguration（下面的代码块）监视着这些istio rbac资源。

[root@gke-client-tf leilichao]# k get validatingwebhookconfigurations istio-galley -oyaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: "2020-08-04T15:00:59Z"
  generation: 1
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    istio: galley
    release: istio
  name: istio-galley
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: istio-galley
    uid: 11fef012-4145-49ac-a43c-2e1d0a460ea4
  resourceVersion: "22484680"
  selfLink: /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/istio-galley
  uid: 6f485e28-3b5a-4a3b-b31f-a5c477c82619
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: 
    .
    .
    .
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitpilot
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: pilot.validation.istio.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - httpapispecs
    - httpapispecbindings
    - quotaspecs
    - quotaspecbindings
    scope: '*'
  - apiGroups:
    - rbac.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    scope: '*'
  - apiGroups:
    - authentication.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
    scope: '*'
  - apiGroups:
    - networking.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - destinationrules
    - envoyfilters
    - gateways
    - serviceentries
    - sidecars
    - virtualservices
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: 
    .
    .
    .
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitmixer
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: mixer.validation.istio.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - rules
    - attributemanifests
    - circonuses
    - deniers
    - fluentds
    - kubernetesenvs
    - listcheckers
    - memquotas
    - noops
    - opas
    - prometheuses
    - rbacs
    - solarwindses
    - stackdrivers
    - cloudwatches
    - dogstatsds
    - statsds
    - stdios
    - apikeys
    - authorizations
    - checknothings
    - listentries
    - logentries
    - metrics
    - quotas
    - reportnothings
    - tracespans
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30

长短行程

对于这个istio问题，我已经花了超过2个星期的时间。我敢肯定，有很多人会为尝试在k8s上拍摄istio带来麻烦。任何建议都欢迎！ 这是我对问题的理解方式，如果我写错了，请纠正我：

  • 日志证据表明，rbac规则不允许我访问资源
  • 我需要更新rbac规则
  • 规则根据ServiceRoleBinding由混合器分发到特使容器
  • 所以我需要更新ServiceRoleBinding
  • 我无法更新ServiceRoleBinding，因为正在验证的进入Webhook或istio混合器都阻止了我这样做

我遇到了以下问题

即使删除验证webhook，我也无法更新ServiceRoleBinding

我试图删除此验证Webhook来更新servicerolebinding。保存编辑后，资源将立即恢复。 验证webhook实际上是从configmap自动生成的，因此我必须对其进行更新以更新webhook。

厨房中是否存在混合器用来分发配置的高速缓存

我找不到任何相关的日志记录，表明rbac.istio.io资源受到istio-system名称空间中的任何服务的保护/验证。

如何获取MIXER的日志

我需要了解哪个组件可以准确控制该政策。我设法更新了日志级别，但是找不到有用的东西

最重要的是如何调试特使容器

我需要调试envoy应用，以了解为什么allow函数返回false。 如果我们无法轻松调试它。是否有一个文档可以让我更新代码以添加更多日志并为GCR构建新图像，因此我可以再次运行并基于日志查看幕后情况。

2 个答案:

答案 0 :(得分：1)

自从我在问题上取得了一些进展以来，回答了我自己的问题。

即使删除验证webhook，我也无法更新ServiceRoleBinding

这是因为ServiceRoleBinding实际上是由配置文件控制器在kubeflow命名空间中而不是验证webhook 中生成/监视/管理的。

我遇到了rbac问题，因为基于配置文件清单文件夹中的params.yaml，规则生成为

request.headers[]: roger.l.c.lei@XXXX.com

代替

request.headers[kubeflow-userid]: roger.l.c.lei@XXXX.com

由于我将值错误地配置为空白，而不是params.yaml中的 userid-header = kubeflow-userid

答案 1 :(得分：0)

检查您的应用命名空间中的 authorizationpolicy 资源。

相关问题
最新问题