阿里云 ACK 部署 logging 组件无法修改 data-elasticsearch-logging-discovery-0 的 pvc 大小

poetL

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

ack 集群

Kubernetes版本信息
Client Version: version.Info{Major:“1”, Minor:“24”, GitVersion:“v1.24.0”, GitCommit:“4ce5a8954017644c5420bae81d72b09b735c21f0”, GitTreeState:“clean”, BuildDate:“2022-05-03T13:46:05Z”, GoVersion:“go1.18.1”, Compiler:“gc”, Platform:“darwin/arm64”}

Kustomize Version: v4.5.4

Server Version: version.Info{Major:“1”, Minor:“22+”, GitVersion:“v1.22.3-aliyun.1”, GitCommit:“937757761d3c1db7080f179102b437c26c521ecd”, GitTreeState:“clean”, BuildDate:“2022-05-26T12:38:52Z”, GoVersion:“go1.16.9”, Compiler:“gc”, Platform:“linux/amd64”}

WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

容器运行时
Version: 0.1.0

RuntimeName: containerd

RuntimeVersion: 1.5.10

RuntimeApiVersion: v1alpha2

KubeSphere版本信息
v3.3 在已有集群安装 kubesphere

问题是什么
阿里云 ACK 部署 logging 组件无法修改 data-elasticsearch-logging-discovery-0 的 pvc 大小
默认是 :elasticsearchMasterVolumeSize: 4Gi
阿里云 ACK 默认 sc 设置为 alicloud-disk-efficiency (alicloud-disk-efficiency 规定不能小于 20Gi)
修改配置部署组件不能生效

poetL

请问这里应该怎么修改一下呢?
我想其实有两种改法一种是改 installer 容器中的 ansible playbook 变量.

另一种是手动创建 PV ,但这两种方法我昨天都试了一下没有成功求大佬们解答一下

wanjunlei

改完 installer 后要把 status 中的es部分删掉，否则不会更新pvc的

poetL

wanjunlei 大佬我进 installer 改发现容器内没权限额 ,这咋操作

wanjunlei

kubectl edit cc -n kubesphere-system ks-installer

在这改

poetL

wanjunlei
感谢大佬提醒请问 installer 容器内怎么修改呢 ? 没有权限修改 yaml 额
这两处都没法修改, exec 进容器是没有权限的

/kubesphere/installer/roles/common/defaults/main.yaml:14: elasticsearchMasterVolumeSize: 4Gi

/kubesphere/installer/roles/common/templates/custom-values-elasticsearch.yaml.j2:86: size: {% if es_master_pv_size is defined %}{{ es_master_pv_size }}{% else %}{{ common.es.master.volumeSize | default(“4Gi”) }}{% endif %}

wanjunlei

installer里面不允许改的，需要通过cc改

poetL

wanjunlei

cc 更改了但是还是建 4G 的 ,大佬这里有方法建 pv 不, 感觉 ansible 这里走不通额

poetL

我手动创建一个 4Gi 的 pv 也是可以的就是不晓得模版是什么样的, 如果有默认模版我就去试试

阿里云这个 20Gi 的最低云盘限制实在是太坑了 , 只允许通过 yaml 手搓小于 20Gi 的 , 默认 sc 的限制在 20 Gi 以上

wanjunlei

elasticsearchMasterVolumeSize 设置的是 master 节点的 pvc， data 节点要用 elasticsearchDataVolumeSize。

再不行先把创建出来的 pvc 删掉。

记得删除 status 中的 es 部分

poetL

wanjunlei
当前操作卸载 logging 全部组件,cc 删除 status 中的 es 部分, 删除集群中的 pv pvc , 增加elasticsearchDataVolumeSize 和 elasticsearchMasterVolumeSize 均验证为 20 Gi, 还是不能修改 data-elasticsearch-logging-discovery-0 生成 pvc 的大小 .生成出来的依旧是 4Gi

wanjunlei

1.ks 默认的 es pv 大小是20g

2 kubectl get statefulsets.apps -n kubesphere-logging-system elasticsearch-logging-data -o jsonpath=‘{.spec.volumeClaimTemplates[0].spec.resources.requests.storage}’ 确认一下 sts 中设置的 pvc 大小

3.exec 进 installers 的 pod，cat /kubesphere/kubesphere/elasticsearch/custom-values-elasticsearch.yaml 确认下 ElasticSearch helm 中配置的 pvc 大小

poetL

wanjunlei

确认了 pvc 集群内为 data-elasticsearch-logging-discovery-0 4Gi pending
statefulsets elasticsearch-logging-data 大小为 20Gi

`updateStrategy:

type: OnDelete

volumeClaimTemplates:
- apiVersion: v1
  
  kind: PersistentVolumeClaim
  
  metadata:
  
  creationTimestamp: null
  
  name: data
  
  spec:
  
  accessModes:
  - ReadWriteOnce
    
    resources:
    
    requests:
    
    storage: 20Gi
    
    volumeMode: Filesystem
    
    status:
    
    phase: Pending`
ElasticSearch helm 中配置的 pvc 大小看起来依然是 4Gi

bash-5.1$ cat /kubesphere/kubesphere/elasticsearch/custom-values-elasticsearch.yaml 
# Default values for elasticsearch.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
appVersion: "6.8.22"

## Define serviceAccount names for components. Defaults to component's fully qualified name.
##
serviceAccounts:
  client:
    create: false
    name:
  master:
    create: true
    name:
  data:
    create: true
    name:

## Specify if a Pod Security Policy for node-exporter must be created
## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
##
podSecurityPolicy:
  enabled: false
  annotations: {}
    ## Specify pod annotations
    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#apparmor
    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp
    ## Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#sysctl
    ##
    # seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
    # seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
    # apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'

image:
  repository: kubesphere/elasticsearch-oss
  tag: 6.8.22
  pullPolicy: "IfNotPresent"
  # If specified, use these secrets to access the image
  # pullSecrets:
  #   - registry-secret

initImage:
  repository: alpine
  tag: 3.14
  pullPolicy: "IfNotPresent"

cluster:
  name: "elasticsearch"
  # If you want X-Pack installed, switch to an image that includes it, enable this option and toggle the features you want
  # enabled in the environment variables outlined in the README
  xpackEnable: false
  # Some settings must be placed in a keystore, so they need to be mounted in from a secret.
  # Use this setting to specify the name of the secret
  # keystoreSecret: eskeystore
  config: {}
  # Custom parameters, as string, to be added to ES_JAVA_OPTS environment variable
  additionalJavaOpts: ""
  # Command to run at the end of deployment
  bootstrapShellCommand: ""
  env:
    # IMPORTANT: https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#minimum_master_nodes
    # To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible
    # node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.
    # MINIMUM_MASTER_NODES: master.replicas/2 +1 
    # EXPECTED_MASTER_NODES: master.replicas
    # EXPECTED_DATA_NODES: data.replicas
    # RECOVER_AFTER_MASTER_NODES: master.replicas/2 +1
    # RECOVER_AFTER_DATA_NODES: data.replicas/2 +1
  # List of plugins to install via dedicated init container
  # plugins: []

master:
  name: master
  exposeHttp: false
  replicas: 1
  heapSize: "512m"
  # additionalJavaOpts: "-XX:MaxRAM=512m"
  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    name: data
    size: 4Gi
  readinessProbe:
    httpGet:
      path: /_cluster/health?local=true
      port: 9200
    initialDelaySeconds: 5
  antiAffinity: "soft"
  nodeAffinity: {}
  nodeSelector: {}
  tolerations: []
  initResources: {}
    # limits:
    #   cpu: "25m"
    #   # memory: "128Mi"
    # requests:
    #   cpu: "25m"
    #   memory: "128Mi"
  resources:
    limits:
      cpu: 2
      memory: 1024Mi
    requests:
      cpu: 25m
      memory: 512Mi
  priorityClassName: ""
  ## (dict) If specified, apply these annotations to each master Pod
  # podAnnotations:
  #   example: master-foo
  podDisruptionBudget:
    enabled: false
    minAvailable: 1  # Same as `cluster.env.MINIMUM_MASTER_NODES`
    # maxUnavailable: 1
  updateStrategy:
    type: OnDelete

data:
  name: data
  exposeHttp: true
  serviceType: ClusterIP
  loadBalancerIP: {}
  loadBalancerSourceRanges: {}
  replicas: 2
  heapSize: "1536m"
  # additionalJavaOpts: "-XX:MaxRAM=1536m"
  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    name: data
    size: 20Gi
  readinessProbe:
    httpGet:
      path: /_cluster/health?local=true
      port: 9200
    initialDelaySeconds: 5
  terminationGracePeriodSeconds: 120
  antiAffinity: "soft"
  nodeAffinity: {}
  nodeSelector: {}
  tolerations: [{key: "CriticalAddonsOnly", operator: "Exists"}, {key: "dedicated", value: "log", effect: "NoSchedule"}]
  initResources: {}
    # limits:
    #   cpu: "25m"
    #   # memory: "128Mi"
    # requests:
    #   cpu: "25m"
    #   memory: "128Mi"
  resources:
    limits:
      cpu: 4
      memory: 2048Mi
    requests:
      cpu: 25m
      memory: 1536Mi
  priorityClassName: ""
  ## (dict) If specified, apply these annotations to each data Pod
  # podAnnotations:
  #   example: data-foo
  podDisruptionBudget:
    enabled: false
    # minAvailable: 1
    maxUnavailable: 1
  updateStrategy:
    type: OnDelete
  hooks:  # post-start and pre-stop hooks
    drain:  # drain the node before stopping it and re-integrate it into the cluster after start
      enabled: true

## Sysctl init container to setup vm.max_map_count
# see https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
# and https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#mlockall
sysctlInitContainer:
  enabled: true
## Additional init containers
extraInitContainers: |

wanjunlei

卸载 es

helm uninstall elasticsearch-logging -n kubesphere-logging-system

确保 pvc data-elasticsearch-logging-discovery-0 删除了，然后清理 cc status 中的 es 部分，重启 ks-installer

重新安装下试试

poetL

wanjunlei

依然不行, 创建出来的还是 4Gi 的, 大佬这里没办法手工创建 pv 吗? 不行我手工创建也可以

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: diskplugin.csi.alibabacloud.com
  creationTimestamp: "2022-08-09T03:19:48Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: elasticsearch
    component: master
    release: elasticsearch-logging
    role: master
  name: data-elasticsearch-logging-discovery-0
  namespace: kubesphere-logging-system
  resourceVersion: "291967623"
  uid: 09f42aaf-0ffd-486a-9b9a-75442782c754
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  storageClassName: alicloud-disk-efficiency
  volumeMode: Filesystem
status:
  phase: Pending

wanjunlei

把你的 cc 发出来看下

poetL

wanjunlei 辛苦大佬

apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"installer.kubesphere.io/v1alpha1","kind":"ClusterConfiguration","metadata":{"annotations":{},"labels":{"version":"v3.3.0"},"name":"ks-installer","namespace":"kubesphere-system"},"spec":{"alerting":{"enabled":true},"auditing":{"enabled":true},"authentication":{"jwtSecret":""},"common":{"core":{"console":{"enableMultiLogin":true,"port":30880,"type":"NodePort"}},"es":{"basicAuth":{"enabled":false,"password":"","username":""},"elkPrefix":"logstash","externalElasticsearchHost":"","externalElasticsearchPort":"","logMaxAge":7},"gpu":{"kinds":[{"default":true,"resourceName":"nvidia.com/gpu","resourceType":"GPU"}]},"minio":{"volumeSize":"20Gi"},"monitoring":{"GPUMonitoring":{"enabled":false},"endpoint":"http://prometheus-operated.kubesphere-monitoring-system.svc:9090"},"openldap":{"enabled":true,"volumeSize":"20Gi"},"redis":{"enableHA":false,"enabled":false,"volumeSize":"2Gi"}},"devops":{"enabled":true,"jenkinsJavaOpts_MaxRAM":"2g","jenkinsJavaOpts_Xms":"1200m","jenkinsJavaOpts_Xmx":"1600m","jenkinsMemoryLim":"2Gi","jenkinsMemoryReq":"1500Mi","jenkinsVolumeSize":"8Gi"},"edgeruntime":{"enabled":false,"kubeedge":{"cloudCore":{"cloudHub":{"advertiseAddress":[""]},"service":{"cloudhubHttpsNodePort":"30002","cloudhubNodePort":"30000","cloudhubQuicNodePort":"30001","cloudstreamNodePort":"30003","tunnelNodePort":"30004"}},"enabled":false,"iptables-manager":{"enabled":true,"mode":"external"}}},"etcd":{"endpointIps":"localhost","monitoring":false,"port":2379,"tlsEnable":true},"events":{"enabled":true},"gatekeeper":{"enabled":false},"local_registry":"","logging":{"enabled":true,"logsidecar":{"enabled":true,"replicas":2}},"metrics_server":{"enabled":false},"monitoring":{"gpu":{"nvidia_dcgm_exporter":{"enabled":false}},"node_exporter":{"port":9100},"storageClass":""},"multicluster":{"clusterRole":"none"},"network":{"ippool":{"type":"none"},"networkpolicy":{"enabled":true},"topology":{"type":"none"}},"openpitrix":{"store":{"enabled":true}},"persistence":{"storageClass":""},"servicemesh":{"enabled":true,"istio":{"components":{"cni":{"enabled":true},"ingressGateways":[{"enabled":true,"name":"istio-ingressgateway"}]}}},"terminal":{"timeout":600}}}
  creationTimestamp: "2022-07-14T09:37:06Z"
  generation: 283
  labels:
    version: v3.3.0
  name: ks-installer
  namespace: kubesphere-system
  resourceVersion: "291972596"
  uid: edb1f6f7-cca0-41e9-861e-4388c8c48195
spec:
  alerting:
    enabled: true
  auditing:
    enabled: true
  authentication:
    jwtSecret: ""
  common:
    core:
      console:
        enableMultiLogin: true
        port: 30880
        type: NodePort
    es:
      basicAuth:
        enabled: true
        password: #####
        username: ####
      elasticsearchDataVolumeSize: 20Gi
      elasticsearchMasterVolumeSize: 20Gi
      elkPrefix: logstash
      externalElasticsearchHost: ""
      externalElasticsearchPort: ""
      logMaxAge: 7
    gpu:
      kinds:
      - default: true
        resourceName: nvidia.com/gpu
        resourceType: GPU
    minio:
      volumeSize: 20Gi
    monitoring:
      GPUMonitoring:
        enabled: false
      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
    openldap:
      enabled: true
      volumeSize: 20Gi
    redis:
      enableHA: false
      enabled: false
      volumeSize: 20Gi
  devops:
    enabled: true
    jenkinsJavaOpts_MaxRAM: 4g
    jenkinsJavaOpts_Xms: 1200m
    jenkinsJavaOpts_Xmx: 1800m
    jenkinsMemoryLim: 4Gi
    jenkinsMemoryReq: 2Gi
    jenkinsVolumeSize: 20Gi
  edgeruntime:
    enabled: false
    kubeedge:
      cloudCore:
        cloudHub:
          advertiseAddress:
          - ""
        service:
          cloudhubHttpsNodePort: "30002"
          cloudhubNodePort: "30000"
          cloudhubQuicNodePort: "30001"
          cloudstreamNodePort: "30003"
          tunnelNodePort: "30004"
      enabled: false
      iptables-manager:
        enabled: true
        mode: external
  etcd:
    endpointIps: localhost
    monitoring: false
    port: 2379
    tlsEnable: true
  events:
    enabled: true
  gatekeeper:
    enabled: false
  local_registry: ""
  logging:
    enabled: true
    logsidecar:
      enabled: true
      replicas: 2
  metrics_server:
    enabled: false
  monitoring:
    gpu:
      nvidia_dcgm_exporter:
        enabled: false
    node_exporter:
      port: 9100
    storageClass: ""
  multicluster:
    clusterRole: none
  network:
    ippool:
      type: none
    networkpolicy:
      enabled: true
    topology:
      type: none
  openpitrix:
    store:
      enabled: true
  persistence:
    storageClass: ""
  servicemesh:
    enabled: true
    istio:
      components:
        cni:
          enabled: true
        ingressGateways:
        - enabled: true
          name: istio-ingressgateway
  terminal:
    timeout: 600
status:
  alerting:
    enabledTime: 2022-08-09T10:25:04CST
    status: enabled
  auditing:
    enabledTime: 2022-08-09T10:30:48CST
    status: enabled
  clusterId: 9ec78457-6add-4690-8bea-dbac41ce82fa-1660011937
  core:
    enabledTime: 2022-08-09T11:20:52CST
    status: enabled
    version: v3.3.0
  devops:
    enabledTime: 2022-08-09T10:23:00CST
    status: enabled
  es:
    enabledTime: 2022-08-09T11:19:49CST
    status: enabled
  events:
    enabledTime: 2022-08-09T10:31:25CST
    status: enabled
  fluentbit:
    enabledTime: 2022-08-09T10:27:02CST
    status: enabled
  logging:
    enabledTime: 2022-08-09T10:31:18CST
    status: enabled
  minio:
    enabledTime: 2022-08-09T10:26:42CST
    status: enabled
  monitoring:
    enabledTime: 2022-08-09T11:23:51CST
    status: enabled
  openldap:
    enabledTime: 2022-08-09T10:26:32CST
    status: enabled
  servicemesh:
    enabledTime: 2022-08-09T10:22:00CST
    status: enabled

wanjunlei

直接把 data-elasticsearch-logging-discovery 这个 sts 删了，然后按照这个的 sts 的 yaml 重新创建一个，创建的时候修改 pvc 为 20g

poetL

wanjunlei 可以了 , 导出 sts 删除已 pending pvc ,
观察 pv pvc 正常创建, 观察 pod elasticsearch-logging-discovery-0 正常创建 . 谢谢大佬