多个具有CrashLoopBackOff状态的MySQL Kubernetes Pod

时间:2018-09-12 08:29:10

标签: mysql sql kubernetes innodb pod

我在Kubernetes集群上部署WordPress + MySQL应用程序时遇到问题。

在使用$('#checkbox_ID').change(function() { if (this.checked) { $('#input_ID').val("Some value") } else { $('#input_ID').val("") } }); 自动缩放我的HorizontalPodAutoscalerwordpress部署时,它对于wordpress-mysql部署而不是wordpress部署很好。
确实,当创建多个MySQL Pod时,某些Pod处于wordpress-mysql状态:

CrashLoopBackOff

当我查看他们的日志时,我得到了:

$ kubectl get all -n wordpress
NAME                                 READY     STATUS             RESTARTS   AGE
po/wordpress-3874566264-7031k        1/1       Running            0          16h
po/wordpress-mysql-898811424-2bdnn   0/1       CrashLoopBackOff   6          16h
po/wordpress-mysql-898811424-dxj92   1/1       Running            146        16h
po/wordpress-mysql-898811424-vs29j   0/1       CrashLoopBackOff   148        16h

NAME                  CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
svc/wordpress         10.254.121.190   10.0.0.13     80:30003/TCP   16h
svc/wordpress-mysql   None             <none>        3306/TCP       16h

NAME                  REFERENCE                    TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
hpa/wordpress         Deployment/wordpress         28% / 80%, 0% / 80%   1         10        1          16h
hpa/wordpress-mysql   Deployment/wordpress-mysql   90% / 80%, 0% / 80%   1         10        3          16h

NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/wordpress         1         1         1            1           16h
deploy/wordpress-mysql   3         3         3            1           16h

NAME                           DESIRED   CURRENT   READY     AGE
rs/wordpress-3874566264        1         1         1         16h
rs/wordpress-mysql-898811424   3         3         1         16h

所以这可能很正常,因为每个MySQL Pod都试图同时访问$ kubectl logs -p wordpress-mysql-898811424-2bdnn -n wordpress 2018-09-12 08:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2018-09-12 08:04:12 0 [Note] mysqld (mysqld 5.6.41) starting as process 436 ... 2018-09-12 08:04:12 436 [Note] Plugin 'FEDERATED' is disabled. 2018-09-12 08:04:12 436 [Note] InnoDB: Using atomics to ref count buffer pool pages 2018-09-12 08:04:12 436 [Note] InnoDB: The InnoDB memory heap is disabled 2018-09-12 08:04:12 436 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2018-09-12 08:04:12 436 [Note] InnoDB: Memory barrier is not used 2018-09-12 08:04:12 436 [Note] InnoDB: Compressed tables use zlib 1.2.3 2018-09-12 08:04:12 436 [Note] InnoDB: Using Linux native AIO 2018-09-12 08:04:12 436 [Note] InnoDB: Using CPU crc32 instructions 2018-09-12 08:04:12 436 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2018-09-12 08:04:12 436 [Note] InnoDB: Completed initialization of buffer pool 2018-09-12 08:04:12 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2018-09-12 08:04:12 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2018-09-12 08:04:12 436 [Note] InnoDB: Retrying to lock the first data file 2018-09-12 08:04:13 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2018-09-12 08:04:13 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2018-09-12 08:04:14 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2018-09-12 08:04:14 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2018-09-12 08:04:15 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 [...] 2018-09-12 08:05:51 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2018-09-12 08:05:52 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11 2018-09-12 08:05:52 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2018-09-12 08:05:52 436 [Note] InnoDB: Unable to open the first data file 2018-09-12 08:05:52 7f57a329f5c0 InnoDB: Operating system error number 11 in a file operation. InnoDB: Error number 11 means 'Resource temporarily unavailable'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2018-09-12 08:05:52 436 [ERROR] InnoDB: Can't open './ibdata1' 2018-09-12 08:05:52 436 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' init function returned error. 2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2018-09-12 08:05:52 436 [ERROR] Unknown/unsupported storage engine: InnoDB 2018-09-12 08:05:52 436 [ERROR] Aborting 2018-09-12 08:05:52 436 [Note] Binlog end 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'partition' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_METRICS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCKS' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_TRX' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'BLACKHOLE' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'ARCHIVE' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MRG_MYISAM' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MyISAM' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MEMORY' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'CSV' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'sha256_password' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_old_password' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_native_password' 2018-09-12 08:05:52 436 [Note] Shutting down plugin 'binlog' 2018-09-12 08:05:52 436 [Note] mysqld: Shutdown complete ,但这是我的问题:是否真的有可能使多个MySQL Pod共享同一数据? ?如果答案是肯定的,那么我应该如何避免这些烦人的错误?

如果您需要其他信息,请告诉我,我将编辑我的帖子。

提前感谢您的帮助!

2 个答案:

答案 0 :(得分:2)

  

所以这可能很正常,因为每个MySQL pod都试图同时访问./ibdata1

是的,如果您尝试这样做(不提供清单),那么这就是您具有CrashLoopBackOff状态的原因。首先启动的instace将锁定它,随后的所有操作都会失败。

  

...尝试同时访问./ibdata1 ...真的有可能使多个MySQL容器共享同一数据吗?

如果我们在两个独立的mysql实例上讨论相同的数据文件夹(具有相同的永久卷,主机路径或nfs共享...),则由于很多原因,不可以,不是真的,也不建议这样做。

如果您需要多个mysql实例(进程,容器或容器)共享同一数据(而不是数据文件夹!),则需要使用复制(带有只读副本或其他...),其中每个实例都有自己的数据文件夹结构,但它们以某种方式在它们之间同步数据。这是kubernetes官方文档上a MySQL single-master topology with multiple slaves running asynchronous replication的一个示例。请注意,这不是生产设置,不是HA设置,而只是一个简单复制方案的说明,可以使您有所了解。

现在,有一些简单的问题:您确定无法使用服务于多个wordpress实例的单个mysql实例来处理负载吗?您是否要进行HA设置?因为对每个问题的答案都需要与“从1开始增加豆荚的数量。”的方法和体系结构决策稍有不同。

答案 1 :(得分:0)

The port name is the same as any pod within your cluster. 
ex:

containers:
  - image: mysql:5.6
    name: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-pass
          key: password
    ports:
    - containerPort: 3306
      name: mysql-wiki
    volumeMounts:
    - name: mysql-persistent-storage
      mountPath: /var/lib/mysql
  volumes:
  - name: mysql-persistent-storage
    persistentVolumeClaim:
      claimName: mysql-pv-claim