Elastic Search Troubleshooting

master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster

無法正確的加入 master node

兩個 env 要一起用，不然就會出現這個錯誤。

- name: discovery.seed_hosts
value: "elasticsearch-master"
- name: cluster.initial_master_nodes
value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"

Ref: https://discuss.elastic.co/t/master-not-discovered-yet-this-node-has-not-previously-joined-a-bootstrapped-v7-cluster/176304/2

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid than local cluster uuid rejecting

要先加入新的 master node 才能砍掉舊的 master node，因為 master node 會存 cluster metadata，需要先 replica 到新的 master node。

CrashLoopBackOff 時無法自動 delete pod

寫錯 yaml 時，造成 container crash，進而造成 CrashLoopBackOff 時，就算 apply 新的 yaml，kubernetes 也無法自動移除舊的 Pod。只能 kubectl delete pod podname

Kind: StatefulSet
Replicas: 1
kuberctl:
Client: v1.14.1
Server: v1.12.7-gke.10