How I debug a certificate didn't renew

I found out my certificate is expired this morning and it’s not renewed automatically. Here’s how I debug it step by step.

Get certificate status

$ kubectl describe cert -n slack slack-tls
Status:
  Conditions:
    Last Transition Time:  2020-01-21T04:15:16Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-08-18T01:20:11Z

Try to force certificate renewal

By adding spec.renewBefore to certificate.

kubectl -n <namespace> patch certificate example-certificate --type=merge -p '{"spec":{"renewBefore":"2159h00m00s"}}'

And the order is still invalid.

$ kubectl -n slack get order
NAME                  STATE   AGE
slack-tls-488818493   invalid   11m

So, I try to see if any event happened.

$ kubectl get event -n slack
LAST SEEN   TYPE      REASON         OBJECT                            MESSAGE
38m         Warning   PresentError   challenge/slack-tls-488818493-0   Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission
12m         Warning   CleanUpError   challenge/slack-tls-488818493-0   Error cleaning up challenge: GoogleCloud API call failed: googleapi: Error 403: Request had insufficient authentication scopes.
More details:

Read More

AWS Route 53 - 實測 Geolocation Routing

AWS Route 53 Geolocation Routing 可以根據不同地理位置，回傳不同的 IP，可以用這個功能讓使用者選擇最近的 Data center，或是回傳不同的網站內容。

我們先加入幾組 Record

使用 https://dnschecker.org/ 檢查，可以看到 US, CA 的部分都是顯示 35.186.255.2(NA)，其餘的都是 35.186.255.1(Default)

這邊特別設定一個新加坡的 35.186.255.４，也能正確顯示。
其他也都是 35.186.255.1(Default)，不過有一個 IONICA LLC 是連到 35.186.255.2(NA)的特例。

Linkerd 2.8 - 實現超級簡單又安全的多叢集(multicluster) Kubernetes 架構

研究動機：如何在 Pod 中呼叫別的 kubernetes cluster service

嘗試過目前市面上現有的三套開源軟體(Linkerd, Consul, Istio)，決定推坑大家 Linkerd 在今年六月釋出的 Linkerd 2.8，實現了 multi-cluster Kubernetes 架構，我們來看看主打超輕量化 service mesh 的 Linkerd 會怎麼表現在 multi-cluster。

目前筆者在同一個 GCP 呼叫不同 cluster 的 service 都需要使用 internal load balancer，除了會增加許多額外的花費，internal load balancer 預設是能夠讓同一個 VPC 存取，代表任何 VM 都能使用這個 service，這當然不是我們所預期的行為。最重要的是很難維護，所以開始尋找有沒有 solution 可以解決這個問題。

  • 可以從 Cluster-A 存取 Cluster-B 的 service
  • Cluster-B 可以決定要 expose 哪些 service 給特定的 cluster
  • connections 低延遲、穩定
  • configuration 越少越好
  • discover service fast
  • private connection with security

Read More

Kubernetes - Access another cluster services with service account token

This post will show you how to access other cluster service with an authorization token

It would like:

curl $APISERVER/api/v1/namespaces/default/services/httpbin:80/proxy/get --header "Authorization: Bearer $TOKEN" --insecure

Deploy resource

Please know what you are going to deploy.

kustomize build github.com/RammusXu/toolkit/k8s/access-service-with-service-account-token

Apply

kustomize build github.com/RammusXu/toolkit/k8s/access-service-with-service-account-token | kubectl apply -f -

Get service account token

NAMESPACE=default
SERVICE_ACCOUNT_NAME=service-proxy
APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
SECRET_NAME=$(kubectl get secrets -n $NAMESPACE | grep ^$SERVICE_ACCOUNT_NAME | cut -f1 -d ' ')
TOKEN=$(kubectl describe secret $SECRET_NAME -n $NAMESPACE | grep -E '^token' | cut -f2 -d':' | tr -d " ")
curl $APISERVER/api/v1/namespaces/default/services/httpbin:http/proxy/get --header "Authorization: Bearer $TOKEN" --insecure

## result
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Host": "35.10.10.10",
    "User-Agent": "curl/7.54.0",
    "X-Forwarded-Uri": "/api/v1/namespaces/default/services/httpbin:http/proxy/get"
  },
  "origin": "59.10.10.10",
  "url": "http://35.10.10.10/get"
}

Clean

kustomize build github.com/RammusXu/toolkit/k8s/access-service-with-service-account-token | kubectl delete -f -

Troubleshooting

“services "httpbin" is forbidden: User "system:serviceaccount:default:default" cannot get resource "services/proxy" in API group "" in the namespace "default"“

It needs RBAC: services/proxy

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rammus:service-proxy
rules:
- apiGroups: [""]
  resources: ["services/proxy"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

no endpoints available for service "httpbin"

"status": "Failure",
"message": "no endpoints available for service \"httpbin\"",
"reason": "ServiceUnavailable",
"code": 503

The correct way to access are:

✅
curl $APISERVER/api/v1/namespaces/default/services/httpbin:80/proxy/get --header "Authorization: Bearer $TOKEN" --insecure
✅
curl $APISERVER/api/v1/namespaces/default/services/httpbin:http/proxy/get --header "Authorization: Bearer $TOKEN" --insecure
❌
curl $APISERVER/api/v1/namespaces/default/services/httpbin/proxy/get --header "Authorization: Bearer $TOKEN" --insecure
❌
curl $APISERVER/api/v1/namespaces/default/services/httpbin:http/proxy --header "Authorization: Bearer $TOKEN" --insecure

Reference