Skip to content

2021

What I learned in 2021.

2021-12-24

Helm - escape {{ }}

ref: https://github.com/helm/helm/issues/2798#issuecomment-890478869

      labels:
        sink_name: loki_syslog
        #Below key useed printf() to escape Helm interprets {{ }}
        message: >-
          {{ printf "{{ message }}" }}

2021-12-15

Git - Clean merged local branches

git branch --merged | egrep -v "(^\*|master)" | xargs git branch -d

2021-08-19

Redis - benchmark

< 1 KB , 200k rps, 2ms
> 65 KB, < 100k rps, 4ms
reclaim: Is TTL setted.
aws redis node: 65k connections limited

Redis - GUI tools with analysis

https://redis.com/redis-enterprise/redis-insight/

2021-06-15

Git - don't show more/continue

git config --global pager.log false

2021-05-06

Kubernetes - Upgrade cert-manager to v1.3.1

Check exited CRD resources.

kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

Backup resources.

kubectl get -o yaml \
     --all-namespaces \
     issuer,clusterissuer,certificates > cert-manager-backup.yaml

Use Helm to deploy cert-manager:v1.3.1

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.3.1 \
  --set installCRDs=true

2021-04-29

Redis - Cluster failover

3 master down -> cluster shutdown, can't failover to replicas -> when 2 master back, still not back -> need all 3 master back and re-sync

1 master + 1 replica down -> cluster shutdown -> 1 master back, still not back

2 master down -> cluster shutdown -> 1 master back, still not back

conclusion:

  • If cluster turns down, need all nodes re-sync to get it back.
  • over 1/2 master down at same time will cause cluster down
  • 1 shard down will cause cluster down

2021-04-15

Mongo - sharding not enabled for db

sh.enableSharding("mydatabase")

Mongo - update on a sharded collection must either contain an exact match on _id or must target a single shard

A {multi:false} update on a sharded collection must either contain an exact match on _id or must target a single shard but this update targeted _id (and have the collection default collation) or must target a single shard (and have the simple collation), but this update targeted 2 shards. Update request: { q: { nb: 0 }, u: { $set: { date: new Date(1618511355253), c32: 13, c64: 10818 } }, multi: false, upsert: false }, shard key pattern: { c32: 1.0, c64: 1.0 }, full error: {'index': 0, 'code': 72, 'codeName': 'InvalidOptions', 'errmsg': 'A {multi:false} update on a sharded collection must either contain an exact match on _id or must target a single shard but this update targeted _id (and have the collection default collation) or must target a single shard (and have the simple collation), but this update targeted 2 shards. Update request: { q: { nb: 0 }, u: { $set: { date: new Date(1618511355253), c32: 13, c64: 10818 } }, multi: false, upsert: false }, shard key pattern: { c32: 1.0, c64: 1.0 }'}

New document must contains shard key fields. In this case is c32, c64.

2021-04-14

Mongo - transaction

GKE - NEG get 502 when rollingUpate

credit:

        lifecycle:
          preStop:
            exec:
              # SIGTERM triggers a quick exit; gracefully terminate instead
              command: ["nginx", "-s", "quit"]

2021-04-13

gsutil - Download large file with crcmod on macOS

pip3 install -U crcmod

Before:

# gsutil version -l
gsutil version: 4.55
checksum: adebf7d276641651e3345d12aca978c0 (OK)
boto version: 2.49.0
python version: 3.9.4 (default, Apr  5 2021, 01:47:16) [Clang 11.0.0 (clang-1100.0.33.17)]
OS: Darwin 18.7.0
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /Users/rammus/.boto, /Users/rammus/.config/gcloud/legacy_credentials/rammus.xu@swag.live/.boto
gsutil path: /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False

After:

# gsutil version -l
gsutil version: 4.55
checksum: adebf7d276641651e3345d12aca978c0 (OK)
boto version: 2.49.0
python version: 3.9.4 (default, Apr  5 2021, 01:47:16) [Clang 11.0.0 (clang-1100.0.33.17)]
OS: Darwin 18.7.0
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /Users/rammus/.boto, /Users/rammus/.config/gcloud/legacy_credentials/rammus.xu@swag.live/.boto
gsutil path: /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

2021-03-29

Kubernetes - Install cert-manager v1.2.0

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.2.0 \
  --create-namespace \
  --set installCRDs=true

2021-03-25

docker - Can't resolve host on build time

Resolving github.com (github.com)... failed: Name does not resolve.
wget: unable to resolve host address 'github.com'

Solution

Upgrade docker desktop to > 3.1.0

This happens on my Macbook Pro (docker desktop: 2.5.0.1).

It's fixed after I upgrade to 3.2.2.

2021-03-24

Linkerd vs Istio on 2019

https://medium.com/@michael_87395/benchmarking-istio-linkerd-cpu-c36287e32781

Istio’s Envoy proxy uses more than 50% more CPU than Linkerd’s

  • k8s: 1.12.7-gke.7
  • node: n1-standard-4
  • 4-16 nodes
  • linkerd: 2.3.0 (now: 2.10)
  • istio: 1.0.6 (now: 1.9.0)

2021-03-18

為什麼 TCP 要 3-way handshake

https://draveness.me/whys-the-design-tcp-three-way-handshake/

  • TCP 是雙向對話 (bi-directional communication protocol)
  • 雙方都需要 SYN, ACK
Client ------SYN-----> Server
Client <---ACK/SYN---- Server
Client ------ACK-----> Server

2021-03-15

Docker - Run different platform architecture images on Apple M1 mini

rammus.xu@mac-mini ~ % docker run -it --rm alpine uname -a
Linux ffb2f47751c8 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 aarch64 Linux
rammus.xu@mac-mini ~ % docker run -it --rm --platform amd64 alpine uname -a
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ba3557a56b15: Pull complete
Digest: sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
Status: Downloaded newer image for alpine:latest
docker: Error response from daemon: image with reference alpine was found but does not match the specified platform: wanted darwin/amd64, actual: linux/amd64.
See 'docker run --help'.
rammus.xu@mac-mini ~ % docker run -it --rm --platform linux/amd64 alpine uname -a
Linux f313db1d82a1 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 x86_64 Linux
rammus.xu@mac-mini ~ % docker run -it --rm --platform linux/i386 alpine uname -a
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
86205afa28f6: Pull complete
Digest: sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
Status: Downloaded newer image for alpine:latest
Linux 4aff8fb39ae9 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 i686 Linux

2021-03-09

Kubernetes - Increase metrics-server resources (cpu/memory)

Metrics not available for pod

Ref: https://github.com/kubernetes/autoscaler/tree/master/addon-resizer

Environment:

  • Kubernetes: v1.17.15-gke.800

It's 3-5 minutes downtime.

> kubectl apply -f metrics-server-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: metrics-server-config
  namespace: kube-system
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration
    baseCPU: 200m
    cpuPerNode: 2m
    baseMemory: 150Mi
    memoryPerNode: 4Mi
> kubectl delete deployment -n kube-system metrics-server-v0.3.6
deployment.apps "metrics-server-v0.3.6" deleted

2021-02-23

2021 Client Side Real User Monitoring (RUM)

Cloud Hosted:

Self Hosted:

2021-02-05

Kubernetes - Use annotation and initContainer to donwload GCS files

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: demo
  replicas: 1
  template:
    metadata:
      labels:
        app: demo
      annotations:
        update: "true"
    spec:
      volumes:
      - name: assets
        emptyDir: {}
      initContainers:
      - name: download-assets
        image: gcr.io/google.com/cloudsdktool/cloud-sdk:315.0.0-alpine
        command: ["/bin/bash","-c"]
        env:
        - name: IF_UPDATE
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['update']
        args:
        - |
          [ "$IF_UPDATE" = "false" ] && exit 0
          gsutil -m cp -r gs://rammus.tw/assets /assets

        volumeMounts:
        - name: assets
          mountPath: /assets
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: assets
          mountPath: /assets

2021-01-26

Docker - Build a userspace Wireguard image

ghcr.io/linuxserver/wireguard default to use Linux kernal library. If you want to use userspace library or your machine doesn't support Linux kernal library.

FROM rust:1.40-slim-buster AS builder

ARG BORINGTUN_VERSION=0.3.0

RUN cargo install boringtun --version ${BORINGTUN_VERSION}

###

FROM ghcr.io/linuxserver/wireguard:version-v1.0.20200827

COPY --from=builder /usr/local/cargo/bin/boringtun /usr/local/bin

ENV WG_QUICK_USERSPACE_IMPLEMENTATION=boringtun \
    WG_SUDO=1

2021-01-25

MongoDB - Get wiredTiger stats

ref: https://docs.mongodb.com/manual/reference/command/serverStatus/#wiredtiger

db.runCommand( { serverStatus: 1 } ).wiredTiger

2021-01-21

mkdocs-material lexers - code block supported programming languages

mkdocs-material 使用 pygments 當作語法分析器,以下是能支援的程式語言

as3, actionscript3
as, actionscript
mxml
bc
gap
mathematica, mma, nb
mupad
at, ambienttalk, ambienttalk/2
ampl
apl
adl
cadl
odin
arrow
c-objdump
ca65
cpp-objdump, c++-objdumb, cxx-objdump
d-objdump
dasm16
gas, asm
hsail, hsa
llvm
llvm-mir-body
llvm-mir
nasm
objdump-nasm
objdump
tasm
autoit
ahk, autohotkey
bare
bbcbasic
blitzbasic, b3d, bplus
blitzmax, bmax
cbmbas
monkey
qbasic, basic
vbscript
bst, bst-pybtex
bib, bibtex
boa
abap
cobolfree
cobol
gooddata-cl
maql
openedge, abl, progress
c
cpp, c++
arduino
charmci
clay
cuda, cu
ec
mql, mq4, mq5, mql4, mql5
nesc
pike
swig
vala, vapi
capnp
chapel, chpl
clean
apacheconf, aconf, apache
augeas
cfengine3, cf3
docker, dockerfile
ini, cfg, dosini
kconfig, menuconfig, linux-config, kernel-config
lighty, lighttpd
nginx
pacmanconf
pkgconfig
properties, jproperties
registry
singularity
squidconf, squid.conf, squid
toml
termcap
terminfo
terraform, tf
pypylog, pypy
vctreestatus
cr, crystal
csound-document, csound-csd
csound, csound-orc
csound-score, csound-sco
css
less
sass
scss
croc
d
minid
smali
None
jsonld, json-ld
json, json-object
yaml
devicetree, dts
dpatch
diff, udiff
wdiff
boo
aspx-cs
csharp, c#
fsharp, f#
nemerle
aspx-vb
vb.net, vbnet
alloy
crmsh, pcmk
flatline
mscgen, msc
pan
protobuf, proto
puppet
rsl
snowball
thrift
vgl
zeek, bro
dylan-console, dylan-repl
dylan
dylan-lid, lid
ecl
eiffel
elm
email, eml
iex
elixir, ex, exs
erlang
erl
aheui
befunge
brainfuck, bf
camkes, idl4
capdl
redcode
ezhil
factor
fan
felix, flx
floscript, flo
forth
fortranfixed
fortran
foxpro, vfp, clipper, xbase
freefem
gdscript, gd
go
abnf
bnf
jsgf
peg
cypher
asy, asymptote
glsl
gnuplot
hlsl
postscript, postscr
pov
agda
cryptol, cry
haskell, hs
hspec
idris, idr
koka
lagda, literate-agda
lcry, literate-cryptol, lcryptol
lhs, literate-haskell, lhaskell
lidr, literate-idris, lidris
hx, haxe, hxsl
haxeml, hxml
systemverilog, sv
verilog, v
vhdl
hexdump
dtd
haml
html
pug, jade
scaml
xml
xslt
idl
igor, igorpro
limbo
control, debcontrol
nsis, nsi, nsh
spec
sourceslist, sources.list, debsources
inform6, i6
i6t
inform7, i7
tads3
io
j
coffee-script, coffeescript, coffee
dart
earl-grey, earlgrey, eg
js, javascript
juttle
kal
lasso, lassoscript
live-script, livescript
mask
objective-j, objectivej, obj-j, objj
ts, typescript
jlcon
julia, jl
aspectj
ceylon
clojure, clj
clojurescript, cljs
golo
gosu
gst
groovy
ioke, ik
jasmin, jasminxt
java
kotlin
pig
sarl
scala
xtend
cpsa
common-lisp, cl, lisp
emacs, elisp, emacs-lisp
fennel, fnl
hylang
newlisp
racket, rkt
scheme, scm
shen
extempore
basemake
cmake
make, makefile, mf, bsdmake
bbcode
groff, nroff, man
md, markdown
trac-wiki, moin
css+mozpreproc
mozhashpreproc
javascript+mozpreproc
mozpercentpreproc
xul+mozpreproc
rst, rest, restructuredtext
tex, latex
tid
matlab
matlabsession
octave
scilab
mime
fstar
ocaml
opa
reason, reasonml
sml
bugs, winbugs, openbugs
jags
modelica
stan
modula2, m2
monte
mosel
ncl
nim, nimrod
nit
nixos, nix
componentpascal, cp
logos
objective-c, objectivec, obj-c, objc
objective-c++, objectivec++, obj-c++, objc++
swift
ooc
parasail
antlr-as, antlr-actionscript
antlr-csharp, antlr-c#
antlr-cpp
antlr-java
antlr
antlr-objc
antlr-perl
antlr-python
antlr-ruby, antlr-rb
ebnf
ragel-c
ragel-cpp
ragel-d
ragel-em
ragel-java
ragel
ragel-objc
ragel-ruby, ragel-rb
treetop
ada, ada95, ada2005
delphi, pas, pascal, objectpascal
pawn
sp
perl6, pl6, raku
perl, pl
php, php3, php4, php5
psysh
zephir
pointless
pony
praat
logtalk
prolog
promql
cython, pyx, pyrex
dg
numpy
python2, py2
py2tb
pycon
python, py, sage, python3, py3
pytb, py3tb
qvto, qvt
rconsole, rout
rd
splus, s, r
shexc, shex
sparql
turtle
rebol
red, red/system
resource, resourcebundle
ride
rnc, rng-compact
roboconf-graph
roboconf-instances
robotframework
fancy, fy
rbcon, irb
rb, ruby, duby
rust, rs
sas
scdoc, scd
applescript
chai, chaiscript
easytrieve
hybris, hy
jcl
lsl
lua
moocode, moo
ms, miniscript
moon, moonscript
rexx, arexx
sgf
bash, sh, ksh, zsh, shell
console, shell-session
bat, batch, dosbatch, winbatch
execline
fish, fishshell
doscon
powershell, posh, ps1, psm1
ps1con
slurm, sbatch
tcsh, csh
tcshcon
sieve
slash
newspeak
smalltalk, squeak, st
nusmv
snobol
solidity
raw
text
mysql
plpgsql
psql, postgresql-console, postgres-console
postgresql, postgres
rql
sql
sqlite3
tsql, t-sql
stata, do
sc, supercollider
tcl
html+ng2
ng2
html+cheetah, html+spitfire, htmlcheetah
js+cheetah, javascript+cheetah, js+spitfire, javascript+spitfire
cheetah, spitfire
xml+cheetah, xml+spitfire
cfc
cfm
cfs
css+django, css+jinja
css+erb, css+ruby
css+genshitext, css+genshi
css+php
css+smarty
django, jinja
erb
html+evoque
evoque
xml+evoque
genshi, kid, xml+genshi, xml+kid
genshitext
html+handlebars
handlebars
html+django, html+jinja, htmldjango
html+genshi, html+kid
html+php
html+smarty
js+django, javascript+django, js+jinja, javascript+jinja
js+erb, javascript+erb, js+ruby, javascript+ruby
js+genshitext, js+genshi, javascript+genshitext, javascript+genshi
js+php, javascript+php
js+smarty, javascript+smarty
jsp
css+lasso
html+lasso
js+lasso, javascript+lasso
xml+lasso
liquid
css+mako
html+mako
js+mako, javascript+mako
mako
xml+mako
mason
css+myghty
html+myghty
js+myghty, javascript+myghty
myghty
xml+myghty
rhtml, html+erb, html+ruby
smarty
ssp
tea
html+twig
twig
html+velocity
velocity
xml+velocity
xml+django, xml+jinja
xml+erb, xml+ruby
xml+php
xml+smarty
yaml+jinja, salt, sls
ttl, teraterm, teratermmacro
cucumber, gherkin
tap
awk, gawk, mawk, nawk
vim
pot, po
http
irc
kmsg, dmesg
notmuch
todotxt
coq
isabelle
lean
tnt
rts, trafficscript
typoscriptcssdata
typoscripthtmldata
typoscript
icon
ucode
unicon
urbiscript
usd, usda
vcl
vclsnippets, vclsnippet
boogie
silver
webidl
cirru
duel, jbst, jsonml+bst
qml, qbs
slim
xquery, xqy, xq, xql, xqm
whiley
x10, xten
xorg.conf
yang
zig

2021-01-19

ACK - failed to provision volume with StorageClass "alicloud-disk-ssd": rpc error: code = Internal desc = SDK.ServerError

ErrorCode: InvalidDiskSize.NotSupported

因為申請太小的容量,所以出現這個錯誤。

ref: https://help.aliyun.com/document_detail/127601.html

  • 高效云盘:最小20 GiB。
  • SSD云盘:最小20 GiB。
  • ESSD云盘:最小20 GiB。

Nginx - Use NFS to share proxy_cache

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=storage:128m inactive=1y max_size=64G use_temp_path=off;

Must to add use_temp_path=off which default to on.

2021-01-08

gsutil - Upload file to AWS S3

Least AWS user permission.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::backup.rammus.tw/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::backup.rammus.tw"
        }
    ]
}

Generate configuration

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

cat << EOF > ~/.boto
[Credentials]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY

[s3]
calling_format = boto.s3.connection.OrdinaryCallingFormat
use-sigv4=True
host=s3.ap-northeast-1.amazonaws.com
EOF

Start to uplaod

gsutil cp a s3://backup.rammus.tw