2022

What I learned in 2022.

2022-12-27¶

Kafka optimize¶

https://developers.redhat.com/articles/2022/05/03/fine-tune-kafka-performance-kafka-optimization-theorem#kafka_priorities_and_the_cap_theorem

2022-11-11¶

Github Action - Publish markmap to Github Page¶

need markmap/ folders

jobs:
  update-markmap:
    name: Update Markmap
    runs-on: [self-hosted]
    steps:
      - uses: actions/checkout@v3
      - name: Install markmap-cli
        run: |
          yarn global add markmap-cli markmap-common
          echo "`yarn global bin`" >> $GITHUB_PATH
      - name: Git init
        run: |
          git config --global user.name "${GITHUB_ACTOR}"
          git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com"
      - name: Build html
        run: |
          touch .nojekyll
          ls markmap | cut -d '.' -f1 | xargs -I@ markmap markmap/@.md --output @.html --no-open
      - name: Deploy
        run: |
          git checkout --orphan gh-pages
          git reset
          git add *.html .nojekyll
          git commit -m "Generate markmap by Github Action"
          git push -u origin HEAD:gh-pages -f

2022-11-07¶

Mermaid example with better theme color¶

flowchart TD
    A[Start] --> B{Is it?}
    B -->|Yes| C[OK]
    C --> D[Rethink]
    D --> B
    B ---->|No| E[End]

2022-10-04¶

Kibana - Generate csv Max size reached¶

Max size reached

Increase maxSizeBytes, it defaults to 10MB.

[root@ip-10-16-41-158 kibana]# vi kibana.yml

# xpack.reporting.csv.maxSizeBytes: 104857600

[root@ip-10-16-41-158 kibana]# systemctl restart kibana

2022-07-14¶

ElasticSeach - Snapshot to S3 throughtput¶

770MB/s

env:

AWS: c5.9xlarge
50 GB/shard, 20 shard, 1TB data
Total run time: 1300s
thread pool: snapshot:5

2022-06-02¶

Elasticsearch - Searchable snapshot won't delete index from hot nodes¶

Use explain can see

GET _cluster/allocation/explain

"node_decision" : "no",
"deciders" : [
  {
    "decider" : "has_frozen_cache",
    "decision" : "NO",
    "explanation" : "node setting [xpack.searchable.snapshot.shared_cache.size] is set to zero, so frozen searchable snapshot shards cannot be allocated to this node"
  }
]

So, just add new node with

node.roles: ["data_frozen"]

2022-05-27¶

Elasticsearch - Failed step attempt-rollover¶

Index lifecycle error
exception: Concurrent modification of alias [rammus-log] during rollover

GET .ds-rammus-log-002774/_ilm/explain
POST .ds-rammus-log-002773/_ilm/retry

2022-05-03¶

openssl - p12 expired date¶

openssl pkcs12 -in elastic-certificates.p12 -nodes | openssl x509 -noout -enddate

Demo

❯ openssl pkcs12 -in elastic-certificates.p12 -clcerts -nodes | openssl x509 -noout -enddate

Enter Import Password:
MAC verified OK
notAfter=Jan 12 08:40:45 2021 GMT

2022-05-02¶

Free image 免費高清圖庫¶

https://unsplash.com/

2022-04-28¶

EKS - Can't create NLB¶

{"level":"error","ts":1651135177.483026,"logger":"controller-runtime.manager.controller.targetGroupBinding","msg":"Reconciler error","reconciler group":"elbv2.k8s.aws","reconciler kind":"TargetGroupBinding","name":"k8s-eckpoc-quicksta-6b38f0416c","namespace":"eckpoc","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/poc-cluster for eni eni-0e226022e7dd6aa80, got: [sg-011da083ca19a29ef sg-0446f7a0f894dd9a3]"}
eni eni-0e226022e7dd6aa80, got: [sg-011da083ca19a29ef sg-0446f7a0f894dd9a3]"}

Solution

Remove the sg with description: EKS created security group applied to ENI that is attached to EKS Control Plane master nodes, as well as any managed workloads.

I guess it doesn't matters to remove which sg. Both can work.

Ref:

2022-04-26¶

Google Sheet - Insert image from Google Drive¶

share image to Everyone
Get link: https://drive.google.com/file/d/1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x/view?usp=sharing
Use url in these ways: a. https://lh6.googleusercontent.com/d/1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x b. https://drive.google.com/thumbnail?id=1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x

2022-03-29¶

ElasticSearch - Kibana can't compatible with downstair versions.¶

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.16.3-x86_64.rpm
yum localinstall -y kibana-7.16.3-x86_64.rpm

{"type":"log","@timestamp":"2022-03-29T07:53:52+00:00","tags":["error","elasticsearch-service"],"pid":10677,"message":"This version of Kibana (v7.16.3) is incompatible with the following Elasticsearch nodes inyour cluster:v7.9.3 @ 10.16.41.65:9200 (10.16.41.65), v7.9.3 @ 10.16.41.126:9200 (10.16.41.126) ...}

Kibana - Install in yum¶

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.16.3-x86_64.rpm
yum localinstall -y kibana-7.16.3-x86_64.rpm

2022-03-15¶

ElasticSearch - cluster full restart needs .security-7 index shard¶

master might store auth data in memory
when masters shutdown, they will need data node contains .security-7 index shard

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

2022-03-11¶

ElasticSearch - Change mapping fail, cannot be changed from type [text] to [keyword]¶

"mapper [message] cannot be changed from type [text] to [keyword]"

solution:

Create new index with new mapping

POST _reindex
{
  "source": {
    "index": "rammus-poc"
  },
  "dest": {
    "index": "rammus-poc-20220311"
  }
}

Use _reindex to migrate data

PUT rammus-poc-20220311
{
    "settings": {
        "index": {
            "routing": {
                "allocation": {
                    "total_shards_per_node": "1"
                }
            },
            "number_of_shards": "1",
            "priority": "500",
            "number_of_replicas": "1"
        }
    },
    "mappings": {
        "properties": {
          ...
        }
    }
}

2022-03-09¶

awscli - disable ec2 termination protection¶

aws ec2 modify-instance-attribute --no-disable-api-termination --instance-id i-xxxxx --profile rammus-dev

2022-03-07¶

ElasticSearch - update keystore password¶

echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.transport.ssl.keystore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.transport.ssl.truststore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.http.ssl.keystore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.http.ssl.truststore.secure_password"

Teleport - for loop run command in multiple machines¶

for ip in 172.21.1.1 172.21.1.2 172.21.1.3
do
  tsh ssh ec2-user@${ip} sudo systemctl restart elasticsearch
done

amazon linux - install and start docker¶

yum install docker

## Run docker daemon
systemctl start docker

## Add permission
newgrp docker
sudo usermod -aG docker ec2-user