Skip to content

2022

What I learned in 2022.

2022-11-11

Github Action - Publish markmap to Github Page

  1. need markmap/ folders
jobs:
  update-markmap:
    name: Update Markmap
    runs-on: [self-hosted]
    steps:
      - uses: actions/checkout@v3
      - name: Install markmap-cli
        run: |
          yarn global add markmap-cli markmap-common
          echo "`yarn global bin`" >> $GITHUB_PATH
      - name: Git init
        run: |
          git config --global user.name "${GITHUB_ACTOR}"
          git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com"
      - name: Build html
        run: |
          touch .nojekyll
          ls markmap | cut -d '.' -f1 | xargs -I@ markmap markmap/@.md --output @.html --no-open
      - name: Deploy
        run: |
          git checkout --orphan gh-pages
          git reset
          git add *.html .nojekyll
          git commit -m "Generate markmap by Github Action"
          git push -u origin HEAD:gh-pages -f

2022-11-07

Mermaid example with better theme color

%%{init: {'theme': 'base','themeVariables': { 'secondaryColor': '#ffffff'}}}%%
flowchart TD
    A[Start] --> B{Is it?}
    B -->|Yes| C[OK]
    C --> D[Rethink]
    D --> B
    B ---->|No| E[End]

2022-10-04

Kibana - Generate csv Max size reached

Max size reached

Increase maxSizeBytes, it defaults to 10MB.

[root@ip-10-16-41-158 kibana]# vi kibana.yml

# xpack.reporting.csv.maxSizeBytes: 104857600

[root@ip-10-16-41-158 kibana]# systemctl restart kibana

2022-07-14

ElasticSeach - Snapshot to S3 throughtput

770MB/s

env:

  • AWS: c5.9xlarge
  • 50 GB/shard, 20 shard, 1TB data
  • Total run time: 1300s
  • thread pool: snapshot:5

2022-06-02

Elasticsearch - Searchable snapshot won't delete index from hot nodes

Use explain can see

GET _cluster/allocation/explain

"node_decision" : "no",
"deciders" : [
  {
    "decider" : "has_frozen_cache",
    "decision" : "NO",
    "explanation" : "node setting [xpack.searchable.snapshot.shared_cache.size] is set to zero, so frozen searchable snapshot shards cannot be allocated to this node"
  }
]

So, just add new node with

node.roles: ["data_frozen"]

2022-05-27

Elasticsearch - Failed step attempt-rollover

Index lifecycle error
exception: Concurrent modification of alias [rammus-log] during rollover
GET .ds-rammus-log-002774/_ilm/explain
POST .ds-rammus-log-002773/_ilm/retry

2022-05-03

openssl - p12 expired date

openssl pkcs12 -in elastic-certificates.p12 -nodes | openssl x509 -noout -enddate

Demo

❯ openssl pkcs12 -in elastic-certificates.p12 -clcerts -nodes | openssl x509 -noout -enddate

Enter Import Password:
MAC verified OK
notAfter=Jan 12 08:40:45 2021 GMT

2022-05-02

Free image 免費高清圖庫

2022-04-28

EKS - Can't create NLB

{"level":"error","ts":1651135177.483026,"logger":"controller-runtime.manager.controller.targetGroupBinding","msg":"Reconciler error","reconciler group":"elbv2.k8s.aws","reconciler kind":"TargetGroupBinding","name":"k8s-eckpoc-quicksta-6b38f0416c","namespace":"eckpoc","error":"expect exactly one securityGroup tagged with kubernetes.io/cluster/poc-cluster for eni eni-0e226022e7dd6aa80, got: [sg-011da083ca19a29ef sg-0446f7a0f894dd9a3]"}
eni eni-0e226022e7dd6aa80, got: [sg-011da083ca19a29ef sg-0446f7a0f894dd9a3]"}

Solution

Remove the sg with description: EKS created security group applied to ENI that is attached to EKS Control Plane master nodes, as well as any managed workloads.

I guess it doesn't matters to remove which sg. Both can work.

Ref:

2022-04-26

Google Sheet - Insert image from Google Drive

  1. share image to Everyone
  2. Get link: https://drive.google.com/file/d/1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x/view?usp=sharing
  3. Use url in these ways: a. https://lh6.googleusercontent.com/d/1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x b. https://drive.google.com/thumbnail?id=1uw0e0ifHq2ibLIAFw8HH7RPKGe-u7i5x

2022-03-29

ElasticSearch - Kibana can't compatible with downstair versions.

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.16.3-x86_64.rpm
yum localinstall -y kibana-7.16.3-x86_64.rpm

{"type":"log","@timestamp":"2022-03-29T07:53:52+00:00","tags":["error","elasticsearch-service"],"pid":10677,"message":"This version of Kibana (v7.16.3) is incompatible with the following Elasticsearch nodes inyour cluster:v7.9.3 @ 10.16.41.65:9200 (10.16.41.65), v7.9.3 @ 10.16.41.126:9200 (10.16.41.126) ...}

Kibana - Install in yum

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.16.3-x86_64.rpm
yum localinstall -y kibana-7.16.3-x86_64.rpm

2022-03-15

ElasticSearch - cluster full restart needs .security-7 index shard

  • master might store auth data in memory
  • when masters shutdown, they will need data node contains .security-7 index shard
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

2022-03-11

ElasticSearch - Change mapping fail, cannot be changed from type [text] to [keyword]

"mapper [message] cannot be changed from type [text] to [keyword]"

solution:

  1. Create new index with new mapping

    POST _reindex
    {
      "source": {
        "index": "rammus-poc"
      },
      "dest": {
        "index": "rammus-poc-20220311"
      }
    }
    

  2. Use _reindex to migrate data

    PUT rammus-poc-20220311
    {
        "settings": {
            "index": {
                "routing": {
                    "allocation": {
                        "total_shards_per_node": "1"
                    }
                },
                "number_of_shards": "1",
                "priority": "500",
                "number_of_replicas": "1"
            }
        },
        "mappings": {
            "properties": {
              ...
            }
        }
    }
    

2022-03-09

awscli - disable ec2 termination protection

aws ec2 modify-instance-attribute --no-disable-api-termination --instance-id i-xxxxx --profile rammus-dev

2022-03-07

ElasticSearch - update keystore password

echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.transport.ssl.keystore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.transport.ssl.truststore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.http.ssl.keystore.secure_password"
echo "" | /usr/share/elasticsearch/bin/elasticsearch-keystore add -x -f "xpack.security.http.ssl.truststore.secure_password"

Teleport - for loop run command in multiple machines

for ip in 172.21.1.1 172.21.1.2 172.21.1.3
do
  tsh ssh ec2-user@${ip} sudo systemctl restart elasticsearch
done

amazon linux - install and start docker

yum install docker

## Run docker daemon
systemctl start docker

## Add permission
newgrp docker
sudo usermod -aG docker ec2-user

Last update: November 11, 2022