Ingest Node

• 可以預先處理, 或是 pipeline _doc
• 用量很大的話可以獨立出來

Shard

• each index in Elasticsearch is allocated 5 primary shards and 1 replica
• a primary shard can technically contain up to 2147483519 (Integer.MAX_VALUE - 128) documents
• 50 GB per shard
  • https://discuss.elastic.co/t/limit-for-shard-size/40032

Config

• Don’t Cross 32 GB!
  • The moral of the story is this: even when you have memory to spare, try to avoid crossing the 32 GB heap boundary. It wastes memory, reduces CPU performance, and makes the GC struggle with large heaps.
  • Java uses a trick called compressed oops to prevent wasting pointer space
  • https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#heap-sizing
• bootstrap.memory_lock: true
• Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches.

Index

• Force-merge read-only indices

Reference

• http://alexander.holbreich.org/elasticsearch-configuration/
• https://www.elastic.co/guide/en/elasticsearch/guide/current/deploy.html

之前 Mount 的 Volume 無法部署到現在的 cluster，必須要用同一個 az 的機器，才能將 Volume 掛載到 Node 上。

kops edit ig nodes

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-12-27T09:07:29Z
  labels:
    kops.k8s.io/cluster: stage.k8s.feversocial.com
  name: nodes
spec:
  image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17
  machineType: t3.medium
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - ap-southeast-1a
  - ap-southeast-1c

kops update cluster --yes

如果不希望前面的 stage 失敗時，後面的 job 繼續執行，就要使用以下設定

build:
  when: manual
  allow_failure: false

deploy:
  when: on_success

Reference

• https://gitlab.com/gitlab-org/gitlab-ce/issues/30850
• https://docs.gitlab.com/ee/ci/yaml/#when

VPC

• AZ 之間延遲 0.5ms
• Osaka (大阪) 是日本的災難備援區

S3

Standard -> Infrequent Access (IA)-> Glacier
原價 -> 九折 -> 一折

IAM

檢查權限

https://policysim.aws.amazon.com/home/index.jsp?#

Amazon Polly

是將文字轉換成逼真說話方式的服務，能夠讓您建立會說話的應用程式和打造全新的啟用語音產品類別

EMR

RDS -> storage

Kinesis

Streaming

BREACH attack: 因為 gzip 是使用 HTTP compression，可以觀察 HTTPS 加密後的文本規則，藉此取得用戶的資訊

• 可以繞過 HTTPS ，取得加密前的文本
• static page 不影響
• 不重要的 cookie 不影響
• 可以取得 CSRF token
  • Framework/Library 可以稍微緩解被取得的可能性
  • 我們的網站目前沒有防禦 CSRF
• Facebook 原本也是用 accountID + datetime 每天產生 CSRF token
  • 為了防禦這個問題，進化成每次 request 都有新的 CSRF token

可能影響範圍

• 用戶資料外流

Reference

• Facebook Preventing a “BREACH” Attack https://www.facebook.com/notes/protect-the-graph/preventing-a-breach-attack/1455331811373632/
• Akamai BREACH Attack https://www.akamai.com/uk/en/resources/breach-attack.jsp
• 新的 HTTPS 攻擊：BREACH Compression Attack https://blog.gslin.org/archives/2013/08/16/3434/%E6%96%B0%E7%9A%84-https-%E6%94%BB%E6%93%8A%EF%BC%9Abreach-compression-attack/
• BREACH vulnerability in compressed HTTPS https://www.kb.cert.org/vuls/id/987798/
• 讓我們來談談 CSRF https://blog.techbridge.cc/2017/02/25/csrf-introduction/