Linux-Manual

Elasticsearch 知识

Docker 单节点部署

5.6.x

version: '3'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.8
    container_name: elasticsearch-5.6.8
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "cluster.name=elasticsearch"
      - "network.host=0.0.0.0"
      - "http.host=0.0.0.0"
      - "xpack.security.enabled=false"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - /data/docker/elasticsearch/data:/usr/share/elasticsearch/data

6.7.x(带 ik 分词)

version: '3'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.2
    container_name: elasticsearch-6.7.2
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "cluster.name=elasticsearch"
      - "network.host=0.0.0.0"
      - "http.host=0.0.0.0"
      - "xpack.security.enabled=false"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - 9200:9200
      - 9300:9300
    volumes:
      - /data/docker/elasticsearch-6.7.2/data:/usr/share/elasticsearch/data
      - /data/docker/ik:/usr/share/elasticsearch/plugins/ik
http://localhost:9200/
_analyze?pretty   POST


{"analyzer":"ik_smart","text":"安徽省长江流域"}
ik_max_word: 会将文本做最细粒度的拆分,比如会将“中华人民共和国国歌”拆分为“中华人民共和国,中华人民,中华,华人,人民共和国,人民,人,民,共和国,共和,和,国国,国歌”,会穷尽各种可能的组合,适合 Term Query;
ik_smart: 会做最粗粒度的拆分,比如会将“中华人民共和国国歌”拆分为“中华人民共和国,国歌”,适合 Phrase 查询。

Elasticsearch 6.5.x 安装(适配与 5.5.x,6.6.x)

环境

先配置部分系统变量

fs.file-max=65535
vm.max_map_count=262144
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
* soft nofile 262144
* hard nofile 262144

开始安装

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

RPM 安装后的一些配置位置说明

配置

打开这个注释:#cluster.name: my-application
集群名称最好是自己给定,不然有些 client 端会连不上,或者要求填写

打开这个注释:#network.host: 192.168.0.1
改为:network.host: 0.0.0.0

安装 X-Pack(6.5.x 默认带了 x-pack)

GUI 客户端工具

安装 Chrome 扩展的 Head

其他细节

创建索引并设置 mapping

curl -XPUT 'http://127.0.0.1:9200/grafanadb' -H 'Content-Type: application/json' -d'
{
  "settings": {
    "refresh_interval": "5s",
    "number_of_shards": 5,
    "number_of_replicas": 0
  },
  "mappings": {
    "radar": {
      "properties": {
        "request_num": {
          "type": "long"
        },
        "post_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||epoch_millis"
        }
      }
    }
  }
}
'

批量增加 / 删除测试数据

curl -X POST "http://127.0.0.1:9200/_bulk" -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "grafanadb", "_type" : "radar", "_id" : "100001" } }
{ "post_date" : "2018-12-01 10:00:00", "request_num" :  1 }
{ "index" : { "_index" : "grafanadb", "_type" : "radar", "_id" : "100002" } }
{ "post_date" : "2018-12-01 10:00:05", "request_num" :  2 }
{ "index" : { "_index" : "grafanadb", "_type" : "radar", "_id" : "100003" } }
{ "post_date" : "2018-12-01 10:00:10", "request_num" :  3 }
{ "index" : { "_index" : "grafanadb", "_type" : "radar", "_id" : "100004" } }
{ "post_date" : "2018-12-01 10:00:15", "request_num" :  4 }
{ "index" : { "_index" : "grafanadb", "_type" : "radar", "_id" : "100005" } }
{ "post_date" : "2018-12-01 10:00:20", "request_num" :  5 }
'
curl -X POST "http://127.0.0.1:9200/_bulk" -H 'Content-Type: application/json' -d'
{ "delete": { "_index": "grafanadb", "_type": "radar", "_id": "100001" } }
{ "delete": { "_index": "grafanadb", "_type": "radar", "_id": "100002" } }
'
curl -X POST "http://127.0.0.1:9200/索引名称/类型名称/_delete_by_query?refresh&slices=5&pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_all": {}
  }
}
'

Elasticsearch 5.2.0 安装

环境

zip 解压安装

cluster.name: youmeek-cluster
node.name: youmeek-node-1
path.data: /opt/elasticsearch/data
path.logs: /opt/elasticsearch/log
bootstrap.memory_lock: true
network.host: 0.0.0.0 # 也可以是本机 IP
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.127"]  #如果有多个机子集群,这里就写上这些机子的 IP,格式:["192.168.1.127","192.168.1.126"]
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
* soft nofile 262144
* hard nofile 262144
vm.max_map_count=262144
{
  "name" : "youmeek-node-1",
  "cluster_name" : "youmeek-cluster",
  "cluster_uuid" : "c8RxQdOHQJq-Tg8rrPi_UA",
  "version" : {
    "number" : "5.2.0",
    "build_hash" : "24e05b9",
    "build_date" : "2017-01-24T19:52:35.800Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.0"
  },
  "tagline" : "You Know, for Search"
}

安装 Kibana 5.2.0

tar.gz 解压安装

server.port: 5601
server.host: "0.0.0.0" # 请将这里改为 0.0.0.0 或是当前本机 IP,不然可能会访问不了
erver.name: "youmeek-kibana"
elasticsearch.url: "http://192.168.1.127:9200"
elasticsearch.username: "elasticsearch"
elasticsearch.password: "123456"

Beats

Beats 资料

安装 X-Pack 或是其他插件


2.4.X

安装 elasticsearch 集群

下载

tar 解压安装

cluster.name: gitnavi-cluster
node.name: gitnavi-node-1
path.data: /opt/elasticsearch/data
path.logs: /opt/elasticsearch/log
bootstrap.memory_lock: true
network.host: 0.0.0.0 # 也可以是本机 IP
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.1.127", "192.168.1.126"]  #这个为两台机子的 IP 地址
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
* soft nofile 262144
* hard nofile 262144
[2017-03-13 18:42:51,170][INFO ][node                     ] [gitnavi-node-1] version[2.4.1], pid[21156], build[c67dc32/2016-09-27T18:57:55Z]
[2017-03-13 18:42:51,177][INFO ][node                     ] [gitnavi-node-1] initializing ...
[2017-03-13 18:42:51,821][INFO ][plugins                  ] [gitnavi-node-1] modules [reindex, lang-expression, lang-groovy], plugins [head, kopf], sites [head, kopf]
[2017-03-13 18:42:51,852][INFO ][env                      ] [gitnavi-node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [12.4gb], net total_space [17.4gb], spins? [unknown], types [rootfs]
[2017-03-13 18:42:51,852][INFO ][env                      ] [gitnavi-node-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2017-03-13 18:42:54,094][INFO ][node                     ] [gitnavi-node-1] initialized
[2017-03-13 18:42:54,094][INFO ][node                     ] [gitnavi-node-1] starting ...
[2017-03-13 18:42:54,175][INFO ][transport                ] [gitnavi-node-1] publish_address {192.168.1.127:9300}, bound_addresses {[::]:9300}
[2017-03-13 18:42:54,178][INFO ][discovery                ] [gitnavi-node-1] gitnavi-cluster/-XywT60EScO-9lgzjfnsgg
[2017-03-13 18:42:57,344][INFO ][cluster.service          ] [gitnavi-node-1] new_master {gitnavi-node-1}{-XywT60EScO-9lgzjfnsgg}{192.168.1.127}{192.168.1.127:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-03-13 18:42:57,410][INFO ][gateway                  ] [gitnavi-node-1] recovered [0] indices into cluster_state
[2017-03-13 18:42:57,414][INFO ][http                     ] [gitnavi-node-1] publish_address {192.168.1.127:9200}, bound_addresses {[::]:9200}
[2017-03-13 18:42:57,414][INFO ][node                     ] [gitnavi-node-1] started
{
  "name" : "gitnavi-node-1",
  "cluster_name" : "gitnavi-cluster",
  "cluster_uuid" : "0b66dYpnTd-hh7x4Phfm1A",
  "version" : {
    "number" : "2.4.1",
    "build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
    "build_timestamp" : "2016-09-27T18:57:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}
index.analysis.analyzer.default.tokenizer : "ik_max_word"
index.analysis.analyzer.default.type: "ik"
[
  {
    "token": "这是",
    "start_offset": 0,
    "end_offset": 2,
    "type": "CN_WORD",
    "position": 0
  },
  {
    "token": "一个",
    "start_offset": 2,
    "end_offset": 4,
    "type": "CN_WORD",
    "position": 1
  },
  {
    "token": "一",
    "start_offset": 2,
    "end_offset": 3,
    "type": "TYPE_CNUM",
    "position": 2
  },
  {
    "token": "个",
    "start_offset": 3,
    "end_offset": 4,
    "type": "COUNT",
    "position": 3
  },
  {
    "token": "针对",
    "start_offset": 4,
    "end_offset": 6,
    "type": "CN_WORD",
    "position": 4
  },
  {
    "token": "程序员",
    "start_offset": 6,
    "end_offset": 9,
    "type": "CN_WORD",
    "position": 5
  },
  {
    "token": "程序",
    "start_offset": 6,
    "end_offset": 8,
    "type": "CN_WORD",
    "position": 6
  },
  {
    "token": "序",
    "start_offset": 7,
    "end_offset": 8,
    "type": "CN_WORD",
    "position": 7
  },
  {
    "token": "员",
    "start_offset": 8,
    "end_offset": 9,
    "type": "CN_CHAR",
    "position": 8
  },
  {
    "token": "优化",
    "start_offset": 9,
    "end_offset": 11,
    "type": "CN_WORD",
    "position": 9
  },
  {
    "token": "导航",
    "start_offset": 12,
    "end_offset": 14,
    "type": "CN_WORD",
    "position": 10
  },
  {
    "token": "航",
    "start_offset": 13,
    "end_offset": 14,
    "type": "CN_WORD",
    "position": 11
  },
  {
    "token": "gitnavi.com",
    "start_offset": 14,
    "end_offset": 25,
    "type": "LETTER",
    "position": 12
  },
  {
    "token": "gitnavi",
    "start_offset": 14,
    "end_offset": 21,
    "type": "ENGLISH",
    "position": 13
  },
  {
    "token": "com",
    "start_offset": 22,
    "end_offset": 25,
    "type": "ENGLISH",
    "position": 14
  }
]

构建 elasticsearch 集群

资料