elasticsearch集群部署文档

2020-11-24 leiting (3563阅读)

标签 Elasticsearch

ES集群分布：

192.168.2.23:

master/data/data

192.168.2.115:

master/data/data/client

192.168.2.116:

master/data/data/client

192.168.2.117:

data/data/client

192.168.2.118:

data/data/client

单机ES部署步骤：

需要安装：

es安装包（5.5.1已集成ik、x-pack）

jdk(1.8.0-161)

apache-tomcat(http提供分词文件)

kibaba-5.5.1

elasticsearch-head/node

以上可以在@192.168.2.23:/DATA1/esInstallEnv中找到

一、配置elasticsearch

1.es配置

将esInstallEnv中esHotel.tar.gz解压到 /DATA(192.168.2.23为DATA1，下面都以192.168.2.23举例)

tar -zxvf esHotel.tar.gz -C /DATA1

cd /DATA1/es

/DATA1/es下为4个目录：

elasticsearch-5.5.1-client-9206

elasticsearch-5.5.1-data-9207

elasticsearch-5.5.1-data-9208

elasticsearch-5.5.1-master-9205

为配置好的两个data、一个master、一个client，最后的数字代表端口号，按需启用项目即可，以master为例，配置：

vim elasticsearch-5.5.1-master-9205/config/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: hotelcluster

# ------------------------------------ Node ------------------------------------

node.name: ${HOSTNAME}-master9205

node.master: true

node.data: false

node.ingest: false

node.attr.rack: first

# ----------------------------------- Paths ------------------------------------

# Path to log files:

#path.logs: /path/to/logs

# ----------------------------------- Memory -----------------------------------

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------

network.host: 192.168.2.

# http.port: 9205

transport.tcp.port: 9305

# --------------------------------- Discovery ----------------------------------

discovery.zen.ping.unicast.hosts: ["192.168.2.23:9305","192.168.2.115:9305","192.168.2.116:9305"]

discovery.zen.minimum_master_nodes: 2

# ---------------------------------- Various -----------------------------------

xpack.security.enabled: false

http.cors.enabled: true

http.cors.allow-origin: "*"

基本配置都已设置好，一般情况下只用设置network.host为本机ip如：

network.host: 192.168.2.23

多块硬盘的机器，可以将data数据指定不同的硬盘，能提高效率和可靠性

path.data: /DATA1/esdata/,/DATA2/esdata/

jvm配置文件config/jvm.options一般情况也不做变动

## JVM configuration

-Xms10g

-Xmx10g

## GC configuration

-XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

-XX:+AlwaysPreTouch

## basic

-Xverify:none

-XX:NewRatio=2

-XX:SurvivorRatio=4

-XX:+UseParNewGC

-XX:+CMSScavengeBeforeRemark

-XX:+ScavengeBeforeFullGC

# force the server VM (remove on 32-bit client JVMs)

-server

# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)

-Xss1m

# set to headless, just in case

-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)

-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one

-Djna.nosys=true

# use old-style file permissions on JDK9

-Djdk.io.permissionsUseCanonicalPath=true

# flags to configure Netty

-Dio.netty.noUnsafe=true

-Dio.netty.noKeySetOptimization=true

-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2

-Dlog4j.shutdownHookEnabled=false

-Dlog4j2.disable.jmx=true

-Dlog4j.skipJansi=true

## heap dumps

-XX:+HeapDumpOnOutOfMemoryError

## GC logging

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-XX:+PrintGCDateStamps

#-XX:+PrintClassHistogram

#-XX:+PrintTenuringDistribution

-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps

# ensure the directory exists

-Xloggc:../logs/gc.log

# By default, the GC log file will not rotate.

# By uncommenting the lines below, the GC log file

# will be rotated every 128MB at most 32 times.

-XX:+UseGCLogFileRotation

-XX:NumberOfGCLogFiles=32

-XX:GCLogFileSize=128M

ik分词器配置文件：plugins/ik/config/IKAnalyzer.cfg.xml一般情况不要修改

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">

<comment>IK Analyzer 扩展配置</comment>

<entry key="ext_dict">custom/mydict.dic;custom/mydict2.dic;custom/single_word_low_freq.dic;custom/sougou.dic;custom/mdd.dic</entry>

<entry key="ext_stopwords">custom/ext_stopword.dic</entry>

<entry key="remote_ext_dict">http://192.168.2.23:8080/ikdic/extHotwords.dic</entry>

<entry key="remote_ext_stopwords">http://192.168.2.23:8080/ikdic/extStopwords.dic</entry>

</properties>

如果程序执行中报错找不到ik分词器：failed to find analyzer [ik]

是因为新版本的ik分词器变为了ik_max_word(默认)和ik_smart

在新建索引时设置settings自定义分词器ik指向ik_max_word即可：

"index.analysis.analyzer.ik.tokenizer" : "ik_max_word",

2.系统参数配置

ES需要修改一些系统参数来发挥性能：

vim /etc/security/limits.conf

elastic soft memlock unlimited

elastic hard memlock unlimited

vim /etc/sysctl.conf

vm.swappiness = 1

vm.max_map_count = 262144

net.core.somaxconn = 2048

net.netfilter.nf_conntrack_max = 1048576

使sysctl生效：

sysctl -p

所有ES节点配置完毕先不要启动，先确认jdk是否安装以及apache-tomcat是否在线（http方式提供分词文件）

二、JDK安装

解压jdk1.8.0_161.tar.gz到/usr/java目录

tar -zxvf jdk1.8.0_161.tar.gz -C /usr/java

配置环境变量

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_161/

export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile

三、apache-tomcat

tar -zxvf apache-tomcat-8.5.20.tar.gz -C /DATA1

在apache-tomcat-8.5.20/webapps/ROOT/中创建目录ikdic

在ikdic中新建文件extHotwords.dic、extStopwords.dic（对应上面ik分词配置中的字典名称）

cd /DATA1/apache-tomcat-8.5.20/webapps/ROOT/ikdic

touch extHotwords.dic

touch extStopwords.dic

启动tomcat

/DATA1/apache-tomcat-8.5.20/bin/startup.sh

停止等操作脚本也在这个目录

四、kibana安装

tar -zxvf kibana-5.5.1-linux-x86_64.tar.gz -C /DATA1

vim /DATA1/kibana-5.5.1-linux-x86_64/config/kibana.yml

server.port: 5601

server.host: "0.0.0.0"

#与ES一致

server.name: "hotelcluster"

#任意一个ES client节点

elasticsearch.url: "http://192.168.2.23:9206"

xpack.security.enabled: false

安装X-Pack

vim /DATA1/kibana-5.5.1-linux-x86_64/bin/kibana-plugin install x-pack

等ES client节点启动后再启动kibana

五、elasticsearch-head/node.js安装

elasticsearch-head不是必须的，想用的话可以装一个，需要先暗中node.js

tar xvJf node-v6.11.2-linux-x64.tar.xz -C /DATA1

配置环境变量/etc/profile，在java的path后增加node.js目录

export PATH=$PATH:$JAVA_HOME/bin:/DATA1/node-v6.11.2-linux-x64/bin

source /etc/profile

从git拉取head

git clone https://github.com/mobz/elasticsearch-head.git cd elasticsearch-head

npm install

nohup npm run start &

即可访问

http://ip:9100/

启动集群：

启动ES

新建elastic用户

useradd elastic

将es及其中文件修改为elastic用户

chown elastic:elastic /DATA1/es -R

su elastic

启动es必须进入elasticsearch的bin目录，启动集群时先启动所有的master节点，以192.168.2.23为例：

cd /DATA1/es/elasticsearch-5.5.1-master-9205/bin/

./elasticsearch -d

将192.168.2.23/115/116的master节点启动以后，再启动所有所需的data与client节点

启动kibana

cd /DATA1/kibana-5.5.1-linux-x86_64/

nohup bin/kibana > logs/kibana.log 2>&1 &

x-pack license更新

官方文档：https://www.elastic.co/guide/en/x-pack/5.6/installing-license.html

license文件需要向es集群提交，elastic用户密码：123456，需执行命令：

curl -XPUT -u elastic 'http://192.168.2.115:9206/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json

license.json是license的json文件

IT PHP 编程语言 Linux 开发编程科技 Elasticsearch HTML/CSS/XML 网络数据库 JAVA 面试 NoSQL C/C++ Golang Git 算法操作系统正则表达式 Redis 互联网 MySql JavaScript 运维软件国际架构设计 Mac OS TCP/IP Excel Vim Windows Socket Oracle VR MongoDB Python 运营 MemCache 商业硬件电子娱乐设计 nginx 摄影游戏 WordPress HTTP 团建数码电器

略微加速

略速 - 互联网笔记