Microfaster, 让我们略微加速

elasticsearch集群部署文档

2019-04-18 13:55:53 lvesu (294阅读)

标签 Elasticsearch

ES集群分布:

192.168.2.23:

master/data/data

192.168.2.115:

master/data/data/client

192.168.2.116:

master/data/data/client

192.168.2.117:

data/data/client

192.168.2.118:

data/data/client


单机ES部署步骤:

需要安装:


es安装包(5.5.1已集成ik、x-pack)



jdk(1.8.0-161)



apache-tomcat(http提供分词文件)



kibaba-5.5.1



elasticsearch-head/node



以上可以在@192.168.2.23:/DATA1/esInstallEnv中找到



一、配置elasticsearch

1.es配置

将esInstallEnv中esHotel.tar.gz解压到 /DATA(192.168.2.23为DATA1,下面都以192.168.2.23举例)


tar -zxvf esHotel.tar.gz -C /DATA1



cd /DATA1/es


/DATA1/es下为4个目录:

elasticsearch-5.5.1-client-9206

elasticsearch-5.5.1-data-9207

elasticsearch-5.5.1-data-9208

elasticsearch-5.5.1-master-9205

为配置好的两个data、一个master、一个client,最后的数字代表端口号,按需启用项目即可,以master为例,配置:

vim elasticsearch-5.5.1-master-9205/config/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: hotelcluster

# ------------------------------------ Node ------------------------------------

node.name: ${HOSTNAME}-master9205

node.master: true

node.data: false

node.ingest: false

node.attr.rack: first

# ----------------------------------- Paths ------------------------------------

# Path to log files:

#path.logs: /path/to/logs

# ----------------------------------- Memory -----------------------------------

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

# ---------------------------------- Network -----------------------------------

network.host: 192.168.2.

# http.port: 9205

transport.tcp.port: 9305

# --------------------------------- Discovery ----------------------------------

discovery.zen.ping.unicast.hosts: ["192.168.2.23:9305","192.168.2.115:9305","192.168.2.116:9305"]

discovery.zen.minimum_master_nodes: 2

# ---------------------------------- Various -----------------------------------

xpack.security.enabled: false

http.cors.enabled: true

http.cors.allow-origin: "*"

基本配置都已设置好,一般情况下只用设置network.host为本机ip如:


network.host: 192.168.2.23


多块硬盘的机器,可以将data数据指定不同的硬盘,能提高效率和可靠性


path.data: /DATA1/esdata/,/DATA2/esdata/


jvm配置文件config/jvm.options一般情况也不做变动

## JVM configuration

-Xms10g

-Xmx10g

## GC configuration

-XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

-XX:+AlwaysPreTouch

## basic

-Xverify:none

-XX:NewRatio=2

-XX:SurvivorRatio=4

-XX:+UseParNewGC

-XX:+CMSScavengeBeforeRemark

-XX:+ScavengeBeforeFullGC

# force the server VM (remove on 32-bit client JVMs)

-server

# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)

-Xss1m

# set to headless, just in case

-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)

-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one

-Djna.nosys=true

# use old-style file permissions on JDK9

-Djdk.io.permissionsUseCanonicalPath=true

# flags to configure Netty

-Dio.netty.noUnsafe=true

-Dio.netty.noKeySetOptimization=true

-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2

-Dlog4j.shutdownHookEnabled=false

-Dlog4j2.disable.jmx=true

-Dlog4j.skipJansi=true

## heap dumps

-XX:+HeapDumpOnOutOfMemoryError

## GC logging

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-XX:+PrintGCDateStamps

#-XX:+PrintClassHistogram

#-XX:+PrintTenuringDistribution

-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps

# ensure the directory exists

-Xloggc:../logs/gc.log

# By default, the GC log file will not rotate.

# By uncommenting the lines below, the GC log file

# will be rotated every 128MB at most 32 times.

-XX:+UseGCLogFileRotation

-XX:NumberOfGCLogFiles=32

-XX:GCLogFileSize=128M

ik分词器配置文件:plugins/ik/config/IKAnalyzer.cfg.xml一般情况不要修改

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">

<properties>

<comment>IK Analyzer 扩展配置</comment>

<!--用户可以在这里配置自己的扩展字典 -->

<entry key="ext_dict">custom/mydict.dic;custom/mydict2.dic;custom/single_word_low_freq.dic;custom/sougou.dic;custom/mdd.dic</entry>

         <!--用户可以在这里配置自己的扩展停止词字典-->

        <entry key="ext_stopwords">custom/ext_stopword.dic</entry>

        <!--用户可以在这里配置远程扩展字典 -->

        <entry key="remote_ext_dict">http://192.168.2.23:8080/ikdic/extHotwords.dic</entry>

        <!--用户可以在这里配置远程扩展停止词字典-->

        <entry key="remote_ext_stopwords">http://192.168.2.23:8080/ikdic/extStopwords.dic</entry>

</properties>

如果程序执行中报错找不到ik分词器:failed to find analyzer [ik]

是因为新版本的ik分词器变为了ik_max_word(默认)和ik_smart

在新建索引时设置settings自定义分词器ik指向ik_max_word即可:


"index.analysis.analyzer.ik.tokenizer" : "ik_max_word",


2.系统参数配置

ES需要修改一些系统参数来发挥性能:


vim /etc/security/limits.conf


elastic soft memlock unlimited

elastic hard memlock unlimited


vim /etc/sysctl.conf


vm.swappiness = 1

vm.max_map_count = 262144

net.core.somaxconn = 2048

net.netfilter.nf_conntrack_max = 1048576

使sysctl生效:


sysctl -p


所有ES节点配置完毕先不要启动,先确认jdk是否安装以及apache-tomcat是否在线(http方式提供分词文件)


二、JDK安装

解压jdk1.8.0_161.tar.gz到/usr/java目录


tar -zxvf jdk1.8.0_161.tar.gz -C /usr/java


配置环境变量


vim /etc/profile


export JAVA_HOME=/usr/java/jdk1.8.0_161/

export PATH=$PATH:$JAVA_HOME/bin


source /etc/profile



三、apache-tomcat


tar -zxvf apache-tomcat-8.5.20.tar.gz -C /DATA1


在apache-tomcat-8.5.20/webapps/ROOT/中创建目录ikdic

在ikdic中新建文件extHotwords.dic、extStopwords.dic(对应上面ik分词配置中的字典名称)


cd /DATA1/apache-tomcat-8.5.20/webapps/ROOT/ikdic



touch extHotwords.dic



touch extStopwords.dic


启动tomcat


/DATA1/apache-tomcat-8.5.20/bin/startup.sh



停止等操作脚本也在这个目录



四、kibana安装


tar -zxvf kibana-5.5.1-linux-x86_64.tar.gz -C /DATA1



vim /DATA1/kibana-5.5.1-linux-x86_64/config/kibana.yml


server.port: 5601

server.host: "0.0.0.0"

#与ES一致

server.name: "hotelcluster"

#任意一个ES client节点

elasticsearch.url: "http://192.168.2.23:9206"

xpack.security.enabled: false

安装X-Pack


vim /DATA1/kibana-5.5.1-linux-x86_64/bin/kibana-plugin install x-pack


等ES client节点启动后再启动kibana


五、elasticsearch-head/node.js安装

elasticsearch-head不是必须的,想用的话可以装一个,需要先暗中node.js


tar xvJf node-v6.11.2-linux-x64.tar.xz -C /DATA1


配置环境变量/etc/profile,在java的path后增加node.js目录


export PATH=$PATH:$JAVA_HOME/bin:/DATA1/node-v6.11.2-linux-x64/bin


source /etc/profile

从git拉取head

git clone https://github.com/mobz/elasticsearch-head.git cd elasticsearch-head

npm install

nohup npm run start &

即可访问


http://ip:9100/



启动集群:


启动ES

新建elastic用户


useradd elastic


将es及其中文件修改为elastic用户


chown elastic:elastic /DATA1/es -R



su elastic


启动es必须进入elasticsearch的bin目录,启动集群时先启动所有的master节点,以192.168.2.23为例:


cd /DATA1/es/elasticsearch-5.5.1-master-9205/bin/



./elasticsearch -d


将192.168.2.23/115/116的master节点启动以后,再启动所有所需的data与client节点


启动kibana


cd /DATA1/kibana-5.5.1-linux-x86_64/



nohup bin/kibana > logs/kibana.log 2>&1 &



x-pack license更新

官方文档:https://www.elastic.co/guide/en/x-pack/5.6/installing-license.html

license文件需要向es集群提交,elastic用户密码:123456,需执行命令:


curl -XPUT -u elastic 'http://192.168.2.115:9206/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @license.json


license.json是license的json文件


北京半月雨文化科技有限公司.版权所有 京ICP备12026184号