Elasticsearch服务部署

2024-11-29 15:52:13 Albin Wong 444

## 依赖服务部署

依赖服务的目录结构

```bash
/opt/soft/elastic/
└── elasticsearch-7.17.4
└── elasticsearch-head
├── logstash-8.13.3
└── search-guard-ssl-5.3-22
```

### 准备安装文件

```bash
mkdir -p /opt/down/elastic/7.17.4/plugins

cd /opt/down/elastic/7.17.4/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.4.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-8.13.3.tar.gz

# logstash-input-jdbc jdbc_driver_library
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-6.0.6.tar.gz

# SSL tool
wget -O search-guard-ssl-5.3-22.tar.gz https://github.com/floragunncom/search-guard-ssl/archive/v5.3-22.tar.gz

cd plugins/
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.3.2/elasticsearch-analysis-ik-5.3.2.zip
wget https://oss.sonatype.org/service/local/repositories/releases/content/com/floragunn/search-guard-5/5.3.2-15/search-guard-5-5.3.2-15.zip
```

### ElasticSearch 部署

一、安装 ElasticSearch

```bash
mkdir -p /opt/soft/elastic

cd /opt/down/elastic/7.17.4/
tar xzf elasticsearch-7.17.4.tar.gz -C /opt/soft/elastic/
```

二、安装 elasticsearch-analysis-ik

```bash
bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.17.4/elasticsearch-analysis-ik-7.17.4.zip
```

```bash
mkdir /opt/soft/elastic/elasticsearch-7.17.4/plugins/ik

cd /opt/down/elastic/7.17.4/plugins
unzip elasticsearch-analysis-ik-5.3.2.zip -d /opt/soft/elastic/elasticsearch-7.17.4/plugins/ik/
```

```bash
bin/elasticsearch-plugin install https://get.infini.cloud/elasticsearch/analysis-pinyin/7.17.4
```

三、安装 search-guard

3.1 离线安装
```bash
./bin/elasticsearch-plugin install -b file:///opt/down/elastic/7.17.4/plugins/search-guard-5-5.3.2-15.zip
```

在线安装
```bash
./bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:5.3.2-15
```

3.2 生成证书
- PKI 脚本 [官方文档](https://docs.search-guard.com/v5/tls-certificates-sample-scripts)

官方有多种方式来生成证书，出于安全考虑，使用离线生成证书。

```bash
cd /opt/down/elastic/7.17.4
tar xzf search-guard-ssl-5.3-22.tar.gz -C /opt/soft/elastic/

cd /opt/soft/elastic/search-guard-ssl-5.3-22/example-pki-scripts/
sh example.sh

# 将jks证书文件，复制到ES config目录中
cp node-1-keystore.jks truststore.jks /opt/soft/elastic/elasticsearch-7.17.4/config/
```

3.3 配置 ElasticSearch

编辑配置文件`/opt/soft/elastic/elasticsearch-7.17.4/config/elasticsearch.yml`，根据实际情况写入以下内容
```
######## Start Search Guard Configuration ########
searchguard.ssl.transport.keystore_filepath: node-1-keystore.jks
searchguard.ssl.transport.keystore_password: changeit
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: changeit
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
  - "CN=kirk, OU=client, O=client, L=Test, C=DE"
script.inline: true

######## Enable CORS in elasticsearch ########
# elasticsearch-head
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: "X-Requested-With,Content-Type,Content-Length,Authorization"
```

启动 ElasticSearch
```
/opt/soft/elastic/elasticsearch-7.17.4/bin/elasticsearch -d
```

四、初始化 ES Search-Guard

`sgconfig/sg_internal_users.yml`文件中定义了各个用户的密码，如需自定义，请酌情修改。

```
cd /opt/soft/elastic/elasticsearch-7.17.4/plugins/search-guard-5/tools
./sgadmin.sh \
   -cd ../sgconfig/ \
   -ks ../../../config/kirk-keystore.jks \
   -kspass changeit \
   -ts ../../../config/truststore.jks \
   -tspass changeit \
   -nhnv \
   -icl
```

验证

弹出认证框，用户密码默认均是：`admin` ，能够验证通过并打开页面即可。
```bash
http://10.183.1.127:9200/_searchguard/authinfo
```

### elasticsearch-head 安装
```bash
cd /opt/soft/elastic/
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head

npm config set registry https://registry.npm.taobao.org
npm install
npm run start

# 放入后台启动
# nohup npm run start &> /dev/null &
```

elasticsearch-plugin
### 创建项目索引
访问`ElasticSearch-head`服务，连接ES集群。

复合查询 ---> 查询地址(默认) ---> 输入索引名 ---> 使用PUT方法 ---> 内容区输入Mapping内容

Mapping内容见：[es.qiaomen.index.mapping](es.qiaomen.index.mapping)

- 索引名命名: qiaomen_v20240509
每次重建index，可以根据日期来命名，之后将qiaomen的别名映射到新的索引上。
- 根据实际集群节点情况修改`number_of_replicas`，单机ES部署设置为0.（总节点数-1）

> 命令行创建：curl -XPUT 127.0.0.1:9200/qiaomen_v20240509 -d '{ mapping_content }'

更多说明见：[索引管理](index_manage.md)

### Logstash
部署配置Logstash从数据库中读取所需信息，写入ES集群中。

一、部署
```bash
cd /opt/down/elastic/7.17.4/
tar xzf logstash-8.13.3.tar.gz -C /opt/soft/elastic/
tar xzf mysql-connector-java-6.0.6.tar.gz -C /opt/soft/elastic/logstash-8.13.3/lib/
```

二、配置

从项目`docment`目录中，复制`logstash_import_index`目录到`logstash` `config`目录下。
并命名其与索引名相同，以方便后续重建管理。
```bash
cp -rf /www/wwwroot/api.izuohua.com/document/logstash_import_index /opt/soft/elastic/logstash-8.13.3/config/qiaomen_v20240509
```

根据实际情况配置`import_index.conf`配置，然后启动Logstash查看日志情况。
```bash
cd /opt/soft/elastic/logstash-8.13.3/
./bin/logstash -f  config/qiaomen_v20181130/import_index.conf
```

logstash 读取数据库内容写入ES，在sql目录下会产生`.last`结尾的文件，此文件记录读取数据库的进度。

### 配置索引别名
项目`.env`中配置读取ES集群的`qiaomen`索引。

可以登录elasticsearch-head服务，通过WEB管理alias

也可以命令行通过`curl`来配置。
```bash
curl -XPOST -u es_username:es_password 127.0.0.1:9200/_aliases -d '{
    "actions": [
        { "remove": { "index": "qiaomen_v_old", "alias": "qiaomen" }},
        { "add":    { "index": "qiaomen_v_new", "alias": "qiaomen" }}
    ]
}'
```

### 参考
- [Github - search-guard](https://github.com/floragunncom/search-guard)
- [Github - search-guard-ssl](https://github.com/floragunncom/search-guard-ssl)
- [Github - elasticsearch-analysis-ik](https://github.com/medcl/elasticsearch-analysis-ik)
- [Github - elasticsearch-head](https://github.com/mobz/elasticsearch-head)
- [Github - logstash-input-jdbc](https://github.com/logstash-plugins/logstash-input-jdbc)

(3) Comments

John Doe- Reply

February 26, 2013 at 1:18 pm
- saqib sarwar- Reply
  
  February 26, 2013 at 1:20 pm
  
  Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Typi non habent claritatem insitam; est usus legentis in iis qui facit eorum claritatem. Investigationes demonstraverunt lectores legere me lius quod ii legunt saepius. Claritas est etiam processus dynamicus, qui sequitur mutationem consuetudium lectorum. Mirum est notare quam littera gothica, quam nunc putamus parum claram, anteposuerit litterarum formas humanitatis per seacula quarta decima et quinta decima. Eodem modo typi, qui nunc nobis videntur parum clari, fiant sollemnes in futurum.
John Doe- Reply

February 26, 2013 at 1:27 pm

Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum.

Your email address will not be published. Required fields are marked *

Name *

Email *

Website

Comment

You can use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Elasticsearch服务部署

(3) Comments

John Doe- Reply

saqib sarwar- Reply

John Doe- Reply

Leave a Reply