elasticsearch安装dynamic-synonym插件

elasticsearch安装dynamic-synonym插件

​ 今天就来和大家讲讲如何在es中安装dynamic-synonym插件，首先我们需要去github上下载与es版本对应的插件，一般github上基本都是本地词库和远程文本词库的，在gitee上可以找到采用数据库作为词库的源码，大致思路就是修改一些参数配置，然后自己创建一个表作为同义词词库，最后将打包好的jar包插件丢到es-plugins目录下面，最后重启一下就能跑起来了。但是！！！作者没有跑起来，遇到了好多问题【哭泣泣】，因为我是在docker容器中运行的es，而容器一直报的是Java权限问题，我在网络上找了一圈才东拼西凑的把这个问题给解决，真的太高兴啦！！！

​ 接下来就开始讲讲思路

1. 下载源码，修改dynamic-synonym配置
2. 新增MySQL代码
3. 创建一个dynamic-synonym的表
4. 修改docker中es容器的Java.policy文件**【非常重要】**
5. 将打包好的jar包放入到 {es-root}/es-plugins目录下面
6. docker重启es容器
7. 新建es的dynamic-synonym索引测试

**文章末尾会给出作者已经配置好的插件代码！！！！！！ 请注意签收！！！！！**可以直接跳到四或者五，根据你自己的需求来选择

一、下载源码并且修改配置

​ github好多好多的源码啊，真的是看都看不过来，下载之后要结合自己es版本切换分支，这里建议直接下载最原始的源码，链接为：https://github.com/bells/elasticsearch-analysis-dynamic-synonym，下载好了之后需要切换与es版本对应代码分支，作者的es版本为7.12.1，修改一下pom文件的配置

1.1 修改pom.xml文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0modelVersion><groupId>com.bellszhu.elasticsearchgroupId><artifactId>elasticsearch-analysis-dynamic-synonymartifactId><version>7.12.1version><packaging>jarpackaging><name>elasticsearch-dynamic-synonymname><description>Analysis-plugin for synonymdescription><properties><project.build.sourceEncoding>UTF-8project.build.sourceEncoding><elasticsearch.version>${project.version}elasticsearch.version><maven.compiler.target>1.8maven.compiler.target><elasticsearch.plugin.name>analysis-dynamic-synonymelasticsearch.plugin.name><elasticsearch.assembly.descriptor>${project.basedir}/src/main/assemblies/plugin.xmlelasticsearch.assembly.descriptor><elasticsearch.plugin.classname>com.bellszhu.elasticsearch.plugin.DynamicSynonymPluginelasticsearch.plugin.classname><elasticsearch.plugin.jvm>trueelasticsearch.plugin.jvm>properties><licenses><license><name>The Apache Software License, Version 2.0name><url>http://www.apache.org/licenses/LICENSE-2.0.txturl><distribution>repodistribution>license>licenses><parent><groupId>org.sonatype.ossgroupId><artifactId>oss-parentartifactId><version>9version>parent><scm><connection>scm:git:git@github.com:bells/elasticsearch-analysis-dynamic-synonym.gitconnection><developerConnection>scm:git:git@github.com:bells/elasticsearch-analysis-dynamic-synonym.gitdeveloperConnection><url>https://github.com/bells/elasticsearch-analysis-dynamic-synonymurl>scm><dependencies><dependency><groupId>org.elasticsearchgroupId><artifactId>elasticsearchartifactId><version>${elasticsearch.version}version>dependency><dependency><groupId>org.codelibs.elasticsearch.modulegroupId><artifactId>analysis-commonartifactId><version>7.10.2version>dependency><dependency><groupId>junitgroupId><artifactId>junitartifactId><version>4.13.1version><scope>testscope>dependency><dependency><groupId>org.apache.httpcomponentsgroupId><artifactId>httpclientartifactId><version>4.5.13version>dependency><dependency><groupId>mysqlgroupId><artifactId>mysql-connector-javaartifactId><version>8.0.22version>dependency><dependency><groupId>org.apache.logging.log4jgroupId><artifactId>log4j-coreartifactId><version>2.13.2version><scope>providedscope>dependency><dependency><groupId>org.apache.logging.log4jgroupId><artifactId>log4j-apiartifactId><version>2.11.1version><scope>providedscope>dependency><dependency><groupId>org.codelibsgroupId><artifactId>elasticsearch-cluster-runnerartifactId><version>7.10.2.0version><scope>testscope>dependency>dependencies><build><plugins><plugin><groupId>org.apache.maven.pluginsgroupId><artifactId>maven-compiler-pluginartifactId><version>2.3.2version><configuration><source>${maven.compiler.target}source><target>${maven.compiler.target}target>configuration>plugin><plugin><groupId>org.apache.maven.pluginsgroupId><artifactId>maven-surefire-pluginartifactId><version>2.11version><configuration><includes><include>**/*Tests.javainclude>includes>configuration>plugin><plugin><groupId>org.apache.maven.pluginsgroupId><artifactId>maven-source-pluginartifactId><version>2.1.2version><executions><execution><id>attach-sourcesid><goals><goal>jargoal>goals>execution>executions>plugin><plugin><artifactId>maven-assembly-pluginartifactId><configuration><appendAssemblyId>falseappendAssemblyId><outputDirectory>${project.build.directory}/releases/outputDirectory><descriptors><descriptor>${basedir}/src/main/assemblies/plugin.xmldescriptor>descriptors><archive><manifest><mainClass>fully.qualified.MainClassmainClass>manifest>archive>configuration><executions><execution><phase>packagephase><goals><goal>singlegoal>goals>execution>executions>plugin>plugins>build>
project>

这里在做链接MySQL数据的时候要注意一下MySQL的驱动jar包，不同版本的url会有所区别。

二、新增MySQL代码

2.1 新增MysqlRemoteSynonymFile文件

public class MySqlRemoteSynonymFile implements SynonymFile{/*** 数据库配置文件名*/private final static String DB_PROPERTIES = "jdbc-reload.properties";private static Logger logger = LogManager.getLogger("dynamic-synonym");private String format;private boolean expand;private boolean lenient;private Analyzer analyzer;private Environment env;// 数据库配置private String location;/*** 数据库地址*/private static final String JDBC_URL = "jdbc.url";/*** 数据库驱动*/private static final String JDBC_DRIVER = "jdbc.driver";/*** 数据库用户名*/private static final String JDBC_USER = "jdbc.user";/*** 数据库密码*/private static final String JDBC_PASSWORD = "jdbc.password";/*** 当前节点的同义词版本号*/private LocalDateTime thisSynonymVersion = LocalDateTime.now();private static Connection connection = null;private Statement statement = null;private Properties props;private Path conf_dir;MySqlRemoteSynonymFile(Environment env, Analyzer analyzer,boolean expand, boolean lenient, String format, String location) {this.analyzer = analyzer;this.expand = expand;this.format = format;this.lenient = lenient;this.env = env;this.location = location;this.props = new Properties();//读取当前 jar 包存放的路径Path filePath = PathUtils.get(new File(DynamicSynonymPlugin.class.getProtectionDomain().getCodeSource().getLocation().getPath()).getParent(), "config").toAbsolutePath();this.conf_dir = filePath.resolve(DB_PROPERTIES);//判断文件是否存在File configFile = conf_dir.toFile();InputStream input = null;try {input = new FileInputStream(configFile);} catch (FileNotFoundException e) {logger.info("jdbc-reload.properties 数据库配置文件没有找到， " + e);}if (input != null) {try {props.load(input);} catch (IOException e) {logger.error("数据库配置文件 jdbc-reload.properties 加载失败，" + e);}}isNeedReloadSynonymMap();}/*** 加载同义词词典至SynonymMap中* @return SynonymMap*/@Overridepublic SynonymMap reloadSynonymMap() {try {logger.info("start reload local synonym from {}.", location);Reader rulesReader = getReader();SynonymMap.Builder parser = RemoteSynonymFile.getSynonymParser(rulesReader, format, expand, lenient, analyzer);return parser.build();} catch (Exception e) {logger.error("reload local synonym {} error! cause: {}", location, e.getMessage());throw new IllegalArgumentException("could not reload local synonyms file to build synonyms", e);}}/*** 判断是否需要进行重新加载* @return true or false*/@Overridepublic boolean isNeedReloadSynonymMap() {try {LocalDateTime mysqlLastModify = getMySqlSynonymLastModify();if (!thisSynonymVersion.isEqual(mysqlLastModify)) {thisSynonymVersion = mysqlLastModify;return true;}} catch (Exception e) {logger.error(e);}return false;}/*** 获取MySql中同义词版本号信息* 用于判断同义词是否需要进行重新加载** @return getLastModify*/public LocalDateTime getMySqlSynonymLastModify() {ResultSet resultSet = null;LocalDateTime mysqlSynonymLastModify = null;try {if (statement == null) {statement = getConnection(props);}resultSet = statement.executeQuery(props.getProperty("jdbc.reload.swith.synonym.last_modify"));while (resultSet.next()) {Timestamp lastModify = resultSet.getTimestamp("last_modify");mysqlSynonymLastModify = lastModify.toLocalDateTime();// logger.info("当前MySql同义词最后修改时间为:{}, 当前节点同义词库最后修改时间为:{}", mysqlSynonymLastModify, thisSynonymVersion);}} catch (SQLException e) {e.printStackTrace();} finally {try {if (resultSet != null) {resultSet.close();}} catch (SQLException e) {e.printStackTrace();}}return mysqlSynonymLastModify;}/*** 查询数据库中的同义词* @return DBData*/public ArrayList<String> getDbData() {ArrayList<String> arrayList = new ArrayList<>();ResultSet resultSet = null;try {if (statement == null) {statement = getConnection(props);}logger.info("正在执行SQL查询同义词列表，SQL:{}", props.getProperty("jdbc.reload.synonym.sql"));resultSet = statement.executeQuery(props.getProperty("jdbc.reload.synonym.sql"));while (resultSet.next()) {String theWord = resultSet.getString("words");arrayList.add(theWord);}} catch (SQLException e) {logger.error(e);} finally {try {if (resultSet != null) {resultSet.close();}} catch (SQLException e) {e.printStackTrace();}}return arrayList;}/*** 同义词库的加载* @return Reader*/@Overridepublic Reader getReader() {StringBuilder sb = new StringBuilder();try {ArrayList<String> dbData = getDbData();for (String dbDatum : dbData) {logger.info("正在加载同义词:{}", dbDatum);// 获取一行一行的记录，每一条记录都包含多个词，形成一个词组，词与词之间使用英文逗号分割sb.append(dbDatum).append(System.getProperty("line.separator"));}} catch (Exception e) {logger.error("同义词加载失败");}return new StringReader(sb.toString());}/*** 获取数据库可执行连接* @param props 配置文件* @throws SQLException 获取连接失败*/private static Statement getConnection(Properties props) throws SQLException {try {Class.forName(props.getProperty(JDBC_DRIVER));} catch (ClassNotFoundException e) {logger.error("驱动加载失败", e);}if (connection == null) {connection = DriverManager.getConnection(props.getProperty(JDBC_URL),props.getProperty(JDBC_USER),props.getProperty(JDBC_PASSWORD));}return connection.createStatement();}
}

2.2 在getSynonymFile新增MySQL的连接方式

修改的DynamicSynonymTokenFilterFactory的资源获取代码

SynonymFile getSynonymFile(Analyzer analyzer) {try {SynonymFile synonymFile;if ("MySql".equals(location)) {synonymFile = new MySqlRemoteSynonymFile(environment, analyzer, expand, lenient, format, location);} else if (location.startsWith("http://") || location.startsWith("https://")) {synonymFile = new RemoteSynonymFile(environment, analyzer, expand, lenient,  format, location);} else {synonymFile = new LocalSynonymFile(environment, analyzer, expand, lenient, format, location);}if (scheduledFuture == null) {scheduledFuture = pool.scheduleAtFixedRate(new Monitor(synonymFile),interval, interval, TimeUnit.SECONDS);}return synonymFile;} catch (Exception e) {logger.error("failed to get synonyms: " + location, e);throw new IllegalArgumentException("failed to get synonyms : " + location, e);}
}

三、创建一个dynamic-synonym的表

3.1 建库建表

​ 作者这边的数据库名称为word，表名为synonym

/*Navicat Premium Data TransferSource Server         : localhostSource Server Type    : MySQLSource Server Version : 50717Source Host           : localhost:3306Source Schema         : authTarget Server Type    : MySQLTarget Server Version : 50717File Encoding         : 65001Date: 05/01/2022 17:01:31
*/SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;-- ----------------------------
-- Table structure for synonym
-- ----------------------------
DROP TABLE IF EXISTS `synonym`;
CREATE TABLE `synonym`  (`id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键',`words` text CHARACTER SET utf8 COLLATE utf8_bin NULL COMMENT '同义词',`last_modify` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP(0) ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '最后更新时间',PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Dynamic;-- ----------------------------
-- Records of synonym
-- ----------------------------
INSERT INTO `synonym` VALUES (1, '西红柿,番茄,洋柿子', '2022-01-05 16:48:24');SET FOREIGN_KEY_CHECKS = 1;

3.2 修改数据库连接的配置文件

在项目的src同级目录下新增config/jdbc-reload.properties文件

# permission java.net.SocketPermission "*", "connect,resolve";
# CHCP 65001jdbc.url=jdbc:mysql://192.168.255.132:3306/word?serverTimezone=GMT
jdbc.user=root
jdbc.driver=com.mysql.cj.jdbc.Driver
jdbc.password=123456
# 查询词库
jdbc.reload.synonym.sql=select words from synonym
# 查询更新时间
jdbc.reload.swith.synonym.last_modify=SELECT MAX(last_modify) last_modify FROM synonym

四、修改docker中es容器的Java.policy文件**【非常重要】**

这里作者用的是docker容器化部署，如果是直接装在windows系统或者centos系统下，就要去修改es依赖的Jdk，直接修改系统的jdk的java.policy文件。在这里不直接修改系统jdk的java.policy文件是因为docker容器化部署的es是独立于系统的jdk运行的，这个es有一套自己的输出逻辑。

4.1 找到Java.policy

首先进入到容器内部操作 docker exec -it es /bin/bash，然后直接打开 cd /usr/share/elasticsearch/jdk/conf/security/文件夹，找到Java.policy文件。

[root@localhost ~]# docker exec -it es /bin/bash
[root@ee5fd3f35131 elasticsearch]# cd /usr/share/elasticsearch/jdk/conf/security/
[root@ee5fd3f35131 security]# ls
java.policy  java.security  policy
[root@ee5fd3f35131 security]# vi java.policy

4.2 修改java.policy文件

下面文件的全部内容：

//
// This system policy file grants a set of default permissions to all domains
// and can be configured to grant additional permissions to modules and other
// code sources. The code source URL scheme for modules linked into a
// run-time image is "jrt".
//
// For example, to grant permission to read the "foo" property to the module
// "com.greetings", the grant entry is:
//
// grant codeBase "jrt:/com.greetings" {
//     permission java.util.PropertyPermission "foo", "read";
// };
//
grant codeBase "file:${{java.ext.dirs}}/*" {permission java.security.AllPermission;
};// default permissions granted to all domains
grant {// allows anyone to listen on dynamic portspermission java.net.SocketPermission "localhost:0", "listen";// "standard" properies that can be read by anyonepermission java.util.PropertyPermission "java.version", "read";permission java.util.PropertyPermission "java.vendor", "read";permission java.util.PropertyPermission "java.vendor.url", "read";permission java.util.PropertyPermission "java.class.version", "read";permission java.util.PropertyPermission "os.name", "read";permission java.util.PropertyPermission "os.version", "read";permission java.util.PropertyPermission "os.arch", "read";permission java.util.PropertyPermission "file.separator", "read";permission java.util.PropertyPermission "path.separator", "read";permission java.util.PropertyPermission "line.separator", "read";permission java.util.PropertyPermission"java.specification.version", "read";permission java.util.PropertyPermission "java.specification.vendor", "read";permission java.util.PropertyPermission "java.specification.name", "read";permission java.util.PropertyPermission"java.vm.specification.version", "read";permission java.util.PropertyPermission"java.vm.specification.vendor", "read";permission java.util.PropertyPermission"java.vm.specification.name", "read";permission java.util.PropertyPermission "java.vm.version", "read";permission java.util.PropertyPermission "java.vm.vendor", "read";permission java.util.PropertyPermission "java.vm.name", "read";permission java.net.SocketPermission "*", "connect,resolve";permission java.lang.RuntimePermission "setContextClassLoader";permission java.lang.RuntimePermission "accessDeclaredMembers";permission java.lang.RuntimePermission "createClassLoader";permission java.security.AllPermission;
};

五、将打包好的jar包放入到 {es-root}/es-plugins目录下面

5.1 在打包之前一定要注意自己es的版本号

5.2 打包完成之后解压文件并且上传到服务器中的es的plugins目录

​ 这里作者用的docker的容器部署，如果是windows本地直接找到plugins目录放进去就可以了。

六、docker重启es容器

如果直接安装在系统上，就直接去找到elasticsearch/bin目录下重启一下就可以啦。作者这里是容器部署的哈。

docker restart es

容器重启之后记得查看一下docker的控制台输出，看看有没有什么问题，如果出现权限之类的问题，那基本上就是java.policy文件没有配置正确，如果出现数据库之类的问题，请在本地建个Java项目连接一下试试，看看能不能跑的起来。

docker logs -f es

七、新建es的dynamic-synonym索引测试

PUT synonyms_index
{"settings": {"number_of_shards": 1,"number_of_replicas": 1,"analysis": {"analyzer": {"synonym": {"type":"custom","tokenizer": "ik_smart","filter": ["synonym_custom"]}},"filter": {"synonym_custom": {"type": "dynamic_synonym","synonyms_path": "MySql"}}}},"mappings": {"properties": {"name": {"type": "text","analyzer": "synonym"}}}
}

GET /synonyms_index/_analyze
{"text": "西红柿","analyzer": "synonym"
}

这样子就算运行成功啦，开心撒花！！！

delete synonyms_index

八、总结

8.1 源码地址

为了做这个项目，作者搞了大概得有一天，为了让大家节省时间，这里可以直接下载我已经配置好的源码

8.2 小节

​ 经过一天的研究，终于大致弄明白es插件的运行过程了，为后续实现自动补全功能、优化搜索、广告推荐、聚合查询做好了前提条件。

以后如果做这些功能了再将博客补上，最后，感谢大家的支持

本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！