Hive 配置高可用 hiveserver2

2019-11-28

分布式环境

CentOS7 安装 Hive 2.3.4 准备分布式环境。

host CPU memory ip service
node01 1c 2G 10.4.96.4 hiveserver2
node02 1c 2G 10.4.96.5 hiveserver2
node03 1c 1G 10.4.96.6
node04 1c 1G 10.4.96.7
node05 1c 1G 10.4.96.8
node06 1c 1G 10.4.96.9 zookeeper
node07 1c 1G 10.4.96.10 zookeeper
node08 1c 1G 10.4.96.11 zookeeper
node09 1c 1G 10.4.96.12 MySQL

1. 配置 HDFS

配置 hdfs,达到从 hiveserver2 以任何用户登录均在 hdfs 中代理成 god 用户(分布式文件系统中工作用户)

core-site.xml 中添加 hadoop.proxyuser.god.groupshadoop.proxyuser.god.hosts 配置

[god@node01 ~]$ vim /opt/bigdata/hadoop-2.6.5/etc/hadoop/core-site.xml
...
<configuration>
    <!--指定namenode所属集群-->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
    </property>
    <!--指定zookeeper服务的集群地址-->
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>node06:2181,node07:2181,node08:2181</value>
    </property>
    <property>
        <name>hadoop.proxyuser.god.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.god.hosts</name>
        <value>*</value>
    </property>
</configuration>

分发配置文件

[god@node01 ~]$ for i in `seq 2 8`;do scp /opt/bigdata/hadoop-2.6.5/etc/hadoop/core-site.xml god@node0$i:/opt/bigdata/hadoop-2.6.5/etc/hadoop/core-site.xml;done

在两个 NameNode 节点刷新配置

[god@node01 ~]$ hdfs dfsadmin -fs hdfs://node01:8020 -refreshSuperUserGroupsConfiguration
[god@node01 ~]$ hdfs dfsadmin -fs hdfs://node02:8020 -refreshSuperUserGroupsConfiguration

对 hdfs 中 /tmp 目录赋予 777 权限

[god@node01 ~]$ hdfs dfs -chmod 777 /tmp

2. 配置 hive

node01node02 上配置 hive-site.xml 文件,其中只有 hive.server2.thrift.bind.host 配置不同,为各自主机名

$ vim /opt/bigdata/hive-2.3.4/conf/hive-site.xml
...
<configuration>
    <!--指定hive仓库目录-->
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/user/hive_remote/warehouse</value>
    </property>
    <!--指定元数据库MySQL信息,注意XML文件中 & 符号的转义问题-->
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://node09:3306/hive_remote?createDatabaseIfNotExist=true&amp;verifyServerCertificate=false&amp;useSSL=false</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>Az123456_</value>
    </property>
    <!--启用hiveserver2权限管理-->
    <property>
        <name>hive.security.authorization.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.server2.enable.doAs</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.users.in.admin.role</name>
        <value>god</value>
    </property>
    <property>
        <name>hive.security.authorization.manager</name>
        <value>org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory</value>
    </property>
    <property>
        <name>hive.security.authenticator.manager</name>
        <value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
    </property>
    <!--配置hiveserver2高可用-->
    <property>
        <name>hive.server2.support.dynamic.service.discovery</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.server2.zookeeper.namespace</name>
        <value>hiveserver2_zk</value>
    </property>
    <property>
        <name>hive.zookeeper.quorum</name>
        <value>node06:2181,node07:2181,node08:2181</value>
    </property>
    <property>
        <name>hive.zookeeper.client.port</name>
        <value>2181</value>
    </property>
    <property>
        <name>hive.server2.thrift.bind.host</name>
        <value>node01/node02</value>
    </property>
    <property>
        <name>hive.server2.thrift.port</name>
        <value>10001</value>
    </property>
</configuration>

3. 启动 hiveserver2

[god@node01 ~]$ screen -S hiveserver2
[god@node01 ~]$ hive --service hiveserver2
[god@node02 ~]$ screen -S hiveserver2
[god@node02 ~]$ hive --service hiveserver2

两者皆为阻塞界面,按 Ctrl + A + D 非中断退出

zkCli 界面:

[zk: localhost:2181(CONNECTED) 82] ls /hiveserver2_zk
[serverUri=node02:10001;version=2.3.4;sequence=0000000005]

连接测试

beeline> !connect jdbc:hive2://node06,node07,node08/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk god 123456
Connecting to jdbc:hive2://node06,node07,node08/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk
19/11/27 22:18:18 [main]: INFO jdbc.HiveConnection: Connected to node02:10001
Connected to: Apache Hive (version 2.3.4)
Driver: Hive JDBC (version 2.3.4)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://node06,node07,node08/test> show tables;
+-----------+
| tab_name  |
+-----------+
| logtbl    |
| psn2      |
| psn21     |
| psn22     |
| psn5      |
| psn6      |
| psn7      |
+-----------+
7 rows selected (2.965 seconds)
0: jdbc:hive2://node06,node07,node08/test>
public class HiveJdbcClient2 {

	private static String driverName = "org.apache.hive.jdbc.HiveDriver";

	public static void main(String[] args) throws SQLException {
		try {
			Class.forName(driverName);
		} catch (ClassNotFoundException e) {
			e.printStackTrace();
		}

		Connection conn = DriverManager.getConnection("jdbc:hive2://node01,node02,node03/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk", "god", "");
		Statement stmt = conn.createStatement();
		String sql = "select * from tbl";
		ResultSet res = stmt.executeQuery(sql);
		while (res.next()) {
			System.out.println(res.getString(1));
		}
	}
}

标题:Hive 配置高可用 hiveserver2
作者:散宜生
地址:https://17kblog.com/articles/2019/11/28/1574954229733.html