Logstash connect to Kerberos authenticated Hive Service

 

Normally, one can write syntax like below to create a JDBC connection with Hive - 

input { jdbc { jdbc_driver_library => "hive-jdbc-2.0.0.jar,hive2.jar,hive-common-2.3.1.jar,hadoop-core-1.2.1-0.jar" jdbc_driver_class => "org.apache.hive.jdbc.HiveDriver" jdbc_connection_string => "" } } output { # Publish out in command line stdout { codec => json } }

But, you will get problem if you need to do Kerberos authentication for using Hive JDBC. Relating to this, set following JVM Options. Note that these can be set with either within config/jvm.options file or setting the LS_JAVA_OPTS variable will additive override JVM settings. Refer - https://www.elastic.co/guide/en/logstash/current/jvm-settings.html

• -Djava.security.auth.login.config=<Jass_config_file_path> (Required)
• -Djava.security.krb5.conf=<Path to krb5.conf> (if it is not in default location under /etc/)

if KRB5.conf is not specified then you can manually specify KDC and Realm using below options - 

• -Djava.security.krb5.kdc=<KDC_Server>
• -Djava.security.krb5.realm=<Kerberos REALM>

Sample Jaas.conf file can look something like below - 

Client {

  com.sun.security.auth.module.Krb5LoginModule required

  useKeyTab=true

  doNotPrompt=true

  useTicketCache=false

  serviceName="hive"

  keyTab="/path/to/awsemraccount.keytab"

  principal="awsemraccount@EMR.LOCAL"

  storeKey=true

  client=true;

};

Sample Hive JDBC URL would look like - 

jdbc:hive2://myhiveserver:10000/default;principal=hive/_HOST@REALM;;AuthMech=1;KrbRealm=REALM;KrbHostFQDN=hostfqdn;KrbServiceName=hive