Skip to main content



Pig step-by-step installation with integrated HCatalog

1) Download tar file "pig-0.13.0.tar.gz"

2)  Gunzip and Untar the file at /opt/ds/app/pig-0.13.0

 3) Change directory to /opt/ds/app/pig-0.13.0/conf

 4) Create log4j.properties from template file

5) Update pig.properties for HCatalog. For example:

 hcat.bin=/opt/ds/app/hive-0.13.0/hcatalog/bin/hcat

 6) Edit .bashrc

export PIG_HOME=/opt/ds/app/pig-0.13.0
export PATH=$PATH:$PIG_HOME/bin
export HCAT_HOME=/opt/ds/app/hive-0.13.0/hcatalog
export PATH=$PATH:$HCAT_HOME/bin

 7) It is assumed that you have already set  HADOOP_HOME, JAVA_HOME, HADOOP_COMMON_LIB_NATIVE_DIR, HADOOP_OPTS, YARN_OPTS
8) Optionally, you can create .pigbootup in User home directory

9) Execute command from user home directory

 > source .bashrc

10) Execute

>pig -useHCatalog
11) Say you had created a table in Hive with name "hivetesting". Now, try to load with below command to verify installation.

grunt> A  = LOAD 'hivetesting' USING org.apache.hcatalog.pig.HCatLoader();
grunt> describe A;

==========

 Some times you may get problem after installing Pig like below:-

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
 at org.apache.hcatalog.common.HCatUtil.checkJobContextIfRunningFromBackend(HCatUtil.java:88)
 at org.apache.hcatalog.pig.HCatLoader.setLocation(HCatLoader.java:162)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:540)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:322)
 at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:199)
 at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
 at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
 at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
 at org.apache.pig.PigServer.execute(PigServer.java:1341)




Many blogs suggest you to recompile the Pig by executing command:


    

  • ant clean jar-all -Dhadoopversion=23
    • 









or recompile piggybank.jar by executing below steps






    

  • cd contrib/piggybank/java
    • 

  • ant clean
    • 

  • ant -Dhadoopversion=23
    • 





But this may not solve your problem big time. The actual cause here is related to HCatalog. Try updating it!!. In my case, I was using Hive0.13 and Pig.0.13. And I was using HCatalog provided with Hive0.13.






Then I updated Pig to 0.15 and used separate hive-hcatalog-0.13.0.2.1.1.0-385 library jars. And problem was resolved....






Because later I identified it was not Pig who was creating problem rather it was Hive-HCatalog libraries. Hope this may help.   












































Comments











Post a Comment




















Popular posts



















Spring MongoDB Rename field with derived Value of another field




























Input Collection -  [  {     'k' : 'Troubleshooting' ,     'hour' : '2024-10-10T16' ,     'v' : [ 'WebPage, Login'  ]   },   {     'k' : 'TroubleshootingMe' ,     'hour' : '2024-10-07T01' ,     'v' : [ 'Accounts, etc'  ]   }  ] Expected Output -  [  {     'hour' : '2024-10-10T16' ,     'Troubleshooting' : [ 'WebPage, Login'  ]   },   {     'hour' : '2024-10-07T01' ,     'TroubleshootingMe' : [ 'Accounts, etc'  ]   }  ]   Above Can be achieved by  $replaceRoot / $replaceWith as follows - {     $replaceWith : {       $mergeObjects : [         {           hour : "$hour"          },         {           "$arrayToObject" : [             [               {                 k : "$k" ,                 v : "$v"                }             ]           ]         }       ]     }   } or {     $replaceRoo...




















Post a Comment









Read more



































What is Leadership 




























 




















Post a Comment









Read more

































Spark Error - missing part 0 of the schema, 2 parts are expected




























 Exception -  Caused by: org.apache.spark.sql.AnalysisException : Could not read schema from the hive metastore because it is corrupted. (missing part 0 of the schema, 2 parts are expected).; Analysis -  ·          Check for table definition. In TBLProperties, you might find something like this – > spark.sql.sources.schema.numPartCols > 'spark.sql.sources.schema.numParts' 'spark.sql.sources.schema.part.0' > 'spark.sql.sources.schema.part.1' 'spark.sql.sources.schema.part.2' > 'spark.sql.sources.schema.partCol.0'           > 'spark.sql.sources.schema.partCol.1' That’s what error seems to say that part1 is defined but part0 is missing.  Solution -  Drop & re-create table. If Table was partitioned  then all partitions  would have been removed. So do either of below -  ·          Msck repair table <db_name>.<table_name>   ·    ...




















Post a Comment









Read more

































Spark MongoDB Connector Not leading to correct count or data while reading




























  We are using Scala 2.11 , Spark 2.4 and Spark MongoDB Connector 2.4.4 Use Case 1 - We wanted to read a Shareded Mongo Collection and copy its data to another Mongo Collection. We noticed that after Spark Job successful completion. Output MongoDB did not had many records. Use Case 2 -  We read a MongoDB collection and doing count on dataframe lead to different count on each execution. Analysis,  We realized that MongoDB Spark Connector is missing data on bulk read as a dataframe. We tried various partitioner, listed on page -  https://www.mongodb.com/docs/spark-connector/v2.4/configuration/  But, none of them worked for us. Finally, we tried  MongoShardedPartitioner  this lead to constant count on each execution. But, it was greater than the actual count of records on the collection. This seems to be limitation with MongoDB Spark Connector. But,  MongoShardedPartitioner  seemed closest possible solution to this kind of situation. But, it per...




















Post a Comment









Read more

































Read from a hive table and write back to it using spark sql




























 In context to Spark 2.2 -   if we read from an hive table and write to same, we get following exception-   scala >  dy . write . mode ( "overwrite" ). insertInto ( "incremental.test2" )               org . apache . spark . sql . AnalysisException :  Cannot  insert overwrite into table that is also being read from .;  org . apache . spark . sql . AnalysisException :  Cannot  insert overwrite into table that is also being read from .;     1. This error means that our process is reading from same table and writing to same table.  2. Normally, this should work as process writes to directory .hiveStaging...   3. This error occurs in case of saveAsTable method, as it overwrites entire table instead of individual partitions.  4. This error should not occur with insertInto method, as it overwrites partitions not the table.  5. A reason why this happening is because Hive table has following Spark TBLProperties in its definition. This problem will solve for insertInto met...




















Post a Comment









Read more