{"id":1180,"date":"2015-05-18T10:23:44","date_gmt":"2015-05-18T18:23:44","guid":{"rendered":"http:\/\/www.developerscloset.com\/?page_id=1180"},"modified":"2018-05-18T10:33:49","modified_gmt":"2018-05-18T18:33:49","slug":"oozie","status":"publish","type":"page","link":"https:\/\/www.developerscloset.com\/?page_id=1180","title":{"rendered":"Oozie"},"content":{"rendered":"<p><a href=\"http:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/oozie-image.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-1181 alignnone\" src=\"http:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/oozie-image-300x111.png\" alt=\"\" width=\"300\" height=\"111\" \/><\/a><\/p>\n<p>Oozie is a workflow coordination service to manage data processing jobs on your cluster. Oozie is essentially a task scheduler that allows advanced workflows.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-69ea2105ecae9\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-69ea2105ecae9\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Configure_Oozie\" >Configure Oozie<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Install_Oozie\" >Install Oozie<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Oozie_Configuration\" >Oozie Configuration<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Configure_Oozies_Whitelist\" >Configure Oozie\u2019s Whitelist<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Test_Oozie\" >Test Oozie<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Oozie_Commands\" >Oozie Commands<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Troubleshooting\" >Troubleshooting<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Oozie_Web_Console_is_Disabled_To_Enable_Oozie_Web_Console_Install_the_Ext_JS_Library\" >Oozie Web Console is Disabled: To Enable Oozie Web Console Install the Ext JS Library<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.developerscloset.com\/?page_id=1180\/#Oozie_Cannot_Create_Oozie_Workflow_Not_able_to_cache_shareLib\" >Oozie: Cannot Create Oozie Workflow: Not able to cache shareLib<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"Configure_Oozie\"><\/span>Configure Oozie<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<h2><span class=\"ez-toc-section\" id=\"Install_Oozie\"><\/span>Install Oozie<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Cloudera Manager distributes Oozie in CDH and offers the following services:<\/p>\n<ul>\n<li><strong>Oozie Server<\/strong>\u00a0\u2013 Install an Oozie Server on a host that is not used by an HBase Master or an HDFS NameNode as Oozie\u00a0may use a high amount of memory. Only one Oozie Server is required.<\/li>\n<\/ul>\n<h2 id=\"Oozie-OozieConfiguration\"><span class=\"ez-toc-section\" id=\"Oozie_Configuration\"><\/span>Oozie Configuration<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<div style=\"max-width: 100%; margin: auto; overflow: hidden;\">\n<div style=\"width: 100%; overflow: auto;\">\n<table class=\"confluenceTable\">\n<tbody>\n<tr>\n<td class=\"confluenceTd\"><strong>Configuration<\/strong><\/td>\n<td class=\"confluenceTd\"><strong>Description<\/strong><\/td>\n<td class=\"confluenceTd\"><strong>Value<\/strong><\/td>\n<td class=\"confluenceTd\" colspan=\"1\"><strong>Calculation<\/strong><\/td>\n<\/tr>\n<tr>\n<td class=\"confluenceTd\" colspan=\"1\">Oozie Server Data Directory<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">Directory where the Oozie Server places its data. Only applicable when using Derby as the database type.<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">\/space1\/oozie\/data<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">Not on root.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<h2 id=\"Oozie-ConfigureOozie\u2019sWhitelist\"><span class=\"ez-toc-section\" id=\"Configure_Oozies_Whitelist\"><\/span>Configure Oozie\u2019s Whitelist<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You will receive the following error if the whitelist has not been configured: ~\/oozie\/examples\/apps\/no-op$ oozie job -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-config job.properties -dryrun<\/p>\n<p>Error: E0901 : E0901: Namenode [servername01:8020] not allowed, not in Oozies whitelist<\/p>\n<p>In Cloudera Manager, browse to Oozie\u2019s Service:<\/p>\n<p>Oozie Server Default Group &gt; Advanced &gt;\u00a0<strong>Oozie Server Configuration Safety Valve for oozie-site.xml<\/strong><\/p>\n<pre>&lt;property&gt;\r\n   &lt;name&gt;oozie.service.HadoopAccessorService.jobTracker.whitelist&lt;\/name&gt;\r\n   &lt;value&gt;servername01:8021,servername02:8021&lt;\/value&gt;\r\n   &lt;description&gt;Whitelisted job tracker for Oozie service.&lt;\/description&gt;\r\n&lt;\/property&gt;\r\n&lt;property&gt;\r\n   &lt;name&gt;oozie.service.HadoopAccessorService.nameNode.whitelist&lt;\/name&gt;\r\n   &lt;value&gt;servername01:8020,servername02:8020&lt;\/value&gt;\r\n   &lt;description&gt;Whitelisted NameNode for Oozie service.&lt;\/description&gt;\r\n&lt;\/property&gt;<\/pre>\n<p>&nbsp;<\/p>\n<p><em>Note: Leave the &lt;value&gt; blank to accept all servers (or for YARN).<\/em><\/p>\n<p>Logs are available at: \/var\/log\/oozie on the Oozie server.<\/p>\n<h1 id=\"Oozie-TestOozie\"><span class=\"ez-toc-section\" id=\"Test_Oozie\"><\/span>Test Oozie<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<p>Load the examples and shared libraries.<\/p>\n<ol>\n<li>Create an oozie sub directory in your home folder &gt;&gt;mkdir oozie<\/li>\n<li>Copy \/usr\/lib\/oozie\/oozie-sharelib.tar.gz to your oozie directory &gt;&gt;cp \/usr\/lib\/oozie\/oozie-sharelib.tar.gz ~\/oozie<\/li>\n<li>Tar the .gz file &gt;&gt;tar xvfz oozie-sharelib.tar.gz<\/li>\n<li>From your oozie folder make a sub directory called examples &gt;&gt;mk examples<\/li>\n<li>copy \/usr\/shared\/doc\/oozie\/oozie-examples.tar.gz to your oozie folder. &gt;&gt;cp \/usr\/shared\/doc\/oozie\/oozie-examples.tar.gz ~\/oozie<\/li>\n<li>tar the .gz file &gt;&gt;tar xvfz oozie-examples.tar.gz<\/li>\n<\/ol>\n<p>So now under your home folder you should have an oozie folder and in oozie and examples folder. Now we need to copy this data to hdfs<\/p>\n<ol>\n<li>make an oozie directory in hdfs. &gt;&gt;sudo -u hdfs hadoop fs -mkdir \/user\/oozie<\/li>\n<li>make an hdfs directory. &gt;&gt;sudo -u hdfs hadoop fs -mkdir \/user\/hdfs<\/li>\n<li>copy oozie files &gt;&gt;sudo -u hdfs hadoop fs -put share \/user\/oozie<\/li>\n<li>copy example files. &gt;&gt;sudo -u hdfs hadoop fs -put examples \/user\/hdfs<\/li>\n<\/ol>\n<p>Run the examples:<\/p>\n<p>sudo -u hdfs oozie job -oozie http:\/\/localhost:11000\/oozie -config examples\/apps\/no-op\/job.properties -run<\/p>\n<p>sudo -u hdfs oozie job -oozie http:\/\/localhost:11000\/oozie -config examples\/apps\/java-main\/job.properties -run<\/p>\n<p>sudo -u hdfs oozie job -oozie http:\/\/localhost:11000\/oozie -config examples\/apps\/pig\/job.properties -run<\/p>\n<p>For each of the runs above you can check its status with the following (you will need to replace the id with the Id each run gives you):<\/p>\n<p>sudo -u hdfs oozie job -oozie http:\/\/localhost:11000\/oozie -info 0000002-130228105804751-oozie-oozi-W<\/p>\n<h1 id=\"Oozie-OozieCommands\"><span class=\"ez-toc-section\" id=\"Oozie_Commands\"><\/span>Oozie Commands<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<p>Get all running jobs:<\/p>\n<p>oozie jobs -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01<\/span>:11000\/oozie | grep RUNNING<\/p>\n<p>Or if the job is a coordinator, not a workflow:<\/p>\n<p>oozie jobs -jobtype coordinator -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span><\/p>\n<p>Get the jobids of the requested jobs then kill the job:<\/p>\n<p>oozie job -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-kill jobid<\/p>\n<p>Submit job (readies a job to be run):<\/p>\n<p>oozie job -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-config oozieProject\/workflowHdfsAndEmailActions\/job.properties -submit job: 0000001-130712212133144-oozie-oozi-W<\/p>\n<p>Run job:<\/p>\n<p>oozie job -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-start 0000001-130712212133144-oozie-oozi-W<\/p>\n<p>Check the status:<\/p>\n<p>oozie job -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-info 0000001-130712212133144-oozie-oozi-W<\/p>\n<h1 id=\"Oozie-Troubleshooting\"><span class=\"ez-toc-section\" id=\"Troubleshooting\"><\/span>Troubleshooting<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<h2 id=\"Oozie-OozieWebConsoleisDisabled:ToEnableOozieWebConsoleInstalltheExtJSLibrary\"><span class=\"ez-toc-section\" id=\"Oozie_Web_Console_is_Disabled_To_Enable_Oozie_Web_Console_Install_the_Ext_JS_Library\"><\/span>Oozie Web Console is Disabled: To Enable Oozie Web Console Install the Ext JS Library<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Download the ext-2.2 Oozie Ext JS library.<\/p>\n<p>Copy the folder here: \/var\/lib\/oozie\/<\/p>\n<p>Set permissions on the folder: sudo chmod -R 755 \/var\/lib\/oozie\/ext-2.2\/<\/p>\n<p>Then you can browse to http:\/\/oozie.servername01:11000\/oozie\/<\/p>\n<h2 id=\"Oozie-Oozie:CannotCreateOozieWorkflow:NotabletocacheshareLib\"><span class=\"ez-toc-section\" id=\"Oozie_Cannot_Create_Oozie_Workflow_Not_able_to_cache_shareLib\"><\/span>Oozie: Cannot Create Oozie Workflow: Not able to cache shareLib<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Stop Oozie Server, from the Actions menu, choose Install Oozie ShareLib.<\/p>\n<p>Run the following command to test the sharedlib:<\/p>\n<p>oozie admin -oozie\u00a0<span class=\"nolink\">http:\/\/oozie.servername01:11000\/oozie<\/span>\u00a0-shareliblist pig<\/p>\n<p>You should see the following:<\/p>\n<p>[Available ShareLib]<\/p>\n<p>pig<\/p>\n<p>hdfs:\/\/oozie.servername01:8020\/user\/oozie\/share\/lib\/lib_20140627103340\/pig\/ant-1.6.5.jar<\/p>\n<p>hdfs:\/\/oozie.servername01:8020\/user\/oozie\/share\/lib\/lib_20140627103340\/pig\/antlr-2.7.7.jar<\/p>\n<p>&#8230;<\/p>\n<p><strong>Resolution:\u00a0<\/strong>It turns out that this was a problem with Hue, see Hue\u2019s problems for more details. I left this in the troubleshooting section because this is useful information about Oozie.<\/p>\n<p><strong>Log:<\/strong>\u00a0vi \/var\/log\/oozie\/oozie-cmf-oozie5-OOZIE_SERVER-oozie.servername01.log.out<\/p>\n<p>2014-06-27 10:04:29,064 ERROR org.apache.oozie.service.ShareLibService: SERVER[oozie.servername01] USER[-] GROUP[-]\u00a0Not able to cache shareLib. Admin need to issue oozlie cli command to update sharelib.<\/p>\n<p>java.lang.NullPointerException<\/p>\n<p>at org.apache.oozie.service.ShareLibService.init(ShareLibService.java:108)<\/p>\n<p>at org.apache.oozie.service.Services.setServiceInternal(Services.java:368)<\/p>\n<p>at org.apache.oozie.service.Services.setService(Services.java:354)<\/p>\n<p>&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Oozie is a workflow coordination service to manage data processing jobs on your cluster. Oozie is essentially a task scheduler that allows advanced workflows. Configure [&#8230;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"class_list":["post-1180","page","type-page","status-publish","hentry"],"jetpack_shortlink":"https:\/\/wp.me\/P1BQ8S-j2","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1180","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1180"}],"version-history":[{"count":4,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1180\/revisions"}],"predecessor-version":[{"id":1185,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1180\/revisions\/1185"}],"wp:attachment":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1180"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}