{"id":1160,"date":"2016-05-14T15:14:05","date_gmt":"2016-05-14T23:14:05","guid":{"rendered":"http:\/\/www.developerscloset.com\/?page_id=1160"},"modified":"2018-05-14T15:17:21","modified_gmt":"2018-05-14T23:17:21","slug":"sqoop","status":"publish","type":"page","link":"https:\/\/www.developerscloset.com\/?page_id=1160","title":{"rendered":"Sqoop"},"content":{"rendered":"<p><a href=\"http:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/sqoop2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1161 alignnone\" src=\"http:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/sqoop2.png\" alt=\"\" width=\"204\" height=\"62\" srcset=\"https:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/sqoop2.png 151w, https:\/\/www.developerscloset.com\/wp-content\/uploads\/2018\/05\/sqoop2-150x46.png 150w\" sizes=\"auto, (max-width: 204px) 100vw, 204px\" \/><\/a><\/p>\n<p>Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. The version supported by Cloudera Manager is Sqoop 2.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_79 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-69ea2105cdece\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-69ea2105cdece\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Configure_Sqoop\" >Configure Sqoop<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Install_Sqoop\" >Install Sqoop<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Configure_Sqoop-2\" >Configure Sqoop<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Troubleshooting\" >Troubleshooting<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Sqoop_Server_Startup_Failure_Upgrade_required_but_not_allowed\" >Sqoop Server Startup Failure: Upgrade required but not allowed<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.developerscloset.com\/?page_id=1160\/#Sqoop_does_not_start_on_the_Hadoop_cluster_after_a_Sqoop_service_restart\" >Sqoop does not start on the \u00a0Hadoop cluster after a Sqoop service restart<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"Configure_Sqoop\"><\/span>Configure Sqoop<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<h2><span class=\"ez-toc-section\" id=\"Install_Sqoop\"><\/span>Install Sqoop<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Sqoop requires a\u00a0<strong>Sqoop 2 Server<\/strong>, we often collocate the Sqoop 2 Server with\u00a0<u>Oozie<\/u>\u00a0or if necessary on the node running the\u00a0<u>HDFS NameNode<\/u>. Try to keep the Sqoop service off nodes running YARN NodeManagers and HBase RegionServers as they will use too much memory.<\/p>\n<ol>\n<li>From\u00a0Cloudera Manager, click Add a New Service.<\/li>\n<li>Select the Sqoop 2 service and add the service to a node, preferably a node running the HDFS NameNode or Oozie. Keep the Sqoop service off nodes running YARN NodeManagers and HBase RegionServers.<\/li>\n<\/ol>\n<h2><span class=\"ez-toc-section\" id=\"Configure_Sqoop-2\"><\/span>Configure Sqoop<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<div style=\"max-width: 100%;margin: auto;overflow: hidden\">\n<div style=\"width: 100%;overflow: auto\">\n<table class=\"confluenceTable tablesorter tablesorter-default stickyTableHeaders\" role=\"grid\">\n<colgroup>\n<col \/>\n<col \/>\n<col \/>\n<col \/><\/colgroup>\n<thead class=\"tableFloatingHeaderOriginal\">\n<tr class=\"tablesorter-headerRow\" role=\"row\">\n<th class=\"confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted\" role=\"columnheader\" scope=\"col\">\n<div class=\"tablesorter-header-inner\">Configuration<\/div>\n<\/th>\n<th class=\"confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted\" role=\"columnheader\" scope=\"col\">\n<div class=\"tablesorter-header-inner\">Description<\/div>\n<\/th>\n<th class=\"confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted\" role=\"columnheader\" scope=\"col\">\n<div class=\"tablesorter-header-inner\">Value<\/div>\n<\/th>\n<th class=\"confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted\" role=\"columnheader\" scope=\"col\">\n<div class=\"tablesorter-header-inner\">Calculation<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr role=\"row\">\n<td class=\"confluenceTd\">Sqoop 2 Server Metastore Directory<\/td>\n<td class=\"confluenceTd\">Directory where the Sqoop 2 Server places its metastore data. This is used only when Sqoop Repository Database Type is Derby.<\/td>\n<td class=\"confluenceTd\">\/space1\/sqoop2<\/td>\n<td class=\"confluenceTd\">Do not allow the directory to be on \/root &#8211; unexpected file growth.<\/td>\n<\/tr>\n<tr role=\"row\">\n<td class=\"confluenceTd\" colspan=\"1\">Java Heap Size of Sqoop 2 Server in Bytes<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">1 GB<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">A small amount of heap is required, use the largest file size to be ingested as an indication of heap size.<\/td>\n<\/tr>\n<tr role=\"row\">\n<td class=\"confluenceTd\" colspan=\"1\">Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">A\u00a0string to be inserted into sqoop.properties for this role only.<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">org.apache.sqoop.connector.autoupgrade=true<\/td>\n<td class=\"confluenceTd\" colspan=\"1\">Set Sqoop to Allow an Auto Upgrade.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<h1 id=\"Sqoop-Troubleshooting\"><span class=\"ez-toc-section\" id=\"Troubleshooting\"><\/span>Troubleshooting<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<h2 id=\"Sqoop-SqoopServerStartupFailure:Upgraderequiredbutnotallowed\"><span class=\"ez-toc-section\" id=\"Sqoop_Server_Startup_Failure_Upgrade_required_but_not_allowed\"><\/span>Sqoop Server Startup Failure: Upgrade required but not allowed<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><strong>Problem:<\/strong>\u00a0After an upgrade from CDH 5.0.2 to CDH 5.0.3, Sqoop failed to start with the following error: Server startup failure, Connector registration failed, Upgrade required but not allowed &#8211; Connector: generic-jdbc-connector.<\/p>\n<p><strong>Resolution:<\/strong>\u00a0Add the following property to the Sqoop 2 Server Advanced Configuration Snippet (Safety Valve) for sqoop.properties, under Cloudera Manager, Sqoop Service, Configuration, Sqoop 2 Server Default Group, Advanced:<\/p>\n<p>org.apache.sqoop.connector.autoupgrade=true<\/p>\n<p>After the upgrade has completed successfully, the property can be removed.<\/p>\n<p><strong>Log File:<\/strong>\u00a0\/var\/log\/sqoop2\/sqoop-cmf-sqoop-SQOOP_SERVER-servername01.log.out<\/p>\n<p>Server startup failure<\/p>\n<p>org.apache.sqoop.common.SqoopException: CONN_0007:Connector registration failed<\/p>\n<p>at org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:236)<\/p>\n<p>at org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:197)<\/p>\n<p>at org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:145)<\/p>\n<p>&#8230;<\/p>\n<p>Caused by: org.apache.sqoop.common.SqoopException: JDBCREPO_0026:Upgrade required but not allowed &#8211; Connector: generic-jdbc-connector<\/p>\n<p>at org.apache.sqoop.repository.JdbcRepository$3.doIt(JdbcRepository.java:190)<\/p>\n<p>at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:90)<\/p>\n<p>at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:61)<\/p>\n<p>&#8230;<\/p>\n<h2 id=\"Sqoop-SqoopdoesnotstartontheHadoopclusterafteraSqoopservicerestart\"><span class=\"ez-toc-section\" id=\"Sqoop_does_not_start_on_the_Hadoop_cluster_after_a_Sqoop_service_restart\"><\/span>Sqoop does not start on the \u00a0Hadoop cluster after a Sqoop service restart<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><strong>Resolution:\u00a0<\/strong>Recreating Sqoop Ddatabase after that Sqoop2 start<\/p>\n<p><strong>Log File<\/strong>:<\/p>\n<p>Can&#8217;t fetch repository structure version.<br \/>\norg.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed<br \/>\nat org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:113)<br \/>\nat org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)<br \/>\nat org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)<br \/>\n&#8230;<br \/>\nCaused by: java.sql.SQLSyntaxErrorException: Schema &#8216;SQOOP&#8217; does not exist<br \/>\nat org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)<br \/>\nat org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)<br \/>\nat org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)<br \/>\n&#8230;<br \/>\nCaused by: java.sql.SQLException: Schema &#8216;SQOOP&#8217; does not exist<br \/>\nat org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)<br \/>\nat org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)<br \/>\n&#8230;<br \/>\nCaused by: ERROR 42Y07: Schema &#8216;SQOOP&#8217; does not exist<br \/>\nat org.apache.derby.iapi.error.StandardException.newException(Unknown Source)<br \/>\nat org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(Unknown Source)<br \/>\nat org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(Unknown Source)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. The version supported by Cloudera [&#8230;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"class_list":["post-1160","page","type-page","status-publish","hentry"],"jetpack_shortlink":"https:\/\/wp.me\/P1BQ8S-iI","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1160","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1160"}],"version-history":[{"count":1,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1160\/revisions"}],"predecessor-version":[{"id":1162,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/pages\/1160\/revisions\/1162"}],"wp:attachment":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1160"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}