The oldWALs folder in HBase has been growing for the past few days – the folder is 1 TB and growing – the oldWALs are not being deleted. Looking into the Master logs, I discover the following exception:<\/p>\n
Error:<\/strong><\/p>\n May 25, 9:13:17.118 AM ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper Resolution:<\/strong><\/p>\n Restart the HBase Master service should correct this problem. If you are using HA (more than one HBase Master) you will not notice an impact in service. If not, only DDL changes will be impacted, so the risk is low. After a restart the\u00a0ReplicationLogCleaner service should be able to connect to ZooKeeper and continue. The oldWALs should clear momentarily.<\/p>\n","protected":false},"excerpt":{"rendered":" The oldWALs folder in HBase has been growing for the past few days – the folder is 1 TB and growing – the oldWALs are […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-1379","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1BQ8S-mf","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1379","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1379"}],"version-history":[{"count":1,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1379\/revisions"}],"predecessor-version":[{"id":1380,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1379\/revisions\/1380"}],"wp:attachment":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1379"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1379"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1379"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nZooKeeper getData failed after 4 attempts
\nMay 25, 9:13:17.118 AM WARN org.apache.hadoop.hbase.zookeeper.ZKUtil
\nreplicationLogCleaner-0x65de7ccd821943e, quorum=zk.server01:2181,zk.server02:2181,zk.server03:2181, baseZNode=\/hbase Unable to get data of znode \/hbase\/replication\/rs
\norg.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for \/hbase\/replication\/rs
\nat org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
\nat org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
\nat org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
\n…
\nMay 25, 9:13:17.118 AM ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher
\nreplicationLogCleaner-0x65de7ccd821943e, quorum=zk.server01:2181,zk.server02:2181,zk.server03:2181, baseZNode=\/hbase Received unexpected KeeperException, re-throwing exception
\norg.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for \/hbase\/replication\/rs
\nat org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
\nat org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
\nat org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
\n…
\nMay 25, 9:13:17.119 AM WARN org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
\nReplicationLogCleaner received abort, ignoring. Reason: Failed to get stat of replication rs node
\nMay 25, 9:13:17.119 AM WARN org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
\nFailed to read zookeeper, skipping checking deletable files<\/p>\n