I woke up to an alert from our Cloudera Manager managed Hadoop cluster:<\/p>\n
Error:\u00a0<\/strong><\/p>\n Solution:<\/strong>\u00a0The JournalNode was missing the cluster configuration file (VERSION). Create the VERSION file in the Storage Directory and add the contents of the VERSION file from another JournalNode (reminds me to schedule config backups):<\/p>\n I woke up to an alert from our Cloudera Manager managed Hadoop cluster: Error:\u00a0 Aug 15, 2:46:01.285 PM\u00a0 WARN\u00a0\u00a0\u00a0 org.apache.hadoop.security.UserGroupInformation PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[76],"tags":[],"class_list":["post-1144","post","type-post","status-publish","format-standard","hentry","category-hdfs"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1BQ8S-is","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1144","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1144"}],"version-history":[{"count":1,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1144\/revisions"}],"predecessor-version":[{"id":1145,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=\/wp\/v2\/posts\/1144\/revisions\/1145"}],"wp:attachment":[{"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1144"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1144"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.developerscloset.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1144"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Aug 15, 2:46:01.285 PM\u00a0 WARN\u00a0\u00a0\u00a0 org.apache.hadoop.security.UserGroupInformation\r\nPriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal Storage Directory\u00a0\/space1\/dfs\/jn\/nameservice1\u00a0not formatted\r\nAug 15, 2:46:01.286 PM\u00a0 INFO\u00a0\u00a0\u00a0 org.apache.hadoop.ipc.Server\u00a0\u00a0\u00a0\r\nIPC Server handler 3 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol.getEditLogManifest from 10.170.176.10:37369 Call#83 Retry#0\r\norg.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal Storage Directory\u00a0\/space1\/dfs\/jn\/nameservice1\u00a0not formatted\r\n\u00a0\u00a0\u00a0\u00a0at org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472)\r\n\u00a0\u00a0\u00a0\u00a0at org.apache.hadoop.hdfs.qjournal.server.Journal.getEditLogManifest(Journal.java:655)\r\n\u00a0\u00a0\u00a0\u00a0at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getEditLogManifest(JournalNodeRpcServer.java:186)\r\n...<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
<\/div>\r\n<div># create folder<\/div>\r\n<div>sudo\u00a0mkdir\u00a0-p\u00a0\/space1\/dfs\/jn\/nameservice1\/current\/\r\n# create the file and add permissions\r\nsudo\u00a0vi\u00a0\/space1\/dfs\/jn\/nameservice1\/current\/VERSION\r\nsudo\u00a0chown\u00a0-R hdfs:hdfs\u00a0\/space1\/dfs\/jn\/nameservice1\/<\/div>\r\n<div><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"