Developer's Closet A place where I can put my PHP, SQL, Perl, JavaScript, and VBScript code.

9Sep/140
Recover Lost Files on Linux

TestDisk includes PhotoRec which is very good at recovering files:

sudo apt-get install testdisk

To run PhotoRec:

sudo photorec

PhotoRec: http://www.cgsecurity.org/wiki/PhotoRec

 

Filed under: Linux, Ubuntu No Comments
6Aug/140
Copy data from one Hadoop cluster to another Hadoop cluster (running different versions of Hadoop)

I had to copy data from one Hadoop cluster to another recently. However, the two clusters ran different versions of Hadoop, which made using distcp a little tricky.

Some notes of distcp: By default, distcp will skip files that already exist in the destination, but they can be overwritten by supplying the -overwrite option. You can also update only files that have changed using the -update option. distcp is implemented as a MapReduce job where the work of copying is done by maps that run in parallel across the cluster. There are no reducers. Each file is copied by a single map, and distcp tries to give each map approximately the same amount of data, by bucketing files into roughly equal allocations.

The following command will copy the folder contents from one Hadoop cluster to a folder on another Hadoop cluster. Using hftp is necessary because the clusters run a different version of Hadoop. The command must be run on the destination cluster. Be sure your user has access to write to the destination folder.

hadoop distcp -pb hftp://namenode:50070/tmp/* hdfs://namenode/tmp/

Note: The -pb option will preserve the block size.

Double Note: For copying between two different versions of Hadoop we must use the HftpFileSystem, which is a read-only files system. So the distcp must be run on the destination cluster.

The following command will copy data from Hadoop clusters that are the same version.

hadoop distcp -pb hdfs://namenode/tmp/* hdfs://namenode/tmp/

Filed under: HDFS, Ubuntu No Comments
1Jul/140
HBase All Regions in Transition: state=FAILED_OPEN

After I added a jar file to the HBase Master I had a problem where regions failed to transition to a RegionServer. Below are the errors; removing the jar file from the hbase/lib folder resolved this problem (full path to jar: /opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hbase/lib/). What tipped me off was the missing class definition: Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/ipc/CoprocessorProtocol.

Failed open of region=REGION.NAME,,4194066667839.6ea7d7ff9276f9c0e9b126c73e25bc54., starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3970)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4276)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4249)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4205)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4156)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:475)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:140)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3967)
... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/ipc/CoprocessorProtocol
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
...

9:10:19.721 AM INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler
Opening of region {ENCODED => 6ea7d7ff9276f9c0e9b126c73e25bc54, NAME => 'REGION.NAME,,4194066667839.6ea7d7ff9276f9c0e9b126c73e25bc54.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 28

30Jun/140
Cron file failed to load: (username~) ORPHAN (no passwd entry)

This problem bothers me a little. The authentication server failed during a cron job that referenced a specific account. Ubuntu could not authenticate the account, and id username failed. So the account was written to the /var/spool/cron/crontabs/ and any time I tried to edit the cron file under /etc/cron.c/username-cron-file, the reload would fail:

cron[17959]: (*system*username-cron-file) RELOAD (/etc/cron.d/username-cron-file)
cron[17959]: Error: bad username; while reading /etc/cron.d/username-cron-file
cron[17959]: (*system*username-cron-file) ERROR (Syntax error, this crontab file will be ignored)
cron[17959]: (username~) ORPHAN (no passwd entry)

I deleted the spool entry and was able to recreate the cron file.

 

Filed under: Linux, Ubuntu No Comments
23Jul/130
Test if a File Exists in Ubuntu and Delete

Use the test command to test if a file exists and delete the file if it exists. Or output a message that the file does not exist:

sudo test -e /tmp/filename && sudo rm /tmp/filename && echo "file exists and was removed" || echo "file does not exist";

More examples:

test -e file = file exists
test -x file = ...is executable
test -w file = ...is writable
test -r file = ...is readable
test -s file = ...is not empty
test -d file = ...is a directory
test -f file = ...is a regular file (not a device driver or directory)
test string1 = string2 = test whether string1 is the same as string2
test string1 != string2 = test whether string1 is not the same as string2
[ expression ] can also be used instead of test expression

man test for more information.

Filed under: Linux, Ubuntu No Comments
7Jun/130
Clean Packages and Get Back Storage in Ubuntu

When I am low on space or I just want to clean house:

Clean Packages
1. Update your sources.list
sudo apt-get update
2. Clean any partial packages left behind during an install
sudo apt-get autoclean
3. Clean your apt-cache
sudo apt-get clean
4. Remove any packages that are no longer necessary
sudo apt-get autoremove
5. Put it all together:
sudo apt-get autoclean && sudo apt-get clean && sudo apt-get autoremove
6. May show that you have broken packages or missing dependencies. Try fixing them with:
apt-get -f install

Note: You may receive the following error:
E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the download directory
If you do, you can discover what is running and kill -9 the process. However, you could just wait until the installation is complete – or reboot.
ps faux | grep apt

Filed under: Linux, Ubuntu No Comments
4Jun/130
Repair Packages in Ubuntu

If you run sudo apt-get update and receive an error similar to the following, you may have a newer version of linux-image-server and linux-headers-server than any linux-server package available from your configured software sources:

dpkg: dependency problems prevent configuration of linux-server:
linux-server depends on linux-image-server (= 3.2.0.37.44); however:
Version of linux-image-server on system is 3.2.0.37.45.
linux-server depends on linux-headers-server (= 3.2.0.37.44); however:
Version of linux-headers-server on system is 3.2.0.37.45.
dpkg: error processing linux-server (--configure):
dependency problems - leaving unconfigured

This could occur for a very short time even if you're using a central server (or for a longer time if there were a problem updating it). This happened to me when I ran out of storage on my server during an update. There are a few ways of dealing with this error:

Ignore It
If you're able to use the package manager for other purposes, for example, to install other packages, then you can just ignore this. When your mirror gets updated, you'll probably just get the package.
It's not a problem not to have the latest linux-server package, as it's a metapackage that doesn't provide any actual software. It exists only so it can list three packages as dependencies, ensuring they remain installed:

• linux-generic-pae (this is only a dependency on i386--32-bit, that is--systems)
• linux-headers-server
• linux-image-server

You already have the latest versions of linux-headers-server and linux-image-server. My guess is that this is a 64-bit system, which is why no message about linux-generic-pae was shown. It seems like everything is fine.

However:

• If the problem is preventing updating and installing other packages from working (as opposed to just showing those messages whenever you do so but not interfering with it), then you should apply some other solution.
• You may also wish to do so, if you just don't like seeing these messages all the time and want to get rid of them.
• It's a little strange you're not getting this package, even if you use a mirror that lags a bit behind the main download server, because this package is also provided by http://security.ubuntu.com. That server should always been enabled in /etc/apt/sources.list. So I recommend checking to make sure that is enabled. You may want to back up sources.list before editing it (sudo cp /etc/apt/sources.list /etc/apt/sources.list.old). The problem could also be a consequence of a caching web proxy (if you use one) with stale data (the solutions below will work around that, too).

Change to a Different Download Server
Change your mirror:

1. Open your source.list using your favorite text editor, ie
sudo nano /etc/apt/sources.list
2. locate http://us.archive.ubuntu.com/ubuntu and replace it with http://archive.ubuntu.com/ubuntu.

After changing your mirror, run:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
If you like, you can change your server back after linux-server is updated.
Manually Download and Install the .deb Package File

1. You can manually download and install the .deb file for the latest version of the linux-servermetapackage in Ubuntu 12.04.
2. cd to wherever you want to download the package.

• If this is a 64-bit system, run:
wget https://launchpad.net/ubuntu/+archive/primary/+files/linux-server_3.2.0.23.25_amd64.deb
• If this is a 32-bit system, run this instead:
wget https://launchpad.net/ubuntu/+archive/primary/+files/linux-server_3.2.0.23.25_i386.deb

Note: You can obtain these links from the Launchpad page for the linux-meta source package in Ubuntu.

3. Now (for both 64-bit and 32-bit), run this command to install the package:
sudo dpkg -i linux-server_3.2.0.23.25_*.deb

Remove the Package
Since it's a metapackage providing software you already have, you can remove the package. Then you won't get the error anymore. You can install it again later if you like (you could periodically run apt-get update && apt-cache policy linux-server to see if the new version is available).

There are two steps, if you want to remove it. The first is to uninstall linux-server. The second is to make sure the packages won't be automatically removed later.
The dependencies will not be removed immediately. So it doesn't matter which of these steps you do first.

1. Uninstall linux-server.
sudo apt-get remove linux-server

2. Mark the dependencies as manually installed. To tell apt about packages that you intend to have installed even if nothing remains that declares them a dependency.
• If this is a 64-bit system:
sudo apt-mark manual linux-image-server linux-headers-server
• If it's a 32-bit system, run this instead:
sudo apt-mark manual linux-image-server linux-headers-server linux-generic-pae

Note: You should have apt-mark but if you don't and you cannot install software, you can use apt-get install instead of apt-mark manual. Specifying a package manually for installation causes it to be marked as manually installed even if was already installed.

Filed under: Linux, Ubuntu No Comments
24May/130
Renew DHCP in Ubuntu Linux

I often have to look this up. Release a DHCP lease in Ubuntu with the following command (for the GUI search for gnome-terminal):

sudo dhclient -r

Next retrieve a new lease:

sudo dhclient

Or, on one line:

sudo dhclient -r && sudo dhclient;

If that does not work, check if networking is up:

ifdown eth0
ifup eth0

Finally, you can always restart networking:

/etc/init.d/network restart

Or:

/etc/init.d/networking restart

Don't forget to reset the certificate used by Ops Manager 2012 (after a hostname rename, look for the hostname does not match the FQDN on certificate error). The following command will clear all previous Ops Manager files (The –f option forces the files in /etc/opt/microsoft/scx/ssl to be overwritten):

/opt/microsoft/scx/bin/tools/scxsslconfig -f -v

Next, restart the agent:

/opt/microsoft/scx/bin/tools/scxadmin -restart

The Ops Manager discovery and install should now be successful.

19Apr/130
Install Oracle Java 7 on Ubuntu Linux

The blogger at http://tuxtuts.com saved me a lot of time, thanks! How to: Install Oracle Java 7 (JDK & JRE) in Ubuntu 12.10 and Linux Mint 13

Also thanks to Andrew from Webupd8 for creating the repository:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
Filed under: Linux, Ubuntu No Comments
12Apr/130
Disable IPv6 in Ubuntu Linux

Recently I was asked to disable IPv6 on an Ubuntu Linux server. Here are the steps:

  1. Test if IPv6 is enabled:

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

0 means it's enabled and 1 - disabled.

  1. To disable IPv6, add the following lines to /etc/sysctl.conf:

#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

  1. Run the following command:

echo "#disable ipv6" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf

  1. You will need to reboot the network interfaces (sysctl):

sudo sysctl -p

  1. Validate:

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

0 means it's enabled and 1 - disabled.

Filed under: Linux, Ubuntu No Comments