hadoop com.sun.tools.javac.Main WordCount.java
The next part is the most difficult - you need to create the jar file. I am not a java guru, so I don't really know what should I get at the end, so I've created the jar using the following command:
jar cf wc.jar WordCount*.class
I will come back to the structure of this wc.jar later, so now Jar file is created and we need only to run it, but before that the hdfs directories should be created:
runas /user:hadoop cmd %HADOOP_HOME%\bin\hadoop fs -mkdir -p /test_project/in %HADOOP_HOME%\bin\hadoop fs -put G:\test\gena_test.txt /test_project/in
Everything is ready now, running the application:
hadoop jar G:\test\wc.jar WordCount /test_project/in /test_project/outDoesn't work!!! Getting the following error:
java.lang.RuntimeException: java.lang.ClassNotFoundException: WordCount$TokenizerMapperHm, this is unexpected. Le't google it. At the beginning, I thought that the problem is related with environment variables, therefore I've set all of them (%JAVA_HOME%, %PATH%, %HADOOP_CLASSPATH%) - doesn't help. The next, I've discovered the following warning:
WARN ... No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
Let's resolve this:
job.setJar("G:\\test\\wc.jar");
//job.setJarByClass(WordCount.class);
But, the error is still shown up. After a couple of hours of useless attempts to resolve this I've found the following topic on stackoverflow.com. The problem was resolved by adding the package into the java class. I tried this and OOOO miracle, the issue is resolved. I think the issue is related with jar file. The jar file I've created previously had two folders :
- test - contains all class files;
- Meta-INF
hadoop jar G:\test\wc.jar test.WordCount /test_project/in /test_project/out
Enjoy.
PS. I've attached the iPhone 6 picture (phone isn't in very good condition I can say :) ). It belongs to one of my associates. The weirdest thing is - it works properly :)
