WebSpider

This assignment is the first ICS class assignment that I could not finish. It is so difficult that I could only finished task 0, and I only partially finished the task 1 such that it could find out the number of links found in the crawling, and this was heavily inspired by Jared Sunouchi's code. The link to my code can be found in here. There are several problems that prevented me from accomplishing the tasks.

The first problem was the JUnit test for the main method. Since there is no test that works for type "void", I had to construct some dummy tests to just satisfy the JUnit and Emma. Although I got 100% coverage, I obviously do not consider them as effective testing. I think I still need to learn more about JUnit to write high quality testing.

The second problem was invoking the program using command line arguments. This really threw me off because the hyphen of some arguments, such as the one for "totallinks". I thought it would be a parameter to invoke the program to perform a specific task, which I totally did not expect it to be an arguments for the main method.

The last problem, which was also the most difficult problem for me to finish this assignment, was the implementation of a web crawler using httpunit. I never write a web crawler program before, thus it takes me a long time to try to figure out a recursive method to find the total links. Disappointedly, I still ended up using an iterative method. The usage of httpunit added more harshness to this task. The relationship among the classes, such as the WebConversation, WebResponse, and WebRequest is still confusing for me.

Since I am not able to completely finish the task 1, my attempt for task 2 was not successful. I could have done task 3 using System.out.println, but I did not do it since it is not efficient and not elegant.

The most important lesson I learn from this assignment was not to procrastinate. It would be much better that if I could start this assignment early, and it also would be more helpful that I could post question early on the discussion page. I could not even get to this far without the help from my classmates.