Ok firstly, I know my code is an absolute mess, sorry bout that, but I'm just trying to get this thing done so I don't feel like I've wasted time.
So, I'm taking an online course at Udacity to build a basic search engine, the class is in Python but I'm doing it in Java for the extra challenge.
The problem is, I'm trying to simultaneously add links to a list, and loop through it to get the next links until all the links are extracted, as you can see in the crawl_web() and get_all_links() methods.
I've looked through the web and found something about Synchronization, but I probably did it wrong and it's not working.
So, can this be done ? Is there a better way to do it ?
The ConcurrentModificationException you are getting is not related to threaded concurrency (despite the name), so synchronization will not help you. In this case, the problem comes from the fact that you are trying to manipulate the list directly while iterating over it using an iterator (the enhanced for loop), which you can't do. Instead, you should look at the API for java.util.List (<- link) which provides a solution.
Joined: Feb 09, 2013
Is the answer "CopyOnWriteArrayList" ? Let me try that one.
No, the name isn't the problem, Stephen is saying the design is.
Regardless of that, your problem (let me repeat this) is not concurrency, threading, or anything like that. Please stay away from threading and concurrency at the moment because you clearly don't understand it.
The fix I mentioned in the API is to use a ListIterator: which allows you to modify a list as you iterate over it. But you can only access, add, or remove to/from the list through that ListIterator - not through the List itself. That means a hefty redesign (which you need to do anyway).
You can do that, or you can index manually (using an integer counter), or you could us a Queue (where you use a while((link = queue.poll())!=null) loop to get the links).