Trouble reading large file

[Johannes Thorén]

I'm trying to put about half a million strings into a linkedlist.
Im reading using bufferedReader, but it just keeps loading and nothing happens, or at best i get a java heap memory error.

Here's the code i use:

public void readFromFile(String nm) throws IOException{
	    BufferedReader 		dis 	=	null;
	    words						=	new	LinkedList<String>();
	    name						=	nm+".txt";
	
	      dis = new BufferedReader(new FileReader(name));
	      while(dis.readLine()!=null) {
	    	words.add(dis.readLine()); 
	      }
	      
	      dis.close();
   
	  }

Any idea how i could solve the problem? I need every line to be read since i have to choose a random word for the game.

[Jeff Verdegan]

No, you don't actually need to read the whole thing into memory at once. Depending on your specific requirements, there are different approaches you could take. However, if you want to start with the simple approach of just reading it all in, then you have two options:

1) Increase the JVM's max available heap at startup time with the -Xmx option. For example, java -Xmx 512m -cp whatever Myclass

2) Start with a smaller file--much, much, much smaller--until you've verified that the logic of your program is correct. This is a good idea regardless of what approach you finally decide on, as it will make testing much simpler.

[Johannes Thorén]

Jeff Verdegan wrote:No, you don't actually need to read the whole thing into memory at once. Depending on your specific requirements, there are different approaches you could take. However, if you want to start with the simple approach of just reading it all in, then you have two options:

1) Increase the JVM's max available heap at startup time with the -Xmx option. For example, java -Xmx 512m -cp whatever Myclass

2) Start with a smaller file--much, much, much smaller--until you've verified that the logic of your program is correct. This is a good idea regardless of what approach you finally decide on, as it will make testing much simpler.

The program works perfect if i use a list with lets say 20 words. It's a hangman game, but the wordlist have like 400k + rows. A part of my task is to encrypt and decrypt all words in the file (seperate program).

But then in the game i need to read in all words, choose a random word for the game, decrypt it and present it to the player.

[Paul Clapham]

Johannes Thorén wrote:The program works perfect if i use a list with lets say 20 words.

I'm surprised. It looks to me like you're only putting half of the words into the list. But I suppose that testing wouldn't really expose that sort of error, since selecting random items from half of the words is almost indistinguishable from selecting random items from all of the words.

[Johannes Thorén]

Paul Clapham wrote:

Johannes Thorén wrote:The program works perfect if i use a list with lets say 20 words.

I'm surprised. It looks to me like you're only putting half of the words into the list. But I suppose that testing wouldn't really expose that sort of error, since selecting random items from half of the words is almost indistinguishable from selecting random items from all of the words.

Yeah i think its more of an iterationproblem. Changed the code to this:

public void readFromFile(String nm) throws IOException{
	    BufferedReader 		dis 	=	null;
	    words						=	new	LinkedList<String>();
	    name						=	nm+".txt";
	    
	    InputStream	is	=	new BufferedInputStream(new FileInputStream(name));
	    byte[] c		=	new byte[1024];
	    int readChars	=	0;
	    while((readChars	=	is.read(c))!= -1){
	    	for(int i=0; i<readChars; i++){
	    		if(c[i]=='\n'){
	    			numOfRows++;
	    		}
	    	}
	    }
	    is.close();
	      
	    dis = new BufferedReader(new FileReader(name));
	    
	    Random	generator	=	new Random();
	    int randomWord		=	generator.nextInt(numOfRows);
	    
	    int i=0;
	    boolean	wordRetrieved	=	false;
	    while(!wordRetrieved){
	    	i++;
	    	if(i==randomWord){
	    		words.add(dis.readLine());
	    		wordRetrieved	=	true;
	    	}else{
	    		dis.readLine();
	    	}
	    }
	      
	      dis.close();
   
	  }

Since i only need one word for the game, i thought of reading just one word from the file. (altho i'm going to have to deal with the problem of encrypting every word on the file later). But it still just keeps loading and loading.>