aspose file tools*
The moose likes I/O and Streams and the fly likes Reg:Reading file with 100,0000 lines Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Reg:Reading file with 100,0000 lines" Watch "Reg:Reading file with 100,0000 lines" New topic
Author

Reg:Reading file with 100,0000 lines

adeeb alexander
Ranch Hand

Joined: May 29, 2008
Posts: 268
Hi,
I am able to write 100,0000 lines to a file successfully but when i try to read the file using BufferedReader i get the exception. The code to read the file is given below

and the error i got is

Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 2970
    
    9
Try opening the file in a text editor. Are the 100000 lines actually on separate lines? Because if they all appear on one very long line, readLine() will try to read the whole file into memory at once. Which may be what's happening to you here.
adeeb alexander
Ranch Hand

Joined: May 29, 2008
Posts: 268
Thanks a lot Mike for your reply. What you said was correct, while entering i dint use the newline. But now its fine. I have one more thing to ask you. Can you please say something about file encryption. Like i have said that the file with 100,0000 lines is a text file i want to encrypt that file to hide data and i use a particular key then only i should be able to view the data. If this is possible then how can i do it.

Thanks and Regards
alexander
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 2970
    
    9
Well, I'm sure it's possible, but offhand I don't know the best way to do it. It's probably better if you address this question to everyone, not just me.

One problem is that most encryption schemes basically create binary files, which means the idea of reading a single line will become difficult, as newline characters may appear in the encrypted data, but that has nothing to do with where the line breaks are in the unencrypted data. Considering the size of your data, you may want to split the contents up into a set of smaller files, each small enough to load completely into memory without an OutOfMemoryError. For example, you could have 10000 files of 1000 lines each, each encrypted individually. (I just made those numbers up - you'd have to experiment to see what works for you.) And you need to think about what types of operations you need to be able to perform. Do you want to be able to look up individual records, without necessarily loading every record in the file? In that case you're probably better off with many small files. Or maybe using a database, with each record stored in a BLOB. The best answer will probably depend on how you need to be able to use the data; I'm just tossing out some ideas about things you might want to think about. And I bet there are others here with more experience in this particular area. I hope that helps.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Reg:Reading file with 100,0000 lines
 
Similar Threads
Reg:java.lang.ArrayIndexOutOfBoundsException:
declaring array of vectors
java.io.IOException: Corrupt GZIP trailer
Awkward Memory Issues
How to use Mark and reset for inputstream ?