• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

File Reading

 
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
How read a huge volume of file (nearly 18 lks lines) by line by line in minimum of time?
 
Sheriff
Posts: 22781
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Using BufferedReader and its readLine() method? You don't need to store each line, just the current one.
 
Deva Devan
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Buffered reader and scanner also taking more time to read and split the large file.
 
Deva Devan
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,
i am reading messages from large file and spliting based on the messages and storing to corresponding folders..
100 mb file taking 25mins of time. i want to reduce the time.

Thanks
 
Rob Spoor
Sheriff
Posts: 22781
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
How long does it take to just read the file line by line? And with just read I mean just that - just read. Don't do anything with the lines you read. Because 25 minutes for a 100MB file seems odd, and I doubt it's the reading part that is causing this slowness. The processing of the lines seems like a more likely candidate.

Never optimize anything before you're sure it needs optimizing. And I don't think the reading needs much optimizing.
 
Ranch Hand
Posts: 119
Oracle Opera Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
BufferedReader and its readLine() method is the only way to read, if you using with visualbasic there is a function called readAll() check for it
 
Ranch Hand
Posts: 68
Netbeans IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
There is the readFully method in RandomAccessFile that reads the entire into a byte[], but having that in memory might be little heavy. The readFully method is specified by the DataInput interface so other classes might implement it as well. 25 minutes for 100 MB is too much so there is a bottleneck elsewhere, other I/O operations is an obvious possibility.
 
Atul Darne
Ranch Hand
Posts: 119
Oracle Opera Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
dear philip thank you for the valuable reply, i will use and try the same. but iam unaware about the Randomaccessfile and DataInput interface
 
Ranch Hand
Posts: 781
Netbeans IDE Ubuntu Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Atul Darne wrote:dear philip thank you for the valuable reply, i will use and try the same. but iam unaware about the Randomaccessfile and DataInput interface



Sorry but I think this is a red-herring! As has already been said, your problem is unlikely to be caused by the file read operations since I can read hundreds of megabytes in just a couple of seconds. In your position I would try to isolate the area that is causing the problem. I would start by creating a program that just reads the file line by line and does nothing else. If you do this you will most probably find that it is your post-read processing that is causing the problem.
 
Deva Devan
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi I have attached my code part.



this process taking long time for reading, matching with regex and splitting files.

Thanks
 
Philip Grove
Ranch Hand
Posts: 68
Netbeans IDE Firefox Browser Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Move the Pattern.compile out of the loop, it's just wasting time recompiling the pattern for every line.
 
Rob Spoor
Sheriff
Posts: 22781
131
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
+1
 
Deva Devan
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks..

But its taking 3 mins for 10mb file.

This is too costly..

 
Philip Grove
Ranch Hand
Posts: 68
Netbeans IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I suspect a second problem is the use of the mkdirs() method of the File class. It creates the entire folder tree but it has to check permissions every step of the way and check whether the folder exists or should be created. It doesn't fail because then you would see the, in my experience, quite rare SecurityException, but each check takes time and is an I/O operation.

Unless it changes per log entry move that part out of the loop as well.
 
Deva Devan
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

How to remove empty lines while we unzip the .txt file ?

Thanks
 
Philip Grove
Ranch Hand
Posts: 68
Netbeans IDE Firefox Browser Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Thamizh Bharathi wrote:Hi,

How to remove empty lines while we unzip the .txt file ?

Thanks



The mainmat.find() should return false for a line that doesn't match so they should be removed. In case it return true the regexp needs to be rewritten which is outside the scope of this forum. You could also simply add a check for whether the line is empty before doing anything with it, that way you can avoid it outputting "unmatched".

Major reworking of your code is in order as you construct dir with date and deviceIp in the parameter string regardless of whether you get a match or not, likewise you use time to build filename.
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic