Win a copy of Microservices Testing (Live Project) this week in the Spring forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Liutauras Vilda
  • Henry Wong
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Al Hobbs
  • Carey Brown
Bartenders:
  • Piet Souris
  • Mikalai Zaikin
  • Himai Minh

searching a large text file >1gb

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi all,

Can anyone suggest one what can be a efficient method to search a large text file >1gb .I tried using file channel but it is throwing an exception.Is there any other way??

Thank You
 
Ranch Hand
Posts: 63
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You'll need to be a bit more specific. A simple way to search a text file of arbitrary size is to simply step through it looking for a keyword. If you're getting an exception due to being out of memory, you have a memory leak somewhere, or are trying to read through the whole file at once.

For any operation, you can typically be efficient with time, memory, or I/O - usually you get to pick two of those three. If you don't have enough memory to do it quickly, you will take longer to do it a different way.
 
Ranch Hand
Posts: 357
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Since the file is very big, I would suggest to use a Scanner since it can do its scanning using an input stream and there will be no need to load all the file data to memory to make the search.

Scanner Documentation
 
Rancher
Posts: 43028
76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What exception are you getting?

Also, what would the result of the search be? A line number? The line in which the search term occurs? Something else?

Using a library like Lucene may also help.
 
reply
    Bookmark Topic Watch Topic
  • New Topic