• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

most recent 3 - 4 posts not included in Lucene reindex

 
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In our mysql database, we only have about 15 posts. Post ids 14 - 34, with a few posts missing in the middle. When I start a new reindex from the admin control panel, the newest 3 - 4 are not included in the new index. Also, during debugging, I ran into a few file locking problems caused by handles not being freed. My code fixes are posted below. I built the project from source. I pulled from cvs on 05 Feb 2008.

This fixed the indexing problem... (LuceneIndexer.java)

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();
throw new SearchException(e);
}
}

These fixes helped the file locking problem... (LuceneIndexer.java)

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false;
e.printStackTrace();
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();

}
}
}
[originally posted on jforum.net by bwfrieds]
 
Migrated From Jforum.net
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
bug # JF-831
[originally posted on jforum.net by bwfrieds]
 
Migrated From Jforum.net
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
After a deployment to IBM Websphere Portal on linux, I saw the problem was not yet fixed. More changes were necessary to get the reindexing to behave reliably. Those changes are below. I placed my initals next to my changes. I think the final fix was the call for writer.flush() before the call to optimize() and after the call to addDocument().

Occasionally I will still see an IOException during testing; when trying to update the .cfs files. I don't know where that file handle is not being released.

LuceneIndexer.java

public void batchCreate(Post post) {
synchronized (MUTEX) {
try {
Document document = this.createDocument(post);
this.ramWriter.addDocument(document);
this.ramWriter.flush(); // bwf
this.flushRAMDirectoryIfNecessary();
} catch (IOException e) {
e.printStackTrace(); //bwf
throw new SearchException(e);
}
}
}

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();// bwf
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();// bwf
throw new SearchException(e);
}
}

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.flush(); // bwf
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false; //bwf
e.printStackTrace();//bwf
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();
}
}
}

private void optimize(IndexWriter writer) throws Exception {
if (writer.docCount() % 100 == 0) {
if (logger.isInfoEnabled()) {
logger.info("Optimizing indexes. Current number of documents is " + writer.docCount());
}
writer.flush(); // bwf
writer.optimize();

if (logger.isDebugEnabled()) {
logger.debug("Indexes optimized");
}
}
}

[originally posted on jforum.net by bwfrieds]
reply
    Bookmark Topic Watch Topic
  • New Topic