Lucene and Google App Engine

[Paul Michael]

To Dear Lucene in Action Authors,

GAE has been getting a lot of buzz lately but one of the major complains is the lack of full-text search capabilities which is what Lucene provides. Is there a chance that Lucene (either Java or Python version) will be usable in GAE?

Thanks.

[Michael McCandless]

There's this project:

http://code.google.com/p/gaelucene/

But I haven't used it personally.

[Erik Hatcher]

And there's this excellent post: http://www.ultrasaurus.com/sarahblog/2010/06/full-text-search-on-app-engine/

[Gian Franco]

...why mix the two solutions?

Gian

[Paul Michael]

Michael: The GAELucene project shows very low activity.
Erik: The JRuby approach doesn't seem to be a complete solution.

I guess we still have to wait for some time for a more complete and mature solution to come out.

Anyway, thanks for the suggestions, I'll try to consider them in my next projects!

[Michael McCandless]

Or, you can help to push to a more complete and mature solution for Lucene running on GAE ;)

Do you have a sense of what challenges/limitations GAE imposes?

[Paul Michael]

Haha that's a nice suggestion Michael...

I'd probably give it more consideration if I were 10 years younger and still single. I'm afraid most of my waking time is already spent on managing the house, making a living and working on ministry projects.

Anyway, here are some of the restrictions for Java according to an article from InfoQ:

* Once a request is sent to the client no further processing can be done. This includes data streaming.
* A request will be terminated if it has taken around 30 seconds without completing. At this point an exception is thrown. If not caught a 500 error is returned to the user.

Moving up the stack are several sandbox restrictions:

* Applications can not write to the file system and must use the App Engine datastore instead.
* Applications may not open sockets
* Applications can not create their own threads or use related utilities such as timer.

java.lang.System has been restricted as follows:

* exit(), gc(), runFinalization(), and runFinalizersOnExit() do nothing.
* JNI access is not allowed.

In addition to these key items there are other limitations such as a JRE class loading white list. From the documentation GAE seems to impose a good deal of its magic using custom classloaders. However, they should allow other application level classloaders as long as they can operate with the above restrictions.

[Michael McCandless]

Those restrictions sound fine w/ Lucene. I guess the biggest thing is not using the filesystem, ie we need a Directory impl backed by the App Engine datastore (and if I remember there's a max file size restriction (maybe 10 MB?) in this?).

You'd have to switch to SerialMergeScheduler, too, since the default ConcurrentMergeScheduler does merges with its own background threads.