[Registry-dev] Integrating Apache Lucene with WSO2 Registry

[kalani]

hi,

I have completed the sample application which includes creating the
index in the database, loading index into the memory, searching and
periodically flushing index updates to the database. 

The plan is to write a handler to manage the index and to introduce a
new method to the Registry interface for content searching in order to
integrate this with the Registry.

Thanks,
Kalani  

On Fri, 2008-06-20 at 04:27 +0530, kalani wrote:
> hi Paul,
> 
> I considered about the JVM clustering option using Terracotta framework
> which is mentioned in the 1st link. However I think it is not necessary
> to go that far if we can create the index in the database (I talked
> about this issue with Azeez too). 
> 
> The Compass project which is under Apache license provides an
> implementation of JdbcDirectory. We can use that implementation to get
> the thing done. To overcome the performance issues we can load the index
> to a RamDirectory and periodically flush updates to the database. At the
> moment I am working on a sample application and I think this way is easy
> and hope it will work fine. 
> 
> Thanks,
> 
> Kalani
> 
> On Mon, 2008-06-09 at 15:42 +0100, Paul Fremantle wrote:
> > You might want to read this blog:
> > 
> > http://orionl.blogspot.com/2006/11/clustering-lucene.html
> > 
> > and this:
> > 
> > http://bugs.sakaiproject.org/confluence/display/SEARCH/IndexClusterOperation
> > 
> > Regards
> > 
> > Paul
> > 
> > kalani wrote:
> > > Hi Paul,
> > > 
> > > I got your point and did a search on working with Lucene in a clustered
> > > environment.
> > > 
> > > A possible option is to store the index inside the shared database.
> > > Lucene, however, does not directly support this feature [1]. I will try
> > > to find an implementation of the suggested JdbcDirectory class as well.
> > > I have already sent a mail regarding this to Lucene user list.
> > > 
> > > Thanks,
> > > 
> > > Kalani
> > > 
> > > [1]http://wiki.apache.org/lucene-java/LuceneFAQ#head-e55d8e6971f9f01daaf3e14ce1d2f34485adba6e
> > > 
> > > On Mon, 2008-06-09 at 09:12 +0100, Paul Fremantle wrote:
> > >> Kalani
> > >>
> > >> The issue is that if you create the index in a local file system, you 
> > >> make the overall system harder to cluster.
> > >>
> > >> This is because - in a typical cluster scenario - a shared database is a 
> > >> standard capability and will already be there. A shared file system adds 
> > >> extra complexity.
> > >>
> > >> Please can you find out if there is any guidance on running Lucene in a 
> > >> cluster. Thanks!
> > >>
> > >> Paul
> > >>
> > >> kalani wrote:
> > >>> Hi all,
> > >>>
> > >>> I am working on implementing the content search functionality of the 
> > >>> Registry using Apache Lucene.
> > >>>
> > >>> Apache Lucene can easily be used to search content by indexing the files
> > >>> inside the file system. Additionally,  a database can also be indexed by
> > >>> adding each and every record to an index. The index should not be
> > >>> recreated at each time a new resource is added to the database. The new
> > >>> resource should be indexed and appended to the existing index at the
> > >>> time it is added.
> > >>>
> > >>> The index is created in the local file system. The Lucene project
> > >>> mentions that there is a performance issue in creating the index in a
> > >>> database. However, they have a mechanism called Directory to support it.
> > >>> We can look into that later if needed.
> > >>>
> > >>> Thanks
> > >>>
> > >>> Kalani
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Registry-dev mailing list
> > >>> Registry-dev at wso2.org
> > >>> http://wso2.org/cgi-bin/mailman/listinfo/registry-dev
> > >>>
> > > 
> > > 
> > > _______________________________________________
> > > Registry-dev mailing list
> > > Registry-dev at wso2.org
> > > http://wso2.org/cgi-bin/mailman/listinfo/registry-dev
> > > 
> > 
> 
> 
> _______________________________________________
> Registry-dev mailing list
> Registry-dev at wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/registry-dev