As for our CLucene index corruption issues, as far as I remember we talked to developers on IRC and to Ben in person. We were never able to reproduce the issues in a controlled environment and when we did get corrupted indexes (for example by sending users external hard disks to copy their corrupt indexes to) we couldn’t work out what was wrong.

The Lucene index is actually very generic, and most of the code is meant for dealing with this generality, to provide you with the ability to use the library in a very broad set of usages. The library size you were complaining about, is actually Lucene’s strongest points. It has a record of 7+ years (CLucene has 7, JLucene even more) where many people have tried it under different circumstances, with different hardware and for different use-cases. They have both proof-tested it, and provided fixes or extensions. No new library or code written from scratch can match this.

But, it is not just about stability. And not even about re-using code. I think in the open-source world, there’s a great value for joining forces and working on something together; re-writing something from scratch, for internal use or releasing it to the open, is something one should do only if there’s a compelling reason to do so. There are mainly two reasons for that – you and others. You don’t have to write all the code for yourself and can keep focused on your own business logic, relying on others to do this work for you, and along the way you help improve the original code base and by that you help others.
I’m saying all this, because I don’t recall seeing any report regarding a corrupted index in CLucene. My memory may be fooling me, or I may be new to the project, but as it appears it is quite common for developers using open-source projects not to provide feedback or own patch work to the original project or developers. I think we all are losing lots of good stuff.

You are right about the learning curve for all non-simple usage, although it is not as steep as it looks. Developing tools (Analyzer, Filter, Scorer etc) for a very customized search pattern is indeed not a task one will do with only basic understanding of the code, but is not too hard a task to learn. If there’s something I learned from your post, is how important in-depth documentation, articles and tutorials are. Hopefully we’ll get the time to write many of them soon; right now we are focused on making some really cool code improvements.

You definitely had a very cold epilogue to your journey. It snowed last night up north and in the Jerusalem area. In Israel, that happens like once or twice a year. A real celebration…

Perhaps Lucene isn’t complicated, but it is large. The compressed tar file is 12MB. That’s daunting when you’re trying to learn, and even when you’re trying to debug why the hell your indexes keep getting corrupted (a recurring problem with CLucene that I never resolved during my tenure at Flock). In some ways large is worse than complicated because it just becomes hard to keep things straight.

Anyway, my intention isn’t to reinvent the wheel. It’s ultimately to understand the problem space better. I wasn’t interested in building something “enterprise-level” because I don’t have an enterprise to serve right now.

What I found interesting while building my 250 line inverted index was that I could build application specific search systems rather than trying to customize general purpose ones like Lucene. A lot of the complexity in a system like Lucene is all of the support for specific use cases. It’s necessary for a general purpose library like Lucene to support these specific use cases because every real world system will involve the general inverted index plus something specific to be useful. If the general system can be implemented relatively easily, perhaps we should just be implementing specific full text search systems for specific applications rather than using Lucene’s various building blocks. I know this is counter to the computer science mantras of abstraction and reuse, but I think that those are often applied to readily. Who knows. I know that next time I need to build a search system for a real application I’ll try Lucene and a few of its friends before building my own

I don’t know when I’ll make it back to Jerusalem. My two months in Israel ends on Tuesday when I fly to Ghana. I was visiting Jerusalem from Tel Aviv every week, but I think it’ll be a couple of years at least before I’m back for a visit. We were in Abu Ghosh and Latrun today, I can’t imagine how cold it’s gotten up in Jerusalem now! Brrr!

Lucene is not all the complicated, really. It is very well-structured, and ready for enterprise-level usage (can you allow for multi-searchers / indexers, or distributed 4GB index with your code?).

IMHO, instead of reinventing the wheel, join existing developers and help their efforts in making it roll faster and more efficiently. If you don’t like reading up docs, call me next time you’re in Jerusalem

That’s when all that silly inverted index shit becomes useful.