Piotr Gabryjeluk

I think using Google is the best bet.

In fact, I would highly recommend using the Google Search API (or whatever it is) and completely integrate it with wikidot for searching all wikis and even individual ones.

My two cents. :)

— hartnell

without any adware blocks - as an application on your server ?

Have I mis-understood this from google ?

I think the current search engine per wiki is amazing — and superior to Google's. I say this because it searches content that is current, whether Google can only search content that it has seen (cached). If you update a page on the wiki, you would need to wait for Google to come to your wiki and update its cache for search results to render up-to-date ones.

But when it comes to searching all Wikidot sites, rendering out-of-date content doesn't really matter. If you want to find a specific thing on a particular site, just go to that site and search it. If you want to find a general context on Wikidot, then Google's out-of-date cache would do perfectly.

We have to mix a few things:

• immediately indexing page
• intelligent searching (tags and page title is more important when searching than content)
• search results in minimum time

With the new module it could be possible to supply which sites to search, i.e. one could insert a module that searches for given phrase in 5 different wikis (probably content-related).

Just as easy as for example (THIS IS JUST A PROPOSAL. DO NOT TRY IT):

[[module Search sites="community handbook snippets"]]

And you have a search box that searches for you in 3 wikis at once (and the sites are indexed immediately after each change), so you get very fresh results.

I was doing similar research on external full text search engines and found this article (in spanish but you can translate) http://www.alfonsojimenez.com/2007/08/30-benchmark-lucene-en-php-vs-lucene-en-java which compares Java Lucene vs the PHP Zend one. According to this, the php one is considerably slower.

I'm leaning towards Sphinx myself. Has a PHP Api and if it can power craigslist well…

Java Lucene is a few times faster than PHP version. It's like 1.5 s (PHP) to 0.5 s (Java) when searching for some simple phrase and like 10 s (PHP) to 1.5 s (Java) when searching for something with ~ (which means "similar" in Lucene query language). But this was tested for about million of documents. Normally, when you don't have so many of them, PHP is really OK, as difference between 0.2 and 0.05 is almost unnoticeable.

In Wikidot we use PHP to index things (because we have nice database-object mapping in PHP) and Java to search (but then PHP to interpret the search results and ask database for more information on found items).

Also, the benchmark doesn't have much to do with real life, as normally you don't search for the same thing 100 times in raw, usually a phrase is searched once. My benchmark tests searching for a phrase once.

This includes delays of loading all Java/PHP classes and stuff. I run the command a few times before to let the kernel cache files and then I measure times of Java and PHP searching for the same phrase.

Java is then about 3 times faster than PHP on a quad-core processor and 500 MB Lucene index (hopefully, with over 1 GB free RAM, all sitting in system cache — in RAM).