Brussels

Last month was full of adventures. It started 1st of July with me going to Brussels meat our friend to talk about wikipedia-like site about art. We're going to help this man build the most complete site about art using Wikidot software!

BTW, this was my first flight in lifetime. Quite a strange feeling, but generally fine.

Forms

I was working on some nice technical and UI improvements to Wikidot, that is crucial for the art site (but really really nice for Wikidot as well, like forms for editing, entering and viewing structured data to wiki pages).

Search issues

That week was also spent on some massive Wikidot.com search engine tweaks. A stupid one-line bug, which was not exporting proper LC_ALL environmental variable in indexing script, caused many sites that used Asian or East-European languages to be not indexed (most notably the great ИСТОРИЈСКА БИБЛИОТЕКА). At first we though that we can re-index the broken sites, but our re-indexing mechanism was way too slow (would last for weeks for all broken sites).

Pieter then challenged me. He said he can index whole Wikidot in 6 hours. I thought it's not even possible, but then I started to work on that and I managed to index the whole Wikidot in less than 2 hours without indexing tags at first. Then with tags, it took 2 hours and 10 minutes or so. That was damn fast!

Inspired by this and an accident of disk full error on /var partition of our webserver (but this is why we keep user-uploaded files and other important things on separate disks), I also rewrote the incremental indexer, to work in similar way to the whole-Wikidot-re-indexer.

search-api reindex

If you care about some technical details:

• all search operations are issued with use of search-api, a separate program that can:
  • re-index whole Wikidot
  • queue indexing page/thread
  • queue deleting page/thread
  • queue re-indexing site
  • flush queue
• search-api is written in Python
• search-api uses PyLucene - a native Java Lucene library binded to CPython objects with PyJCC. Compiled with GNU Java Compiler to native code (like C programs), this binding has improved performance over using Lucene with Sun's Java.
• before rewriting it to only-Python, search-api was written in BASH and was a wrapper to:
  • java -jar searchApiHelper.jar search "phrase-to-search"
  • php search-api-helper.php flush
• search-api also takes care of file locking to assure that
  • only one process tries to modify the index
  • items are added to queue one-after-another
  • when doing some big index modification (read: full re-index) queue is not flushed (so that after the re-index all changes are applied to new index)
  • when flushing queue takes more time, and cron tries to run more flushing processes, they simply end (so only one process flushes the queue at a time)

Union of Rock Festival

Just after week spent in Brussels in nice hotel I went to Węgorzewo, Mazury (Poland biggest lakes distinct) to have fun on rock music festival. Unfortunately, the music level was not very impressive, so I mainly enjoyed the atmosphere on the camping area.

The weather was not great. It was wet everywhere, the ground was covered in 20 centimeters of mud and it was hard to walk around without getting dirty. But during the first day of being there, I learned to do that.

Improved workflow at Wikidot

Some of you noticed, that recently we started to work more efficiently, but this is not quite true. In fact we work as efficiently as before, but we are better organized, and have better priorities on tasks. Also we keep track of what we do, so we can then tell what we've done. So for us, this is a little more work of "documenting" our work (so maybe we work even less efficiently than before?), but for the outside world, we make more noise (in a positive meaning) around that. So basically, people know what we do, what we are going to do, when they can expect changes and most importantly, they understand why some feature request is being postponed. This is (and was) because we have more important things to do, but before they couldn't tell it.

Squark turned into a professional project manager, that manages our time. pieterh decided to talk to the Community and listen to their complaints (he reads or at least skims every post on Community forums). He tells Łukasz what needs to be done, Łukasz knows when we will have time to do this. This way communication inside Wikidot improved. Also we (michal frackowiak and me) no longer look on Community forums (some of you may regret), but this allows us to concentrate on our work.

The work continues

As I mentioned before, we want to introduce a great feature to Wikidot, which is forms. But the implementation now concentrates on the open source version of Wikidot software (once it's ready, working and tested we'll copy the feature to the Wikidot.com service).

aptitude install wikidot

As forms is a huge change, I started to prepare a good ground for it and closed most important bugs in Wikidot open source and I'm about to start making Ubuntu packages for it to allow even-simpler installation on Debian-based systems. Now the installation involves only 6 child-easy steps and in fact can be done by copying&pasting a few commands.

Yesterday's party

Yesterday I went to met some old-school-times friends in the heart of the city. It was meant to be a meeting for "a beer or two" but evolved into beer and dancing till morning. That was first time I get a morning bus (not even the first) to my home just after partying.

It was such a great fun and great folks I met.

Summary

I hope with this long blog post (but divided into friendly sections ;) ) I recompensed long period of not-posting anything here.

by Gabrys

This work was done, because searching the whole Wikidot database was far from being fast. At first we used Google Custom Search Engine to solve the problem, but we wanted to be independent from it. Also we wanted to include search results that are accessible by the person that searches but not by search engines (like private sites).

This worked really good. The only problem was long time spent to display the results. It was like 20 seconds, which was far better than when using previous search, but much worse than Google. This was strange, because from previous tests, we calculated the average search time should be about a second or two.

It started to be clear, when we noticed that only 20-30 searches per day are performed. The index is quite big, and it needs to be cached in RAM to work with sufficient performance.

Today I did more tests under heavy load and it seams Lucene can handle big number of queries. When users search often, the index is partially or even fully cached by the filesystem and searches are really quick!

But our main problem to solve today was slow local search a.k.a. "search this site". Moreover many concurrent queries were degrading database performance (not only for searching), so we decided to enable Lucene for local searches as well.

I must say it works really nice, fast and has a nice set of syntax tricks you can do with it, for example you can search for pages with something in tags. Just search for youtube tags:embed. This would search for pages matching youtube (in tags, title or content) and with embed in tags. If no such pages are found, partial matches are also returned, like: pages matching youtube (but with no embed in tags) or pages with embed in tags, (but not matching youtube).

To sum up, new search is faster, gives more accurate results, saves the database performance (which was the main goal) and allows nicer syntax than the old one.

by Gabrys

I described the system extensively in blog post titled New search for Wikidot's gonna rock.

The new search system replaced the old Google-powered one.

The main advantages over Google Search Engine that has been used till now are:

• search in public sites + those you are a member of
• semantic search: tags, title are more important when searching
• simple to sophisticated queries
  • "blog site:community" — search for anything with "blog" in it but only on site community.wikidot.com
  • "tags:wikidot site:quake" — search for pages tagged "wikidot" on my site (quake.wikidot.com)
• poor quality content filtered

Things left to do:

• add site preview (thumbnail) like the one on right to the search results
• explain in simple words the search syntax
• promote good and/or active sites (give them higher rank)
• decrease delay from editing to updating search index (should be max 5 minutes)
• create custom search module searching in a bunch of (related) sites — this would be nice if you keep separate sites for different areas of a project like private site for project members and public one for project users. The search is intelligent enough to filter restricted items from search results if someone is not member of the private site the items come from.

by Gabrys

This seemed quite easy at first having nice Lucene implementation in PHP — included in Zend Framework and indeed during tests it was fast, simple and powerful. But this was tested on about 100,000 documents (document is a Wikidot page or forum thread) and we have about 2,500,000 documents in Wikidot now. And this is where the problem begins.

After indexing roughly 1,800,000 documents there were some problems with memory consumed by the indexing process (500 MB merory limit was not enough in SOME cases).

Even earlier I realized that the search times weren't good enough. This is why I implemented the searching part in Java, which is the native platform for the Lucene indexer. This sped things up.

Do you think indexing a document in just a second is fast? I though this is a good result. Indexing a document takes about 0.2 s when having small amount of documents in the index already. But when you have 400,000 documents in index, adding another document to the index takes about 0.4 s. And having even this "good" indexing time (below a second), indexing the whole Wikidot would take at least a few days.

This leads me to a conclusion, that Wikidot is really BIG.

A similar situation also applied to the user uploaded files. There was a problem of a limit of filesystem reached, which was about 32,000 directories max in a single directory. Having all user-uploaded files in a directory structure of one-directory-per-wiki, this resulted in a problem when having more than 32,000 wikis.

Replicating this structure to another machine (also known as live-backup of user-uploaded files) was also quite a challenge, because we've reached a limit of directory watches in the kernel-level filesystem-monitoring system (inotify).

It all shows, that things that seem easy are not necessarily easy because of the high scale of the Wikidot, which touches some limits on nearly every piece of software we use. But this is also a great chance to really test those projects and how they react to such a high load.

by Gabrys

After that time of developing and dealing with performance problems (searching the whole Wikidot in 3 seconds is too long!) it's time for test this thing!

Introduction

The whole thing is about Search All Sites module for Wikidot Open Source. This entry is mainly for those of you that run their own Wikidot services. More info on getting Wikidot software run can be found on Ed's site.

Fresh install

After installing Wikidot Open Source from current version, you can instantly test the new Search All Sites module. Just navigate to /search:all page of your main wiki. Example: if your wiki farm runs on the domain mydomain.com and your main wiki is www.mydomain.com, navigate to http://www.mydomain.com/search:all.

Existing install

Updating to the new search engine from already installed version is quite tricky, because you need to pre-populate the search index (which is the actual file that is searched by the index, when looking for term user entered).

You can try the following commands:

• obtain root priviledges

 sudo su 

• navigate to your Wikidot directory (it is /var/www/wikidot by default), update the code and run the lucene_bootstrap.php script as your lighttpd user

 cd /var/www/wikidot svn update cd tests sudo -u www-data php lucene_bootstrap.php 

This command adds every page to the index (normally located at /var/www/wikidot/tmp/lucene_index). Once indexed a page can be searched (if the site containing the page is public or you're a member of the site). The command prints a dot for each 10 indexed items (item is every page and forum/comments thread).

If this runs smoothly (i.e. no error, Segmentation fault at the end is OK, but memory exhausted is not OK) you have all your sites indexed and ready to search through.

When it fails: you can increase the max_memory setting in the corresponding php.ini file and re-run the command. There is no bad thing in running this command more than once as indexing a page always deletes the page from index before adding it again.

Just go to /search:all location at your main wiki and search for some content.

Also you need to update your crontab file. Add:

 * * * * * www-data /var/www/wikidot/bin/job.sh UpdateLuceneIndexJob 

Features

• First of all the new search applies only to the Search All Sites i.e. Search This Site works in the old way.
• Search uses titles and tags intelligently
  • pages with the exact search phrase in the title are placed higher in the result list
  • pages with tags matching search phrase are quite high in the result list
  • pages with title matching search phrase are quite high in the result list
  • pages with content matching search phrase are somewhere low in result list
  • pages with parts of search phrase matching titles and tags can be higher in the result list than the pages having content matching even the exact phrase
  • this all means: tags and titles are more important than content for the search engine
• You can narrow your search to only selected wikis
  • append site:site1,site2,site3 (no spaces between them) to your search query. Example: search for "gabrys site:www,community" searches for gabrys in titles, tags and contents of pages and threads inside of sites www.yourdomain.com and community.yourdomain.com (supposing your Wikidot installation runs on yourdomain.com)
• The search includes public sites plus sites you are a member of. Also the results from your sites are generally more relevant to the search engine (i.e. they appear higher than the results from other sites)
• The search results for given phrase for given user are cached (if memcached is used) for a few minutes. This makes the search even more smooth (no need to search the index again when user only switches the result page from 2 to 3 for example)

Test it!

If you don't run and don't want to run your own Wikidot installation, you can try the new features on the following site:

http://www.wikicomplete.info/search:all

Extras

Highlighting

Like Google's way to highlight the searched words in Google-cached versions of result pages? Now you can add this feature to your Wikidot installation as well.

Open conf/wikidot.ini file and append those lines:

 [search] ; enables highlighting of search phrase in the resulting documents highlight = true 

This will highlight the words user searched for using:

• Google Search
• Yahoo Search
• Your Wikidot installation search:all page

Need more performance or memory limit exhausted

We experienced some low performance when searching through 2 millions of pages and threads of Wikidot.com. The search results were generated in about 3 seconds. This was not enough for us, so we manage to speed things up using the native Java Lucene implementation for searching the index. This works because we use PHP Lucene implementation that is compatible with the Java's one. This means we can index page with PHP and search with Java. And we do it! If you want do this too (experiencing low search performance of getting memory exhausted error messages), just add the following lines to your conf/wikidot.ini file:

 [search] ; enables the use of Java for searching use_java = true 

Notes

• if you already have [search] section in the conf/wikidot.ini file, just add the use_java = true line in the search section
• enabling Java for searching requires you to install java executable for your system. You should know how to do this (try sudo aptitude install openjdk-6-jre).
• you don't need any Java libraries as we already bundled everything needed in the .jar file. The Java source and Ant build script is located normally at the /var/www/wikidot/java directory (assuming you installed the wikidot in /var/www/wikidot.

Summary

Once we assure the search is stable and gives relevant results, we'll introduce it to the Wikidot.com service. I calculated that indexing all the sites would take about 3 days! But searching is done in less than 1 second (using the Java program).

I'm looking for your comment on the features. Especially if you've tried them yourself!

by Gabrys

I must say it's gonna absolutely rock!

First of all it'll give much more relevant results than before:

• title and tags are much important than pure page content for the indexer
• only 1-3 minutes delay of indexing/searching after an edit (compare to Google's a few days)
• searches for given phrase in all public sites plus all the sites you are member of (including private)!
• results from wikis you are member of appear generally higher in the result list, because the indexer gives them more relevance factor than to similar results from other sites.

More features:

• really fast (typically search is done in just 2 seconds or even less if the result is cached)
• thumbnails of sites (feature to come)
• short activity of site information (feature to come)
• supply the sites to search through in the query with special keyword: site:X,Y,Z — searches for given phrase in three sites: X, Y, Z
• supply the sites to search through as the param to SearchSites module (feature and module to come yet)

Are you convinced yet?

by Gabrys

The thing that needs to be said is the Wikidot Source changes. It is not meant to be solid stable yet. The development of .com and OS version is quite separate, although there is a flow in code in two ways:

• fixes for bugs in .com having the way to OS
• many future improvements in .com are first tested and introduced in OS

The latter introduces not-working revisions into the OS repository from time to time, but this is a product in creation (pre-1.0), so this just can happen.

Stability

The current version (revision 317) of Wikidot Open Source is used on Wiki Complete.

It's rather stable, but you have to know the following:

Lighttpd only

It's for lighttpd only (we want support for Apache back in 1.0).

INI file for configuration

File conf/GlobalProperties.php is no longer needed and needs to be deleted. But first have a look at conf/wikidot.ini and try to migrate any custom settings you had in conf/GlobalProperties.php.

The full-blown verbose example of wikidot.ini is stored in conf/full-example-of-wikidot.ini. This is not meant to even work, it's just every possible option listed and described. If the option has an default value it is also mentioned. Use the file as the reference for your wikidot.ini file.

HTML user-uploaded files hosting disabled by default

Since it could be dangerous in certain cases, we have disabled serving of HTML files with the default Wikidot installation. However Internet Explorer (6 and 7) ignores the hint to display HTML files as source, so it's not really a solution.

The solution that really works is having a totally separate domain for hosting uploaded files only (at wikidot it is wdfiles.com). The settings for this would be:

 [security] upload_separate_domain = true upload_domain = your-different-domain.com ; having different domain for uploads we can safely enable user-uploaded HTML files serving upload_restrict_html = false 

Search All Wikis module

The SearchAllModule (allowing to search all wikis) is very experimental now. If you have already any content, that needs to be searched, don't upgrade (or you can restore the previous php/modules/search/SearchAllModule.php file after the upgrade).

The module is being migrated to Zend_Search_Lucene and require doing an initial indexing of all sites.

Also it's very probable the search mechanisms changes any time soon, so I just don't recommend using the new version of module.

Search This Wiki feature is unaffected!

Search Highlight

In the meantime of developing new search engine for Wikidot, we came into an idea to highlight phrases user searches for (using internal Wikidot search or Google/Yahoo search engine).

You can enable it in the wikidot.ini file appending:

 [search] highlight = true 

Search features (the new SearchAllModule and Search Highlighting) is already tested on Wiki Complete, however there are some known limitations it the search module, that need to be improved yet.

Upgrading

As a matter of fact, we don't supply any "upgrade" script. This unfortunately doesn't mean that there are no problems with that.

Once we release the first stable version, we'll supply upgrade scripts for each incremental upgrade i.e. 1.0 -> 1.1 -> 1.2 and so.

This is because it would be really easier if we know what version EXACTLY we're moving from to what EXACT version. Otherwise we would end up with scripts that can work but can break something. If you're in trouble with upgrading, ask on our dev-list.

by Gabrys

Active testing is done on the Wiki Complete wiki farm based on Wikidot software. The same one, that you can test experimental search phrase highlighter at.

You can check the result at the search all wikis page on Wiki Complete.

I look forward any bug reports.

by Gabrys

How does it work?

Here are the Google results for female superheroes wikicomplete query:

locate any link from wikicomplete.info and click. The words female, superheroes, wikicomplete should be highlighted using different colors.

The same mechanism is used for local searches.

hartnell (the main admin of the Wiki Complete) really likes this feature. There is some chance we'll introduce it to Wikidot.com! After the new search for wikidot is done.

Tech

If you wonder how did I do this, here is the answer. I used Zend_Search_Lucene utility for highlighting. It was all quick and easy after parsing the HTTP_REFERER looking for the actual search query.

Summary

This feature was implemented on some random pages I walked through. I hope you like it being implemented on WikiComplete!

by Gabrys

We now host 2 million pages and the full text search engine we had used was not fast enough to satisfy a regular user. The average search time (in all wikis) was about 30 seconds.

Google searches the whole Wikidot in less than 1 second. The downsides of using Google engine are:

• using external service — prestige and dependence
• displaying ads on search results — for those who don't use AdBlock
• pages get indexed after some significant time
• only public wikis can be indexed

One important thing is Google indexes every content from a site. This includes Wikidot.com footer, menus, wiki header, the real content and tags. All of these is treated in an unknown way, so we have no big/real impact on how Google treats different portions of pages.

This leads to a conclusion, that we need a search engine.

We would like:

• to treat tags as more important than the regular content
• not to search from Wikidot.com static elements (like the footer on every page)
• allow searching all wikis available for given user
  • all public wikis
  • all private wikis that the user is a member of

Coming to technical details, one could say, we just need a generic full text search engine. We can use one available in our storage system or one of dedicated search-only engines.

Tsearch

Tsearch — the full text search engine for PostgreSQL (storage used for Wikidot data) is currently used when searching a wiki. This is quite nicely integrated and plays well. But when there are over 20,000 wikis to search (I mean only non-spam public ones), the efficiency is not enough.

Lucene

Lucene — one of the most popular search engines for Java is one of the possible choices. The mechanism it works is the following:

• application pulls some documents to the search index
  • document is a webpage in our situation
  • index is a datastore to be used when searching
• user queries the index with a query
  • a bunch of documents is returned in the order of relevance
  • the documents returned are more or less the same as the documents pulled to the index before

This mechanism requires populating index by application

• updating the index every now and then
• updating document on some change — like page edit

and requires us to define some functions that deal with search results

• the original webpage is usually not stored in the index, but only tokenized, to allow finding it when searching for any word in the document
  • this makes the index smaller and faster
  • this makes we need to store an additional ID of a webpage, to be able to retrieve the full result from the database based on the stored ID

Nutch

Nutch would be a different — more Googlish — approach to the search issue. Nutch indexes mainly HTML files (given URLs) and crawls through the the links. This has both advantages and disadvantages:

The main advantage of using Nutch is that as a search result we get a formatted HTML document

• with links to items found
• with context of the search phrase quoted
• the search phrase words outlined in some way

This is very similar to what we get searching for some phrase with Google.

What I don't like about Nutch is quite big overhead of populating the index. A page must be compiled by the server and HTML must be produced. Then the same HTML must be parsed by the search engine to get important data. There is a lot of information generated by the server and then forgot by the search engine.

Nutch (similar to Lucene) is a Java project and requires some Java environment. This may and may not be a problem, but is a point we must concern when looking for optimal solution.

There is OpenSearch project which aims to make the Nutch results more interchangeable (exporting them as RSS feeds). Using it a PHP application can safely ask for results a HTTP service and get RSS feed to parse and present to user.

Zend_Search_Lucene

There is also a quite nice thing around: Zend_Search_Lucene. It is a search engine written entirely in PHP being a part of Zend Framework for PHP. The internal format of the search index file is compatible with Lucene and this is where the name of the package comes from. Also the query language is the same (or very similar).

It seems, the PHP implementation should be really slow, when searching really big sets of data, but after some testing, we get the search results for almost any query in about 1 second, searching almost the whole Wikidot.

I think this is a really nice solution, because it can integrate well with the existing PHP code of Wikidot. Also the searching can be easily parallelized for many machines. For example, you can have 4 search machines, each getting 1/4 of search queries to carry out. This way we don't reduce the search time, but avoid searching many things in one index at once.

There are some options to consider when dividing the search queries to different machines. We can select the machine to perform the search by random, by turn or by search hash. Search hash would make a MD5 sum (or other hash) of a query, compute the modulo rest from division by number of machines from the hash (treated as an integer) and assign the search to the machine having number of computed modulo. This means the same query will always be performed on the same machine (it can be then better cached or optimized).

The Zend implementation of Lucene is also really trivial to understand and use, so it seems a good start for me. Testing it on the whole public part of Wikidot I got a index of about 500 MB. Adding a single page to the index of this size takes about 2 seconds. Searching — about 1 second.

Sphinx

When I have asked my friend about full text search engines he recommends, he pointed out Sphinx — standalone application for this purpose. It is not very popular software as it haven't find it way to the Ubuntu repository for example, but it seems very interesting.

Sphinx can be fed with XML streams of data from any application, can fetch data from PostgreSQL or MySQL databases or be communicated with via its API and libraries to many languages.

It seems it's somehow similar to Lucene, but implemented using traditional languages, not Java.

The choice

There are probably some other solutions that are worth trying, but I think the most appropriate for now is using the Zend's one as it's the easiest to adapt. We can optionally use some caching mechanisms and queries distribution. Also we need more testing of situations that may appear (like a need to perform 100 queries simultaneously).

by Gabrys