Pieter was the Wikidot front-man for its community and organized good workflow for many community and internal projects. We learned how to talk to the community, how to solve problems and how to concentrate on important things. And we're going to keep being good at it.

With the addition of data forms Wikidot becomes one of the most interesting wiki platfarms (platform + farm) on the Earth. Using ListPages, _templates and forms we allow (almost) regular folks to create applications like CMS for their co-workers, clients or friends. Things usually done by programmers are now available for smart guys without need to find proper hosting, web framework or writing a line of computer code.

What I find most interesting in building applications on Wikidot is that the platform itself is quite bullet-proof. I mean when you forget to put semicolon at the end of some line, you don't get your application crashing because of syntax error. Moreover when making application you get storage with history and e-mail notifications about changes for free, which makes your data quite safe. In case someone destroys your site (which you are informed by e-mail), you can just revert the right version).

But Wikidot applications is not the most important thing Wikidot is used for. Many sites on Wikidot are just plain wiki sites with links to store and share knowledge of certain groups. They don't use advanced features like modules, includes, templates. They usually edit and save :-) from time to time uploading some file or image. What we want to do is to review all common operations people do most often and (really really) simplify them. Sophisticated applications are worth nothing if basic usage of service is not easy.

We want to introduce features people will appreciate, like [[html]] blocks for embedding completely custom HTML, like this:

 [[html]] <h1>Custom HTML</h1> <p>Something else</p> <img src="anything.png" alt="hello ;-)"/> [[/html]] 

This code will just inject the HTML (including JavaScript) into the page in that place. Seems easy to do, but in fact, to make it safe for your site, we needed to implement this as a frame, that autoresizes to keep the content fit inside.

We also want to introduce [[block]] syntax for sandboxing wiki texts, for example:

 [[block]] What put inside is parsed and rendered independently, so even if you put some random [[/div]]s or other nasty things, the page outside will be rendered nicely. [[/block]] 

This will be useful for site designers to use in template pages, like this:

 [[table]] [[row]] [[cell]] [[block]] %%content%% [[/block]] [[/cell]] [[cell]] some other things. [[/cell]] [[/row]] [[/table]] 

Putting %%content%% in block tags you make sure user won't break the original layout (the table). Without block, user could put [[/cell]] inside of his text and thus break the layout.

To let our users know what happens on their sites, we want to make live feeds of activity available for most common start-places on the web — iGoogle, NetVibes, Facebook and Twitter. Most important news from YOUR sites will be next to other things you start your day from.

We have a bunch of other fresh ideas, which we'll be sharing on blog.wikidot.com.

by Gabrys

Oto wynik:

Poza tym co widać, zmiany dotyczą użytego fonta, wyglądu komentarzy i forum, uproszczenia okienka nowego komentarza, ustalenia pozycji tła (żeby przewijanie przewijało samą treść, a nie tło) i dodanie lekkiego prześwitywania tła, żeby było bardziej sexy.

Temat jest hostowany pod adresem: http://themes.wikidot.com/orange-black-improved.

A wszystko po to, żeby nadać naszej nowej stronce: http://parapetowa.wikidot.com/ ładnego wyglądu. A na stronce oprócz informacji co można nam kupić z okazji zbliżającej się parapetówy (jeszcze dokładnie nie wiadomo kiedy) uruchomiliśmy też bloga, gdzie pisać będziemy (a przynajmniej ja) o trudzie przystosowania mieszkania do wymagających lokatorów (czyli nas). Link do bloga:

http://parapetowa.wikidot.com/blog

Serdecznie zapraszamy!

by Gabrys

Maintaining Wikidot is a bit more complex, because, files uploaded to sites are located in web, side to side with some static Wikidot php and javascript files. Also for historical reasons, there are web/files--common and web/files--local directories, which maps to /common--* and /local--* URLs and in fact, the files--local is never served directly by the web server (need to check permissions first).

Also some time ago, we made static files versioned, so that we can apply more aggressive HTTP caching to them (reducing average page load time) and still be able to fix bugs on them without waiting a few days till the cache expire. In current model, URL to static file contains version hash, this may be for example: http://static.wikidot.com/v--b44e0ce810ee/common--javascript/WIKIDOT.js (notice the b44e0ce810ee). The whole static.wikidot.com is now hosted on Amazon's CloudFront, which means you get static Wikidot files from a server nearby your location and not always from USA.

This all become quite complicated, so we decided to make things really clear and simple in the source code. The primary rule: make the source code (updatable from git) separate from files uploaded by users and generated by Wikidot. Second rule: make files that are automatically generated during installation (not in the runtime) separate from persistent files (like the uploaded by users) and from source code.

And at the end there needs to be some place for logs and a place for temporary data (we need this to generate some random cool stuff, but after generating them, files are deleted).

So we end up with something like this:

• WIKIDOT_ROOT
  • data/
    • avatars/ — user avatars
    • sites/ — site files (both generated thumbnails and uploaded files)
  • generated/
    • static/ — generated static files. This dir can be server directly by a fast non-PHP webserver for static.wikidot.com in case we don't want CloudFront anymore
  • tmp/ — temporary files including Smarty compiled versions of templates. Content of this dir can be safely removed
  • logs/ — Wikidot logs
  • everything else — comes from git and is unchangeable by application

Application needs write-access to data, tmp and logs. Generated dir needs write access to one installing or upgrading application.

Wikidot persistent data is now ONLY database and data/ directory, so it's easy to backup and restore the application (if you have enough time to make full backup of this).

There is still one exception to this nice schema which is php/db/base directory, which is autogenerated during installation from XML database definition files, but the cleaning is not over, I still work on this.

Nice thing about this work is that it does not need a lot of code changing, because directory paths are usually stored in one (max two) places in application, so this kind of totally reorganizing directory structure does not break things. As such, it is very very worth doing it. In the end we get clean internal structure of files and it's clear which files you can safely remove, which you can restore from git (and thus you can experiment a little on them — in case of crash, just re-download application), which are "state" of the Wikidot and where to look for logs.

This all is also very important, because we aim to make current Wikidot.com source open and as such we want it to be a nice code.

by Gabrys

Wikidot was down for about 12 hours and the time it was down was full of work for us. We got a few things that could be broken starting from recent changes of the Wikidot software, hardware failure, high load-related kernel bugs or limitations to connection number or maximum possible number or file descriptors.

The problem is Wikidot kind of worked, so some people had their sites loading, some other not and getting "500" errors. We didn't want to stop it, but at some point Wikidot was completely unusable. We switched the database to the other machine, but this was not the solution, then we switched all Wikidot traffic to the machine, still no good, we switched the software to some previous version, but this still seemed bad.

Finally we worked out, there was a site, that had so big traffic, that it killed anything else (and itself as well). When we temporarily disabled it, the whole Wikidot started to work nicely again. Then Michał made some improvements for the high traffic site serving and the situation is stable again.

In the middle of everything, we had huge problems with our hosting company and their service called Portable IP addresses. It seems that switching DNS is much more reliable that using Portable IPs that took hours to switch (and were supposed to take seconds to switch)! DNS switching time was 15 minutes.

We learned a lot from the situation. Hardware upgrade postponed from really long time needs to be done quite quickly. We need more servers, to see which element breaks. For example if database server has high load, we know we need to tune database settings. If we have all on one massive server and one brick on it crashes, it usually causes all the server overloaded and this causes other bricks to crash as well, so it's hard too tell what the real problem is.

Another thing is that we see our users want information on what happens. It's bad when Wikidot crashes, but it's even worse, when it crashes and they have no information about this.

So, the next time a similar disaster happens we'll update on each technical detail possible, to let you know, that we know it's broken and we work hard to fix it. Other thing is, we plan having more fail-over servers in case something dies.

Thank you all for using Wikidot, it's a great pleasure working (and fixing things) for you!

by Gabrys

Brussels

Last month was full of adventures. It started 1st of July with me going to Brussels meat our friend to talk about wikipedia-like site about art. We're going to help this man build the most complete site about art using Wikidot software!

BTW, this was my first flight in lifetime. Quite a strange feeling, but generally fine.

Forms

I was working on some nice technical and UI improvements to Wikidot, that is crucial for the art site (but really really nice for Wikidot as well, like forms for editing, entering and viewing structured data to wiki pages).

Search issues

That week was also spent on some massive Wikidot.com search engine tweaks. A stupid one-line bug, which was not exporting proper LC_ALL environmental variable in indexing script, caused many sites that used Asian or East-European languages to be not indexed (most notably the great ИСТОРИЈСКА БИБЛИОТЕКА). At first we though that we can re-index the broken sites, but our re-indexing mechanism was way too slow (would last for weeks for all broken sites).

Pieter then challenged me. He said he can index whole Wikidot in 6 hours. I thought it's not even possible, but then I started to work on that and I managed to index the whole Wikidot in less than 2 hours without indexing tags at first. Then with tags, it took 2 hours and 10 minutes or so. That was damn fast!

Inspired by this and an accident of disk full error on /var partition of our webserver (but this is why we keep user-uploaded files and other important things on separate disks), I also rewrote the incremental indexer, to work in similar way to the whole-Wikidot-re-indexer.

search-api reindex

If you care about some technical details:

• all search operations are issued with use of search-api, a separate program that can:
  • re-index whole Wikidot
  • queue indexing page/thread
  • queue deleting page/thread
  • queue re-indexing site
  • flush queue
• search-api is written in Python
• search-api uses PyLucene - a native Java Lucene library binded to CPython objects with PyJCC. Compiled with GNU Java Compiler to native code (like C programs), this binding has improved performance over using Lucene with Sun's Java.
• before rewriting it to only-Python, search-api was written in BASH and was a wrapper to:
  • java -jar searchApiHelper.jar search "phrase-to-search"
  • php search-api-helper.php flush
• search-api also takes care of file locking to assure that
  • only one process tries to modify the index
  • items are added to queue one-after-another
  • when doing some big index modification (read: full re-index) queue is not flushed (so that after the re-index all changes are applied to new index)
  • when flushing queue takes more time, and cron tries to run more flushing processes, they simply end (so only one process flushes the queue at a time)

Union of Rock Festival

Just after week spent in Brussels in nice hotel I went to Węgorzewo, Mazury (Poland biggest lakes distinct) to have fun on rock music festival. Unfortunately, the music level was not very impressive, so I mainly enjoyed the atmosphere on the camping area.

The weather was not great. It was wet everywhere, the ground was covered in 20 centimeters of mud and it was hard to walk around without getting dirty. But during the first day of being there, I learned to do that.

Improved workflow at Wikidot

Some of you noticed, that recently we started to work more efficiently, but this is not quite true. In fact we work as efficiently as before, but we are better organized, and have better priorities on tasks. Also we keep track of what we do, so we can then tell what we've done. So for us, this is a little more work of "documenting" our work (so maybe we work even less efficiently than before?), but for the outside world, we make more noise (in a positive meaning) around that. So basically, people know what we do, what we are going to do, when they can expect changes and most importantly, they understand why some feature request is being postponed. This is (and was) because we have more important things to do, but before they couldn't tell it.

Squark turned into a professional project manager, that manages our time. pieterh decided to talk to the Community and listen to their complaints (he reads or at least skims every post on Community forums). He tells Łukasz what needs to be done, Łukasz knows when we will have time to do this. This way communication inside Wikidot improved. Also we (michal frackowiak and me) no longer look on Community forums (some of you may regret), but this allows us to concentrate on our work.

The work continues

As I mentioned before, we want to introduce a great feature to Wikidot, which is forms. But the implementation now concentrates on the open source version of Wikidot software (once it's ready, working and tested we'll copy the feature to the Wikidot.com service).

aptitude install wikidot

As forms is a huge change, I started to prepare a good ground for it and closed most important bugs in Wikidot open source and I'm about to start making Ubuntu packages for it to allow even-simpler installation on Debian-based systems. Now the installation involves only 6 child-easy steps and in fact can be done by copying&pasting a few commands.

Yesterday's party

Yesterday I went to met some old-school-times friends in the heart of the city. It was meant to be a meeting for "a beer or two" but evolved into beer and dancing till morning. That was first time I get a morning bus (not even the first) to my home just after partying.

It was such a great fun and great folks I met.

Summary

I hope with this long blog post (but divided into friendly sections ;) ) I recompensed long period of not-posting anything here.

by Gabrys

This work was done, because searching the whole Wikidot database was far from being fast. At first we used Google Custom Search Engine to solve the problem, but we wanted to be independent from it. Also we wanted to include search results that are accessible by the person that searches but not by search engines (like private sites).

This worked really good. The only problem was long time spent to display the results. It was like 20 seconds, which was far better than when using previous search, but much worse than Google. This was strange, because from previous tests, we calculated the average search time should be about a second or two.

It started to be clear, when we noticed that only 20-30 searches per day are performed. The index is quite big, and it needs to be cached in RAM to work with sufficient performance.

Today I did more tests under heavy load and it seams Lucene can handle big number of queries. When users search often, the index is partially or even fully cached by the filesystem and searches are really quick!

But our main problem to solve today was slow local search a.k.a. "search this site". Moreover many concurrent queries were degrading database performance (not only for searching), so we decided to enable Lucene for local searches as well.

I must say it works really nice, fast and has a nice set of syntax tricks you can do with it, for example you can search for pages with something in tags. Just search for youtube tags:embed. This would search for pages matching youtube (in tags, title or content) and with embed in tags. If no such pages are found, partial matches are also returned, like: pages matching youtube (but with no embed in tags) or pages with embed in tags, (but not matching youtube).

To sum up, new search is faster, gives more accurate results, saves the database performance (which was the main goal) and allows nicer syntax than the old one.

by Gabrys

At Wikidot we use PHP5 as FastCGI backend to Lighttpd light-and-fast webserver. It works like this:

• there are a few hundreds of php5-cgi processes (name is cgi, but they also support FastCGI mode) running and waiting to be used
• lighttpd (only one needed!) process manages the network connections to all the clients and once the request is ready serves a static file or forwards the request to one of PHP backends processes.

We used to use internal Lighttpd FastCGI process manager, meaning the lighttpd processes actually used to start the PHPs.

Problems

We encountered some known problems of 500 (server side) errors appearing after some random time, especially under a high traffic. The typical message appearing at the Lighttpd's error.log was:

 <some date>: (mod_fastcgi.c.2494) unexpected end-of-file (perhaps the fastcgi process died): pid: ... 

There are plenty of reports on this in both Lighttpd's and PHP's forums, bug trackers and even some blogs.

Workarounds

We managed to write some hacky scripts that detected the situation and restarted the backends when needed. The reaction was so quick, that almost no-one noticed the error, but damn, this is not how WE solve problems.

A blind try

We decided to give spawn-fcgi a shot. What is it? It is a program that spawns FastCGI backends (independently from Lighttpd server). Why trying it? I've read somewhere, that it works more reliably than the internal Lighttpd spawner. What's interesting is that this program comes from lighttpd package, so we're in family anyway. It's mainly intended to run the FastCGI backends from different user than the webserver user or to run them on different machine(s) than the webserver machine. This can be used naturally for some smart load-balancing.

The only problem of this solution we encountered was internal limit of number of processes to spawn by a single process which was 256 (hardcoded, fixed in next versions). But at the same time, we decided to build a few FastCGI bridges (each spawning ~200 PHPs) anyway so that was no longer a problem for us.

What was quite surprising (but honestly, I deeply believed in this), our problems with 500 server errors and PHP disappeared. This configuration works for about 2 weeks now with absolutely no hacky scripts involved and no restarting needed. Cool.

Why I wrote this

I wrote this short note just for the record and to let other people know, that using spawn-fcgi instead of the internal Lighttpd's FastCGI spawner might solve their problems with PHP (FastCGI) and 500 internal server errors.

Hope this helps someone.

by Gabrys

Also we decided to move from managing our code in SVN to Git — more precisely to www.github.com. Our project page at GitHub is the following: http://github.com/gabrys/wikidot. Feel free to just follow it or even fork!

Last times, I made Wikidot insanely easy to install (check out the installation guide). It totally rocks (you can install Wikidot within 10 commands and no file editing).

I want to state, that this post is the last about the Wikidot open source software on this blog, as I'm running a new blog just about the Wikidot software at wikidot.org. This will be more practical to filter the posts and will push some life into that site.

At the end I want to invite you to the Wikidot IRC channel #wikidot at irc.freenode.org. That would be probably the easiest way to contact the Wikidot team without much formalism.

(As many posts here were about Wikidot software, this blog will be less regularly updated, but if really care about Wikidot software news, just follow the new one and you won't be spammed about posts about Python, BASH or other things. I hope that blog-split will really help everyone.)

by Gabrys

The biggest change is Wikidot is now single wiki engine by default (you need additional configuration for the wiki farm mode).

What demonstrates the change is the simplified installation process.

Now it's as simple as installing any other software or service:

1. install the dependencies
   • for Ubuntu: sudo aptitude install lighttpd php5 php5-cli tetex-bin tetex-extra gs-gpl imagemagick libmagic1 subversion postgresql-8.3 php5-tidy php5-pgsql php5-gd zip make
2. get the code
   • svn co http://svn.wikidot.org/repos/wikidot1/trunk/ wikidot-install-dir
   • cd wikidot-install-dir
3. if on Ubuntu: configure conf/wikidot.ini and set user at [db] section to your login name
4. prepare database (create role and database)
   • as user postgres: make prepare_db
5. build Wikidot: configure files and make initial dabatase
   • as you: make
   • this prints a secret URL at the end, you'll use it to configure Admin password
6. start Wikidot
   • ./wikidotctl start
7. set Admin password
   • use your browser and navigate to the secret URL make prints at the end

We would like to finally create Debian/Ubuntu packages for that (once it gets more stable and have some bug fixed). So keep tuned!

by Gabrys

Single wiki mode

The main difference is introducing a new Wikidot installation mode: "sigle wiki mode" and setting it as the default instead of "wiki farm mode".

What does it mean?

• you don't have to bother about:
  • domains (DNS)
  • mail configuration
  • root access (you can start Wikidot on custom port — default 8080)
• you get direct administration panel for managing users

But:

• you get only ONE single wiki
• you cannot create more wikis
• every IP/DNS address pointing to the started server (and corresponding port) displays THE wiki
• no mail invitations or password recovery
• no HTTPS

Lighttpd

Our main webserver is now Lighttpd, we forget about Apache. We can do this, because now you can run your regular Apache server on :80 port and Lighttpd serving Wikidot on :8080 (or other). They don't overlap, so there's no need to keep the compatibility.

Wikidot is a full web service with its dedicated web server running on separate port. This is the philosophy.

Scripts

To make Wikidot easily installable, we're going to create a script (possibly a Makefile) that will simplify the whole process. It would be 4 steps:

• install dependencies
• setup the database
• edit ini file (supply database credentials)
• make

Dependencies are Lighttpd, PHP, PostgreSQL, ImageMagick and maybe a few other things. Each of them can be installed without root access, but with root access it would be just as easy as

• aptitude install lighttpd php5-cgi postgresql-8.3 ImageMagick

(or so)

Running/stopping Wikidot:

• ./wikidotctl start
• ./wikidotctl stop

Next steps

Once all this is done (which isn't too much work BTW), we could make a *.deb package, that would take care of dependencies, database, creating a special user for wikidot, setting the root directory by convention and installing init script in proper place. Then you would just:

• /etc/init.d/wikidot start/stop/restart

This should kill all Wikidot-is-to-hard-to-install issues.

… upgrade to wiki farm

Once you have your single-wiki Wikidot up and running you may want to upgrade to full featured wiki farm solution. The one-wiki would become the main wiki (as is www.wikidot.com for the Wikidot.com service), other wikis would be able to be created. On the other hand, as your getting more powerful, you need to configure your DNS (have a domain-class just for your wiki farm), mail service and move the service to the main HTTP port (80).

The hardest thing

Q: What is the hardest thing to do in that plan?
A: The hardest thing is to create a module to manage wiki users.

This should not be too hard, so in the overall, this should work!

I'm waiting for your opinions.

by Gabrys

But the API is also important, because it can enable developers (which are big fraction of our users) to create applications that use Wikidot.com as a service. This can be useful in many ways, the best example I can think of is creating an application, that get some photos from your disk, uploads them to Wikidot, creates a page for each of them and saves some metadata from them into it.

Today, I finally managed to start working on one of the most important API method which is page.save. This method obviously is for saving a page. The page saving mechanism in Wikidot is quite complex, because of advanced site permissions, various performance optimizations, caches and many scenarios of updating a page.

Also, we wanted to make page.save to be one method to update and create pages, which involves other permission checking, because in some cases you have permission to create a page, but not to edit. This is especially true if you have a site category with everybody-can-create-but-only-author-can-edit policy.

I started from dealing with updating already existing pages, so for now page.save can't yet create pages. We have more or less working page.save for the following activities:

Notes:

• the API service is XML-RPC based, so the syntax of calls is syntax of the programming language you use (Python used as an example syntax)
• site: quake and page: dev:api-save means http://quake.wikidot.com/dev:api-save page
• setting tags as a single string (comma- or space-separated) is also supported
• setting of parent page is also supported in similar way to setting the title

Now we're going to review the code, ensure there is no more caching mechanism I forgot about and make final polishing. This includes generating notifications to be send to page-watchers and saving revision comments.

This method is crucial for the API. Once it gets fully working (including creating pages), applications will be able to:

• create pages
• read pages
• modify pages

and this the core functionality we want to provide with the Wikidot API.

Check out the API methods to see if page.save is already out.

If you want to test the (beta) API, drop a comment at the Wikidot API page.

by Gabrys

Normally starting up a machine takes a minute or two and is almost indistinguishable from a network outage or some other temporary failures. But with Wikidot having as many files that our users upload the operation of checking the filesystem takes long time.

Not even because of today's crash, I must confess, we have plans of decentralizing the service and moving it to more distributed environment to let it be (even) more reliable. Even including the crash we have still very high uptime, that would satisfy just everyone. But not us. We aim at having 100% (or more ;-) ) uptime, and make things totally fault-tolerant.

I must say we are really really sorry for what happened today but in the same time I must ensure that we really care about you — the Users — as many of you have noticed for sure. I hope you still believe in us :).

by Gabrys

I described the system extensively in blog post titled New search for Wikidot's gonna rock.

The new search system replaced the old Google-powered one.

The main advantages over Google Search Engine that has been used till now are:

• search in public sites + those you are a member of
• semantic search: tags, title are more important when searching
• simple to sophisticated queries
  • "blog site:community" — search for anything with "blog" in it but only on site community.wikidot.com
  • "tags:wikidot site:quake" — search for pages tagged "wikidot" on my site (quake.wikidot.com)
• poor quality content filtered

Things left to do:

• add site preview (thumbnail) like the one on right to the search results
• explain in simple words the search syntax
• promote good and/or active sites (give them higher rank)
• decrease delay from editing to updating search index (should be max 5 minutes)
• create custom search module searching in a bunch of (related) sites — this would be nice if you keep separate sites for different areas of a project like private site for project members and public one for project users. The search is intelligent enough to filter restricted items from search results if someone is not member of the private site the items come from.

by Gabrys

• blogi
• wizytówki firm
• prezentacje wyrobów i produktów
• bazy wiedzy (najbardziej naturalne dotąd zastosowanie technologii wiki)

Od szczegółu do ogółu

Korzystając z Wikidota skupiamy się na tworzeniu pojedynczych treści, takich jak:

• aktualność
• wpis na blogu
• opis produktu

Każda taka treść otrzymuje swój adres (URL) i możemy do niej przypisać takie właściwości jak:

• kategoria
• etykiety (tagi)
• dodatkowe pliki (np. zdjęcia)

Strony wyświetlające podsumowanie generują się automatycznie dzięki użyciu modułu ListPages. Przykłady użycia modułu:

• wyświetlenie ostatnich 3 aktualności
• wyświetlenie wpisów z bloga z ostatnich 30 dni
• wyświetlenie stron z kategorii blog z etykietą osobiste

Kategorie

Kategorie grupują strony. Każda kategoria może mieć następujące własności:

• własny temat graficzny
• własny szablon (np. wyświetlający datę i autora wpisu nad treścią strony a pod nią moduł komentarzy i oceniania)
• własne uprawnienia (np. możliwość dodawania plików przez niezalogowanych użytkowników plików do stron)
• własne elementy nawigacyjne (menu górne i boczne)

Galerie zdjęć

Wystarczy dodać pliki graficzne do strony a w jej treści wpisać [[gallery]], aby stworzyć estetyczny pokaz zdjęć.

Automatyczna mapa witryny

Wikidot pozwala ustalić dla każdej strony jej stronę nadrzędną. Sprawia to, że łatwo jest wyświetlić drzewo stron obrazujące ich hierarchię np.

• Strona główna
  • O firmie
  • Aktualności
  • Nagrody i historia firmy
• Produkty
  • Rośliny doniczkowe
    • Paprocie
    • Kaktusy
  • Sadzonki
    • Zioła
    • Krzewy owocowe
• Kontakt
  • Biuro w Szczecinie
  • Oddział w Gdańsku

Umożliwia to łatwą nawigację nie tylko użytkownikom ale i wyszukiwarkom internetowym, dzięki którym łatwo będzie można odnaleźć witrynę w Internecie.

Standardy sieciowe i nowoczesne technologie

Wikidot generuje bardzo dobry kod XHTML oraz kanały RSS dla wszystkich istotnych treści prezentowanych na stronie co zapewnia idealny kanał komunikacji z zewnętrznymi serwisami, które mogą wyświetlać aktualności z witryny w innym serwisie.

Zaawansowane statystyki

Dla użytkowników konta Pro+ dostępne są statystyki witryny, które działając w oparciu o program awstats pokazują ilość odwiedzin o danej porze czy w danych dniach tygodnia. Dostajemy również listę najczęściej odwiedzanych w ramach witryny adresów, używanych przeglądarek internetowych i wykres szacowanego czasu spędzonego w witrynie.

Można uzyskać jeszcze bardziej szczegółowe statystyki podłączając usługę Google Analytics w panelu administracyjnym strony.

Własna domena i stopka

W pewnych zastosowaniach przydatna może być opcja ustawienia własnej stopki (i usunięcie odwołań do Wikidota) oraz domeny, która nie kończy się na wikidot.com.

Wygląd

To co dla niektórych jest najważniejsze, to nadanie odpowiedniego wyglądu stronie, ponieważ w przeglądaniu stron internetowych wzrok jest najczęściej wykorzystywanym zmysłem.

Wygląd stron na Wikidocie można w ogromnej mierze dostosować dzięki użyciu arkuszy CSS, które są standardowym sposobem opisywania wyglądu treści w sieci Web (więc potrafi je przygotować każdy grafik internetowy).

Co więcej wygląd może się zmieniać z kategorii na kategorię, co pozwala na tematycznie wyróżnienie pewnych sekcji w obrębie całej witryny.

Przykłady

Przykładowe strony umiesczone w Wikidot.com:

• strona belgijskiej firmy iMatix
• wiki gry Frets On Fire
• prezentacja biżuterii
• mój blog

by Gabrys

The main purpose of this, was to make installation of this pretty piece of software as easy as possible.

Having this said, I'll prove it is really simple on the example of Ubuntu Linux.

Overview

• Step one: Install required packages
• Step two: Configure database
• Step three: Choose a proper place for wdLite and download it
• Step four: Launch web-based installer
• Step five: Your wiki is ready

Step one: Install required packages

Issue the following command:

 sudo apt-get install apache2 php5 imagemagick subversion postgresql postgresql-contrib php5-tidy php5-pgsql php5-gd zip 

Step two: Configure database

Use your user name instead of piotr. Use some (not yours) password instead of strong-password.

 sudo su postgres psql --command "CREATE USER piotr PASSWORD 'strong-password'" createdb --owner piotr piotr 

Step three: Choose a proper place for wdLite and download it

Version a

The best place for Wikidot would be probably somewhere under /var/www

(replace piotr with your user name)

 sudo mkdir /var/www/wiki sudo chown piotr /var/www/wiki cd /var/www/wiki svn checkout http://svn.wikidot.org/repos/wdlite/trunk . chmod a+rwX . -R 

Having wdLite copied there, it'll be accessible at: http://localhost/wiki/

Version b

You can also locate wdLite to your public_html directory.

 cd ~/public_html sudo svn checkout http://svn.wikidot.org/repos/wdlite/trunk . sudo chmod a+rwX . -R 

In order for this to work, you would need to issue the following commands:

 sudo ln -s ../mods-available/userdir* /etc/apache2/mods-enabled sudo /etc/init.d/apache2 restart 

wdLite will be accessible at: http://localhost/~piotr/ (replace piotr with your user name).

Step four: Launch web-based installer

Go to http://localhost/wiki (version a) or http://localhost/~piotr (version b), click the "Click and install a wiki here!" link.

Fill ALL the fields. If you followed this howto, the values would be as follows:

• PostgreSQL settings
  • user: piotr
  • password: strong-password
  • database: piotr
• GMail settings
  • you have to create a GMail account for sending verification emails from it
  • supply the GMail login and password in the fields
• Wiki settings
  • it's totally up to you what to include there
  • the values will be accordingly the wiki title and subtitle (shown at the very top of wiki)
• Wiki administrator — it's you!
  • e-mail: e-mail address you will use to log in
  • password: password you will user to log in
• Members
  • if you have friend you would like to invite to the new wiki, just enter their e-mail addresses separated with spaces. You can leave this single field empty
• Confirm and save
  • click save and wait while the wiki is installing the database and sending invitation e-mails. This should take no more than 5 minutes (usually less than 1 minute)

Step five: Your wiki is ready

Log in using the e-mail address and password that you supplied in the previous step.

Enjoy your wiki!

by Gabrys

Good night

by Gabrys

The primary aim of this project is to make installation dead simple and server requirements really small.

Server requirements

wdLite should be installable on:

• Apache with PHP5 (no safe mode or other limitations) and PostgreSQL on Linux boxes

PHP and PostgreSQL should be already configured to work with each other. You should have a PostgreSQL database and "user/password/database"-based access to it. Wikidot will create tables, but won't create a database. You should either create it as root or have root created it for you before (this process might be automatic on webhosting services with PHP/PostgreSQL).

This configuration should include a whole bunch of virtual hosting providers.

Installation

The installation process should be no harder than this:

• Get a zip or checkout the newest version from repository
• Upload the directory to the server
• Adjust directory permissions
• Go to install.php script with your browser
• Supply mail and PostgreSQL credentials
• Choose your wiki name and create users
• Enjoy your new wiki

What are the differences between "full" Wikidot installation and wdLite

Limitations

• only one wiki
• no page revision diffs
• more limited page size
• lower security (especially for IE users)
• works only with Apache
• some features disabled or non-working
• memcached disabled
• karma disabled
• notifications disabled

Better than full version, because

• works with Apache
• works on any HTTP port (not only 80)
• works within any directory (also in user directory accessible like http://myserver.com/~quake/something/really/deep/wikidot)
• easier installation with a web interface
• no root-access needed
• works well with GMail to send mails from the service
• easily installable on Ubuntu
• the easiest method to start developing with Wikidot
• no need to manually compile additional software

Current work progress

I'm about to pre-release this software, to let you test it.

I have to:

• create a list of things that need to work before a final 1.0 release.
• redirect / to /?/
• create install.php

Things that work already:

• logging in/out
• displaying pages
• editing pages
• saving pages
• some basic modules
• uploading and displaying files
• navigation (links are rewritten from absolute to relative)

How does it work

The wdLite is based on Wikidot OpenSource. It contains wikidot, index.php and a bunch of helpers scripts. The index.php file is a hacky PHP script that

• converts URL-s like http://some.server.com/some/url/?/front:page to http://www.some.server.com/front:page
• tricks Wikidot software to think that the Wikidot domain is some.server.com and the main wiki is www.some.server.com
• sets a bunch of system variables Wikidot relies on, like $_SERVER['REQUEST_URI'], $_SERVER['QUERY_STRING']
• runs a proper one from Wikidot scripts
• … or serves (more like redirects to) a static file
• catches the script output
• runs some transformation on caught output (like converting the links from http://www.some-server.com/some:other-page to ?/some:other-page)
• sends the data back to browser

WARNING, ACHTUNG
The script is more a dirty hack than a version of Wikidot, but this is intentional. We don't want to mantain to many versions of Wikidot. Having this dirty script "only using without modyfing" Wikidot software makes it quite independent from changes in Wikidot. This means the same wdLite script will work for a newer version of Wikidot (=less maintaining work).
WARNING, ACHTUNG

by Gabrys

In simple words this will allow people to write applications that connect to Wikidot.com and perform some actions for the user that runs the application.

Technically, the Wikidot.com API is an XML-RPC service exporting methods from a few especially designed classes.

To connect to an XML-RPC service, you must know its endpoint, which is a regular URL (http:// or https://) address. We decided to use HTTPS to secure the channel from the very start.

The operations we are going to support are:

Browse

• site.categories
• site.pages
• page.get

Above ones are already implemented. Using the API calls you get retrieve almost all data you stored on the Wikidot.com sites!

Modify

• page.save

This will be the basic method to update the content on your site. We plan several other methods, but this is the one that is the most important.

Comments

• page.comments
• page.comment

They will be used to get and post comments on a given page. Using reply_to parameter, there is a possibility to reply to a particular comment.

Forum

• forum.groups
• forum.categories
• forum.threads
• forum.post

This bunch of methods are going to give you full access to the forums you have started on Wikidot.

How to use the API

We haven't yet enabled the API access to the main Wikidot.com server, but testing the API with Python XML-RPC library is as easy as this:

 >>> from xmlrpclib import ServerProxy >>> s = ServerProxy('SOME-URL') >>> s.system.listMethods() ['system.listMethods', 'system.methodHelp', 'system.methodSignature', 'system.multicall', 'site.pages', 'site.categories', 'page.get'] >>> print s.system.methodHelp('site.pages') Get pages from a site Argument array keys: site: site to get pages from category: category to get pages from (optional) >>> s.site.categories({'site': 'gamemaker'}) ['project', 'rpg', '_default', 'action', 'admin', 'badge', 'beginner', 'contests', 'error', 'event', 'example', 'forum', 'gamemaker', 'gml', 'gmlcode', 'gmupload', 'help', 'ide', 'include', 'mamber', 'member', 'nav', 'portal', 'resource', 'search', 'system', 'talk', 'template', 'tutorial', 'video-tutorial', 'wiki', 'challenge', 'howto', 'recent-changes', 'scratch-pad', 'helpdesk', 'helprequest', 'default', 'testimonial', 'testimonials'] >>> [p['name'] for p in s.site.pages({'site': 'gamemaker', 'category': 'badge'})] ['cool', 'flux', 'c-team', 'gml', 'member', 'madman', 'not-a-noob', 'f-madman', 'start', 'break-it'] >>> print s.page.get({'site': 'gamemaker', 'page': 'badge:cool'})['source'] [[table style="width:98%;margin-right:auto;margin-left:auto;margin-bottom:1%;"]][[row]][[cell style="width:360px;"]] [[div class="error-block"]] Included page "include:cool-badge" does not exist ([/include:cool-badge/edit/true create it now]) [[/div]] The Cool Badge is given to people who make something really cool. [[/cell]] [[cell style="vertical-align:top;border:1px solid #ddd;padding:1%;"]] +++ Display Code [[table class="code"]][[row]][[cell]] @@[[include include:cool-badge member=member name]]@@ [[/cell]][[/row]][[/table]] +++ Tag {{cool-badge}} [[/cell]][[/row]][[/table]] [[table style="border:1px solid #ddd;padding:1%;margin-right:auto;margin-left:auto;margin-bottom:1%;width:98%;"]][[row]][[cell]] ++ Earn It * Program something really cool using gml * Make a really cool game +++ Tips * Make sure you post your examples and games on the [[[forum:start|forum]]]. Otherwise no one can see it and nominate you for the cool badge. [[/cell]][[/row]][[/table]] [[table style="border:1px solid #ddd;padding:1%;margin-right:auto;margin-left:auto; width:98%;"]][[row]][[cell]] ++ Members Who Have Earned the Cool Badge [[module ListPages category="member" order="titleAsc" tag="cool-badge" perPage="100" separate="false"]] * %%linked_title%% [[/module]] [[/cell]][[/row]][[/table]] >>> 

A few words of explanation:

• first we import ServerProxy class from XML-RPC library,
• then we construct the ServerProxy object s supplying the endpoint URL (SOME-URL in this case, as we don't have yet decided what the URL is going to be)
• we can see a list of methods by calling system.listMethods on the ServerProxy object
• we get a help message for a method by calling system.methodHelp
• then we get categories of site gamemaker (yeah, it's a part of the wikicomplete.info)
• then we call site.pages method (specifying site and category parameters), but instead of displaying the whole list of structures that describe pages, we only display their names
• calling page.get returns an array with the information about a page, including:
  • wiki source, array key: source
  • generated HTML, array key: html
  • array with various meta-data, array key: meta
• we call page.get passing as the argument array that specifies site and page name, get the page object, but displays only what's stored under the source array key

As you see playing with this is really easy as is browsing the available methods and using them.

Why XML-RPC

We've chosen this protocol because it is an easy way to develop both server and client in almost any programming language. Also it gives some flexibility in passed arguments and return values.

We use struct XML-RPC type as the argument and return value type, which is mapped to associative array or dictionary in client (and server) libraries. Each API method gets a bunch of required and optional parameters, that are basically values stored in the struct passed to API methods.

For example site.pages gets a struct with the following keys:

• site (site name to get pages from) — required
• category (category to get pages from) — optional

This means, you have to create an associative array (when using PHP) or a dictionary (using Python) and pass it as the method argument:

 # PHP $pages = $server->site->pages(array("site" => "my-site", "category" => "my-category")); 

 # Python pages = server.site.pages({"site": "my-site", "category": "my-category"}) 

Using other programming languages, you'll end with something similar. You can almost always create the array/dictionary in-place, so having this convention is not a big deal.

Applications

I'm working on a filesystem based access to Wikidot site (using FUSE and Python).

We plan having a Wikidot application for iPhone.

A save-it-directly-on-wikidot plugin would be a nice thing for various text editors (and probably other applications).

And probably there are billions of other ways to use this API we're not even aware of. If you have any, feel free to leave a comment.

by Gabrys

This seemed quite easy at first having nice Lucene implementation in PHP — included in Zend Framework and indeed during tests it was fast, simple and powerful. But this was tested on about 100,000 documents (document is a Wikidot page or forum thread) and we have about 2,500,000 documents in Wikidot now. And this is where the problem begins.

After indexing roughly 1,800,000 documents there were some problems with memory consumed by the indexing process (500 MB merory limit was not enough in SOME cases).

Even earlier I realized that the search times weren't good enough. This is why I implemented the searching part in Java, which is the native platform for the Lucene indexer. This sped things up.

Do you think indexing a document in just a second is fast? I though this is a good result. Indexing a document takes about 0.2 s when having small amount of documents in the index already. But when you have 400,000 documents in index, adding another document to the index takes about 0.4 s. And having even this "good" indexing time (below a second), indexing the whole Wikidot would take at least a few days.

This leads me to a conclusion, that Wikidot is really BIG.

A similar situation also applied to the user uploaded files. There was a problem of a limit of filesystem reached, which was about 32,000 directories max in a single directory. Having all user-uploaded files in a directory structure of one-directory-per-wiki, this resulted in a problem when having more than 32,000 wikis.

Replicating this structure to another machine (also known as live-backup of user-uploaded files) was also quite a challenge, because we've reached a limit of directory watches in the kernel-level filesystem-monitoring system (inotify).

It all shows, that things that seem easy are not necessarily easy because of the high scale of the Wikidot, which touches some limits on nearly every piece of software we use. But this is also a great chance to really test those projects and how they react to such a high load.

by Gabrys

After that time of developing and dealing with performance problems (searching the whole Wikidot in 3 seconds is too long!) it's time for test this thing!

Introduction

The whole thing is about Search All Sites module for Wikidot Open Source. This entry is mainly for those of you that run their own Wikidot services. More info on getting Wikidot software run can be found on Ed's site.

Fresh install

After installing Wikidot Open Source from current version, you can instantly test the new Search All Sites module. Just navigate to /search:all page of your main wiki. Example: if your wiki farm runs on the domain mydomain.com and your main wiki is www.mydomain.com, navigate to http://www.mydomain.com/search:all.

Existing install

Updating to the new search engine from already installed version is quite tricky, because you need to pre-populate the search index (which is the actual file that is searched by the index, when looking for term user entered).

You can try the following commands:

• obtain root priviledges

 sudo su 

• navigate to your Wikidot directory (it is /var/www/wikidot by default), update the code and run the lucene_bootstrap.php script as your lighttpd user

 cd /var/www/wikidot svn update cd tests sudo -u www-data php lucene_bootstrap.php 

This command adds every page to the index (normally located at /var/www/wikidot/tmp/lucene_index). Once indexed a page can be searched (if the site containing the page is public or you're a member of the site). The command prints a dot for each 10 indexed items (item is every page and forum/comments thread).

If this runs smoothly (i.e. no error, Segmentation fault at the end is OK, but memory exhausted is not OK) you have all your sites indexed and ready to search through.

When it fails: you can increase the max_memory setting in the corresponding php.ini file and re-run the command. There is no bad thing in running this command more than once as indexing a page always deletes the page from index before adding it again.

Just go to /search:all location at your main wiki and search for some content.

Also you need to update your crontab file. Add:

 * * * * * www-data /var/www/wikidot/bin/job.sh UpdateLuceneIndexJob 

Features

• First of all the new search applies only to the Search All Sites i.e. Search This Site works in the old way.
• Search uses titles and tags intelligently
  • pages with the exact search phrase in the title are placed higher in the result list
  • pages with tags matching search phrase are quite high in the result list
  • pages with title matching search phrase are quite high in the result list
  • pages with content matching search phrase are somewhere low in result list
  • pages with parts of search phrase matching titles and tags can be higher in the result list than the pages having content matching even the exact phrase
  • this all means: tags and titles are more important than content for the search engine
• You can narrow your search to only selected wikis
  • append site:site1,site2,site3 (no spaces between them) to your search query. Example: search for "gabrys site:www,community" searches for gabrys in titles, tags and contents of pages and threads inside of sites www.yourdomain.com and community.yourdomain.com (supposing your Wikidot installation runs on yourdomain.com)
• The search includes public sites plus sites you are a member of. Also the results from your sites are generally more relevant to the search engine (i.e. they appear higher than the results from other sites)
• The search results for given phrase for given user are cached (if memcached is used) for a few minutes. This makes the search even more smooth (no need to search the index again when user only switches the result page from 2 to 3 for example)

Test it!

If you don't run and don't want to run your own Wikidot installation, you can try the new features on the following site:

http://www.wikicomplete.info/search:all

Extras

Highlighting

Like Google's way to highlight the searched words in Google-cached versions of result pages? Now you can add this feature to your Wikidot installation as well.

Open conf/wikidot.ini file and append those lines:

 [search] ; enables highlighting of search phrase in the resulting documents highlight = true 

This will highlight the words user searched for using:

• Google Search
• Yahoo Search
• Your Wikidot installation search:all page

Need more performance or memory limit exhausted

We experienced some low performance when searching through 2 millions of pages and threads of Wikidot.com. The search results were generated in about 3 seconds. This was not enough for us, so we manage to speed things up using the native Java Lucene implementation for searching the index. This works because we use PHP Lucene implementation that is compatible with the Java's one. This means we can index page with PHP and search with Java. And we do it! If you want do this too (experiencing low search performance of getting memory exhausted error messages), just add the following lines to your conf/wikidot.ini file:

 [search] ; enables the use of Java for searching use_java = true 

Notes

• if you already have [search] section in the conf/wikidot.ini file, just add the use_java = true line in the search section
• enabling Java for searching requires you to install java executable for your system. You should know how to do this (try sudo aptitude install openjdk-6-jre).
• you don't need any Java libraries as we already bundled everything needed in the .jar file. The Java source and Ant build script is located normally at the /var/www/wikidot/java directory (assuming you installed the wikidot in /var/www/wikidot.

Summary

Once we assure the search is stable and gives relevant results, we'll introduce it to the Wikidot.com service. I calculated that indexing all the sites would take about 3 days! But searching is done in less than 1 second (using the Java program).

I'm looking for your comment on the features. Especially if you've tried them yourself!

by Gabrys