Using Caps Key As Second Tab

1261576185|%e %B %Y

As we all know, Caps Lock key is useless and annoying, so some folks change it to behave as third Control key. I prefer using two Controls I already have, so I mapped the Caps Lock key to Tab (which I use very often).

$ xmodmap -
remove Lock = Caps_Lock
keycode 0x42 = Tab
<hit Ctrl-D now>

This works till you logout. If it does work, you can make this persistent by adding those two lines to your empty or missing ~/.Xmodmap file:

remove Lock = Caps_Lock
keycode 0x42 = Tab

If this doesn't work for you, it's possible your window manager doesn't take care of this file, but if you have a way to execute commands at each start of GNOME/KDE/whatever, just add the following command to those run it startup sequence:

xmodmap ~/.Xmodmap

Comments: 2, Rating: 0

Cleaning Up

1253205773|%e %B %Y

Some of you, following Wikidot code on GitHub may see it's nicely split into templates, php, web and conf directories. But this is the first impression.

Maintaining Wikidot is a bit more complex, because, files uploaded to sites are located in web, side to side with some static Wikidot php and javascript files. Also for historical reasons, there are web/files--common and web/files--local directories, which maps to /common--* and /local--* URLs and in fact, the files--local is never served directly by the web server (need to check permissions first).

Also some time ago, we made static files versioned, so that we can apply more aggressive HTTP caching to them (reducing average page load time) and still be able to fix bugs on them without waiting a few days till the cache expire. In current model, URL to static file contains version hash, this may be for example: http://static.wikidot.com/v--b44e0ce810ee/common--javascript/WIKIDOT.js (notice the b44e0ce810ee). The whole static.wikidot.com is now hosted on Amazon's CloudFront, which means you get static Wikidot files from a server nearby your location and not always from USA.

This all become quite complicated, so we decided to make things really clear and simple in the source code. The primary rule: make the source code (updatable from git) separate from files uploaded by users and generated by Wikidot. Second rule: make files that are automatically generated during installation (not in the runtime) separate from persistent files (like the uploaded by users) and from source code.

And at the end there needs to be some place for logs and a place for temporary data (we need this to generate some random cool stuff, but after generating them, files are deleted).

So we end up with something like this:

• WIKIDOT_ROOT
  • data/
    • avatars/ — user avatars
    • sites/ — site files (both generated thumbnails and uploaded files)
  • generated/
    • static/ — generated static files. This dir can be server directly by a fast non-PHP webserver for static.wikidot.com in case we don't want CloudFront anymore
  • tmp/ — temporary files including Smarty compiled versions of templates. Content of this dir can be safely removed
  • logs/ — Wikidot logs
  • everything else — comes from git and is unchangeable by application

Application needs write-access to data, tmp and logs. Generated dir needs write access to one installing or upgrading application.

Wikidot persistent data is now ONLY database and data/ directory, so it's easy to backup and restore the application (if you have enough time to make full backup of this).

There is still one exception to this nice schema which is php/db/base directory, which is autogenerated during installation from XML database definition files, but the cleaning is not over, I still work on this.

Nice thing about this work is that it does not need a lot of code changing, because directory paths are usually stored in one (max two) places in application, so this kind of totally reorganizing directory structure does not break things. As such, it is very very worth doing it. In the end we get clean internal structure of files and it's clear which files you can safely remove, which you can restore from git (and thus you can experiment a little on them — in case of crash, just re-download application), which are "state" of the Wikidot and where to look for logs.

This all is also very important, because we aim to make current Wikidot.com source open and as such we want it to be a nice code.

Comments: 0, Rating: 0

Wikidot Crashes

1251378203|%e %B %Y

Last night we made Wikidot online again after a great crash.

Wikidot was down for about 12 hours and the time it was down was full of work for us. We got a few things that could be broken starting from recent changes of the Wikidot software, hardware failure, high load-related kernel bugs or limitations to connection number or maximum possible number or file descriptors.

The problem is Wikidot kind of worked, so some people had their sites loading, some other not and getting "500" errors. We didn't want to stop it, but at some point Wikidot was completely unusable. We switched the database to the other machine, but this was not the solution, then we switched all Wikidot traffic to the machine, still no good, we switched the software to some previous version, but this still seemed bad.

Finally we worked out, there was a site, that had so big traffic, that it killed anything else (and itself as well). When we temporarily disabled it, the whole Wikidot started to work nicely again. Then Michał made some improvements for the high traffic site serving and the situation is stable again.

In the middle of everything, we had huge problems with our hosting company and their service called Portable IP addresses. It seems that switching DNS is much more reliable that using Portable IPs that took hours to switch (and were supposed to take seconds to switch)! DNS switching time was 15 minutes.

We learned a lot from the situation. Hardware upgrade postponed from really long time needs to be done quite quickly. We need more servers, to see which element breaks. For example if database server has high load, we know we need to tune database settings. If we have all on one massive server and one brick on it crashes, it usually causes all the server overloaded and this causes other bricks to crash as well, so it's hard too tell what the real problem is.

Another thing is that we see our users want information on what happens. It's bad when Wikidot crashes, but it's even worse, when it crashes and they have no information about this.

So, the next time a similar disaster happens we'll update on each technical detail possible, to let you know, that we know it's broken and we work hard to fix it. Other thing is, we plan having more fail-over servers in case something dies.

Thank you all for using Wikidot, it's a great pleasure working (and fixing things) for you!

Comments: 3, Rating: 1

YAML and PHP

1248980616|%e %B %Y

There are 3 main YAML implementations for PHP:

• Syck (native C library bindings to PHP)
• Symphony YAML (pure PHP)
• Spyc (pure PHP again)

This is the comparison.

What the hell is YAML

Have you heard about XML or JSON? YAML is similarly to JSON and XML a way to store (read and write) structured data like arrays (a.k.a. lists), dictionaries (a.k.a. hash maps) and atomic values like strings and numbers. The structures can be nested, to form a definition of near-real-life objects, for example:

---
Piotr Gabryjeluk:
  company: Wikidot Inc.
  university: Nicolaus Copernicus University, Toruń, Poland
  lives_in: Toruń, Poland
  hobbies:
  - basketball
  - playing the guitar

Which translates to PHP:

<?php
$data = array('Piotr Gabryjeluk' => array(
  'company' => 'Wikidot Inc.',
  'university' => 'Nicolaus Copernicus University, Toruń, Poland',
  'lives_in' => 'Toruń, Poland',
  'hobbies' => array('basketball', 'playing the guitar')
));

So you see YAML is quite nice even when you need to write it yourself.

YAML has its specification (see http://yaml.org), so once we have standard YAML parser and standard YAML dumper we can send arrays from one machine to another and the result should be the same array as was sent.

PHP

So let's see what are the choices if you want to play with YAML in PHP.

Syck

This is the fastest and the most complete YAML dumper and loader library available. This is binding to C library and this is available in PEAR. It is also available as regular package in Ubuntu repository, so install it by simple:

aptitude install php5-syck

In some shared hosting environment this could be a problem, so you need a pure PHP solution.

Spyc

This was the first PHP YAML implementation I saw. It is both dumper and loader and it seemed to work fine, but then I found some bugs, that stopped me from using it as the base and only YAML loader and dumper for Wikidot.

This one has really nice thing, which is nice when you want your users to enter YAML to define things (like we do for forms). It is quite forgiving when it comes to the syntax and ignores things that don't fit and still parses the rest.

Unfortunately as I stated before Spyc dumper so, when you first dump an array and then load it with Spyc you get something different (for example multiple new-lines are treated as one). Not good. Also as a loader it does not fully understand the full YAML specification (which is quite huge BTW).

Symphony YAML

This one is pure-PHP as well, so you don't need special rights, to use it on a PHP-enabled machine.

It's loader does not understand full YAML specification, so for example you can't load documents dumped by Syck. Dumper is good.

Summary

Verdict: loader

Syck is the winner in loading YAML. If you cannot use Syck, use Symphony YAML. If you need to parse user input (which should be human readable/writable similar to YAML), use Spyc.

Actually, this is nice combination for loading:

<?php
try {
    // if syck is available use it
    if (extension_loaded('syck')) {
        return syck_load($string);
    }
    // if not, use the symfony YAML parser
    $yaml = new sfYamlParser();
    return $yaml->parse($string);
} catch (Exception $e) {
    // if YAML document is not correct,
    return Spyc::YAMLLoadString($string);
}

This way, you have the fastest library used if possible, then the best pure-PHP, and if it fails in a way, that document was badly written (by human being for example), you fall-back to Spyc.

Verdict: dumper

In my opinion Symphony YAML dumper is the best from the three in terms of usability, portability and interoperability, because its output can be read by both itself and Spyc.

However, if you dump YAML often, use (hell faster) Syck for both loading and dumping. The generated YAML won't be readable by Symphony YAML or Spyc, but this is because they don't follow the specification (so not Syck's problem in fact).

Also note, that any valid JSON dumper output is readable by standard YAML 1.2 loaders, because JSON is a subset of YAML 1.2. So if using for data exchange (and not for talking to human) any fast JSON dumper can be used.

Comments: 0, Rating: 0