Hacking pympd and Last.fm

26 Sep 2008 22:36

Hello again,

My girlfriend was very unhappy with me not having my played songs submitted to Last.fm social music revolution portal.

That was because used to use Music Player Daemon and its various clients. Most of the clients don't implement the AudioScrobbler protocol, but as a matter of fact this is not needed, because there can be a separate MPD client meant just to submit the info to last.fm, running in parallel to one actually playing music.

I used to use scmpc for this reason, but since I bought my new laptop and migrated to Ubuntu I quit it — because the application was not in their repo.

Today I decided to find some short python-based implementation of audioscrobbler and enrich one of MPD clients with the Last.fm integration. I found pympd really good — nice looking, having plugin architecture, clean and simple. So I decided it to be my new favorite MPD client. Then I quick-hacked some random plugin and created a new one including almost 100% source from python-scrobbler project. The plugin:

• sends "now playing" info to Last.fm on each track-change and loading of plugin
• sends "song played" info to Last.fm on track-change and plugin unload event if song was listened at least to the half of its total length and is longer than 30 seconds.

Now the Last.fm user and password are hardcoded, but I hope to create a quick-and-dirty configuration window for it.

I had some problems with time convertion. The python-scrobbler sources suggest using datetime.utcnow() method while actually using datetime.now() is giving the right results.

You can check my Last.fm records:

• July — submitted by scmpc from Gentoo — the old computer
• first records in October — submitted with MY PLUGIN
• two hours gap — before fixing UTC issue, the songs were submitted as played two hours earlier (timezone difference)

So it seems working!

It's a pretty cool Python day today.

Comments: 4

Working On TagFs

26 Sep 2008 19:23

Today I spent another hour or two on working on tagfs (described in TagFs Idea). It's now pretty much usable. You can now:

• browse files with the tag-directory mapping
  • directory tag1/tag2/tag3 is the same as tag2/tag3/tag1 or tag3/tag1/tag2
• read/write files
  • file tag1/tag2/tag3/file is the same as tag2/tag3/tag1/file AND even tag2/tag1,tag3,file
  • tags are not only allowed as directory names, but also as comma-separated file prefix
• create new files
  • echo Some Content > tag1/tag2/some_file
  • echo Some Other Content > tag1/tag2,some_file
• files have all their "real" properties (owner, group, modification time — this is stored on the back-end filesystem)

Things to do yet:

• forbid creating a file with tags rendering some directory with name of existing file
  • example: you have tag1/great file
  • then you create a tag1/tag2,great,people.txt file
  • this way, you would have a great directory showing in the tag1 directory, because there is a file having both tag1 and great
  • you also have great file there, so you end up with a file and a directory (tag) of the same name
• scan and list files by their properties
  • file type — PDF, JPEG, HTML, …
  • EXIF tags for JPEG — date taken, camera info
  • ID3 tags for MP3 — artist, title, album
• improve directory listing
  • include SOME files in sub-directories if the sub-directories consist of small number of files
  • as a result we get a easily-browsable repository of files
• do some marketing
  • I would like to serve files from such a file system with FTP or Apache to let people feel the system

I think this experiment is really worth working on this.

In case you want to test the FS, drop me a note or comment.

Comments: 1

Interesting Shiny Idea Of Michał

25 Sep 2008 18:49

Now, having Wikidot sending just every form with AJAX and being in early process of planning and developing of Wikidot 2, Michał suggested very nice way to enrich user interface and actually user interaction with modern wiki system.

Two interfaces

Basic HTML

First of all we want to have very simple "basic" interface. This mean serving just the HTML content without any dynamic mechanisms in it. More content — less interface — better for SEO and search bots.

On such a HTML page, there would be one JavaScript file included and responsible for determining if a user is logged in and capable of running the "rich" interface. If this is all true the script redirects the user to the one-page navigation system showing the same content as redirected from.

Rich JavaScript Interface

Such an interface would be one page with some static elements like a top-bar showing who's logged in, new messages and quick link to compose a new one, edit button and probably showing your status on particular site (member/moderator/admin) and link to admin:manage if applicable.

There would go a normal content — the wiki pages. Clicking on any link you would have the UI to load the new page for you. No browser reload thing goes here.

If some page takes really long to load — you have your top-bar still there and can write a PM to someone for example. No need to wait till the browser gets the new content.

More pros:

• simpler CSS theme designing — don't worry about the My Account menu styling — it's out of the desing actually
• back/forward buttons compliance — we don't break things like history and back/forward navigation
• better consistency — users find Wikidot functional elements at the same places for every wiki — despite of theirs custom CSS
• edit button — if you are allowed to edit — always available at top

UPDATE: if you copy and send some URL from rich interface to your friend not logged in to Wikidot — the interface would check if the user is logged in and if not — redirect them back to the basic interface. So we have full URL translation between rich and basic interfaces.

More to come in Wikidot 2

Menus created like now (as the unordered list) would render with JavaScript and not pure-CSS like it happens now. This would allow for better look and feel of them.

A great WYSIWYG editor is coming there. Clicking the edit button at the top your whole page changes into the areas you can edit. You are given an additional top-bar with some formatting functions. In general we want to have the most features we have now — accessible by mouse, which means no need to write (and learn) WikiSyntax. That would be cool.

An optional switch for WikiMasters would allow them to write the pages like now. The difference is, that we would like to change the internal storage format to based on XML for better parsing and converting. So probably we would accept some subset of XHTML with addition of some nifty Wikidot tags:

XHTML formatting tags + Wikidot-specific XML tags = WikiML

another option is that we have a totally different XML-based language to clearly distinguish between the source and the output of wiki processing.

The more I think about, the more I am for the XTML - something + something model.

Summary

This post gives you some basic idea about how Wikidot 2 will improve user interaction. As Wikidot 2 is at stage of planning and early development much can change, but what remains the same — we want to make it absolutely the best Wiki system on the planet.

And remember, probably PRO users will get the Wikidot 2 preview 3 months earlier!

Comments: 5

Problems With Internet Explorer

22 Sep 2008 17:16

Do you ever wondered how much time Wikidot team spends on working on fixing things to let them work in Internet Explorer? A LOT!

• Internet Explorer is hardly following any standard.
• Microsoft claims to have right to produce their proprietary standards.
• They don't even publish them, so working with Microsoft application is just like magic — the bad one.
• They don't supply with any tool for Internet Explorer that would uncover the strange things that happen.

A few examples:

Cross-domain cookies problem

(This "feature" caused custom domains and private wiki files to not function properly in IE).

It seemed that cookies are not sent with a request, when requesting an image or script from other domain than the master page. The same happened when requesting a HTML file from other domain within an iframe.

Then, we discovered, that actually some of cookies are sent. But what the hell happens, that the WIKIDOT_SESSION_ID cookie is not sent? "It's the SESSION in the name of cookie — Michał said — we've done a little experiment and the name of cookie seemed NOT to be a problem. So, it is something else that stops Explorer from sending cookies.

As we cannot easily see what cookies IE sends (Oh dear! No tool like Firebug for IE), we had to create server-side scripts, that printed out the cookie table. As some of situation we analyzed were JavaScript related (the browser requests JavaScript code), we had to write also a server-side script that alerts the cookie table with JavaScript. That was frustrating as hell.

OK. Now we knew exactly what cookies were sent to server and which weren't. But why? Here's where Firebug and Webdeveloper Toolbar for Firefox comes. I printed the cookie table in Firefox and noticed that the cookies that are sent were cookies with no explicit expiration time — i.e. deleted when the browser quits. The cookies that were not sent had some expiration time set.

Without tools for Firefox this would not be so easy to determine what's the difference that makes some of cookies unusable.

NOTE: some of you may wonder if there is any point in not sending some cookies when issuing cross-domain requests. After hours of analysis — we don't see any. And The Rest Of Browsers send all the cookies.

Cross-domain cookies issue continued

Wonder if it's possible for Internet Explorer to set a cookie from an iframe with different domain?

Content-type ignoring

Internet Explorer ignores Content-type, which is completely non-standard and not the case of ANY OTHER browser.

Background

Suppose we know a file is a PNG image, or GIF. This is why we send the Content-type HTTP header before the content of some file requested by browser.

Content-type is also known as a MIME type. Examples: MIME type of HTML files is text/html, PNG images: image/png.

Internet Explorer Behavior

Internet Explorer, once downloaded a file, do a so-called "content sniffing" and decides by itself what's the type of a file. This means it can badly detect a file type and display HTML as a text or vice-versa.

Content-sniffing and JavaScript

• It is possible to attach HTML files to Wikidot
• uploaded file can be iframed within any Wikidot page
• HTML can contain JavaScript
• JavaScript is run and has the access to any properties and cookies of pages that runs the same domain as the JavaScript itself

How to disable this

We just set the Content-type header to text/plain for HTML files, and this makes browsers display the HTML as the source, so JavaScript is displayed and not run.

This is just the case I mentioned: "we know, this file is text file (we want it to be treated like text file), so we send the hint on this".

Unfortunately IE knows better (by ignoring Content-type and doing content-sniffing) and allows JavaScript to be run, fetch the session cookie and to export this to external service. This allows the external service to use your session — i.e. use Wikidot AS YOU.

The only way we know to disable this for even for IE

Move ALL USER-UPLOADED files to other domain. Then JavaScript has limited access to the original domain and can't grab cookies.

Why do we need to move ALL files? Actually we need to move only files that IE detects as HTML, but how to hell know it? IE is closed source and the mechanism itself is never published. We hate you Microsoft!

Tricky CSS

Internet Explorer completely sucks when it comes to CSS support. Even the shiny IE7 is a complete mess. I had to completely rewrite the CSS of the Green Fresh theme because IE6 and IE7 didn't display the search box at all. No need to say every other browser I tested displayed it.

UPDATE: Expires header

Expires and Cache-control HTTP headers are used to control the cache behavior of browser. It lets us decide if a page should be refreshed every time accessed or can be cached for some time to speed up things.

We use Expires headers for example for user-uploaded files. We allow caching them for 2 hours or so. But unfortunately Internet Explorer caches badly PDF (and possibly other) files. So when we allow it to cache it and user accesses it second time, the file is taken from IE cache and displayed as a total mess or not displayed at all!

Because of this BUG we disabled Expires header for IE for user-uploaded files including custom theme images and stylesheets. This leads to IE fetching the files that never changes again and again.

I'm pretty sure Wikidot is not the only service to disable Expires header for IE. That's one of reasons why browsing the Internet with IE is slower than with Firefox for example.

Summary

The popularity of Internet Explorer makes web developers spent extra time on making sure the service works "even on IE". Normally, once you get thing working in Firefox, it's not a big deal to make it work on:

• Safari
• Opera
• Konqueror
• Google Chrome
• Epiphany
• Galeon
• Camino
• any browser based on Gecko or WebKit (also KHTML) engine

Wonder why? Because they all follow W3C standards.

The Internet Explorer is the only one popular browser left that does not follow standards.

SHAME ON YOU MICROSOFT!

UPDATE: The Solution?

There are plugins for Internet Explorer. Someone could make a plugin that renders on the full page size and uses Gecko (the engine of Firefox) to display a page.

Technically — this would mean even Internet Explorer users use Firefox engine, so from this point you could support only standard-implementing engines. For the IE user — it would require them to install an IE plugin to use your service properly.

Comments: 4