Svn2github problems

12 Dec 2014 18:31

I'm the one behind the svn2github.com service. The service was started a few years ago to help me and my team start a PHP project that we wanted to host in Git. The PHP libraries we wanted to use were hosted in Subversion. Given the Composer was not too popular we decided to put the links to the libs into the repo using git submodules.

Given how basic the git-svn is, and how easy it is to have a Git repo in GitHub, we thought, hey, there must be an automatic mirroring tool to clone publicly available SVN repos to GitHub. The service would be probably called svn2github.com.

Such service was not there and we struggled looking for one, but again, since this was so easy to set up, I decided that I can do it myself! So I did.

Svn2github.com operated with only minor administration from my side, but at some point I realized there are over 500 repos mirrored by it, which was fine.

But recently the server svn2github is hosted on started to experience some problems, mostly with I/O throughput and I connected them directly to the operation of the svn2github processes.

Basically with so many repositories stored at the disk, the periodical task to do just git svn rebase and git push if there were any changes might be challenging due to just the number of IO operations needed to accomplish this. Also at some point the (most important) data just stops fitting in cache and the disk IO needs to be requested each time an FS operation is needed.

The problem became apparent to me, because of the other services that I run on the same hardware, mainly the database. It started to be terribly slow meaning all the other apps would take forever to do even a basic task.

I needed to suspend the svn2github operation to let the more important services continue to run, but I planned to bring this useful service back to life. As often in such cases I want to add more features while doing that and make the updates more clever, so they don't consume so much server "life" as they were.

The first step though is restarting svn2github, which means you can now add more SVN repos to be mirrored to GitHub and the repos will be synchronized with one small exception. Any repository that contains more than 2000 files (including the .git files) will not be automatically updated.

I'll update the GitHub descriptions of those "paused" mirrors and if you want them to be "resumed", I'll ask you to contact me and let me know. This way the service will continue to work for the small repos (which are the majority), which don't cause so much trouble for the machine, while the big repos would be only updated when requested (I assume most of them were needed "once" and now no-one really needs them in place).

Happy SVN mirroring! See you on svn2github.com!

UPDATE: Some svn2github stats

To give a notion of scale this project is at here are some stats:

Repositories with less than 2000 files each (including the .git files):

Number of them: 482
Total size of them on disk: 19G
Total number of files on disk: 954321
The biggest one: 635M (DevIL)
The smallest one: just 208k (aszip)

Repositiories with over 2000 files in each:

Number of them: 231
The total size: 308G (took 133m58.527s to compute that)
The biggest one: 42G (testingazuan)

Comments: 13

Working around a buggy sound card

02 Dec 2014 23:34

I found the laptop I bought recently has a weird thing with its soundcard. It plays music OK, it also allows you to change the mixer levels, nothing special, but when you try changing mixer setting while the music is playing the music starts getting choppy and the process handling the mixer tend to freeze. Also you cannot pause the music, sometimes it repeats the same sample over and over and it gets very annoying.

My approach to this issue was to use PulseAudio software mixer capabilities, so the hardware mixer is never adjusted (other than for the initial setup).

Step 1. Getting PulseAudio to use the software mixer

I found no good documentation on this other than PulseAudio uses paths to convert the desired volume to the hardware mixer settings and if it lacks the ability to do so it would turn the volume up/down using software. Reading between the lines I realized if I remove the paths I will get what I want. Just do this and restart pulseaudio and you'll have PulseAudio doing all the mixing in software:

sudo mv /usr/share/pulseaudio/alsa-mixer/paths /usr/share/pulseaudio/alsa-mixer/paths.bak

Verifying it works

Open two terminal windows. In each of them open alsamixer. Both should display very simple controls and say Card: PulseAudio. Use F6 to switch to the real card in one of the windows. Usually the cards have plethora of controls, so should yours.

Now change the output volume in the alsamixer still saying "Card: PulseAudio". If the volume reported by the other alsamixer does not change, it works as expected!

If you now rename the paths directory back and restart the pulseaudio process, updating the PulseAudio volume should update the real device volume as well.

Step 2. Integrating this with XFCE (or other DE)

The way xfce4 handles the volume is two-fold:

  • There's a xfce4-volumed daemon. It only controls the key bindings. It binds volume up, volume down and mute keys to control the real device controls
  • There's a panel applet called "Audio Mixer" that shows the current volume level and lets you change the volume as well.

Both component are optional and both seem to not work with PulseAudio (even through PulseAudio ALSA emulation which tricks alsamixer for instance).

We're going to overcome this problem by replacing these components with two other programs.

pa-applet

pa-applet will replace the "Audio Mixer" applet. The difference is the pa-applet docks to the systray, so is not XFCE-specific. The downside is we don't have control over the specific position of the icon, all we know is it will appear in the systray (so don't forget to have the systray applet added to the panel).

In order to install this software you need to clone the source, and run ./autogen.sh, ./configure, make, sudo make install. You will get information about deps missing, for instance:

configure: error: Package requirements (glib-2.0) were not met:

No package 'glib-2.0' found

You need to find the proper *-dev package for each unmet dependency (for this one it will be libglib2.0-dev).

I came across one additional problem. The software seems not updated after some GTK+ 3 updates and it uses a deprecated function. Because the code is configured in a way that every warning stops the compilation, it wouldn't let you compile the code unless you open the src/Makefile file and then find and remove the -Werror flag from it.

xfce4-volumed-pulse

This part will take care of binding the media keys to the pulse audio controls (rather than to the sound card controls).

You need to download the tar package, untar it, ./configure, make, sudo make install. It also requires quite a few libs to be installed, but I didn't found any issues when compiling.

Hooking this up together

Since I didn't need the original xfce4-volume anymore I decided to remove it:

apt-get remove xfce4-volumed

But because XFCE will try to launch this anyway on each start, I decided to take advantage of it and replace it with a script:

#!/bin/bash

/usr/local/bin/xfce4-volumed-pulse
/usr/local/bin/pa-applet

Add the executable bit, restart the laptop and all should be working correctly at this moment.

Setting the hw mixer controls

Since PulseAudio won't update the hw controls after you follow up this little howto, it's quite important to set all the important channels to 100% (or other number up to your taste). The way to update the hw controls was described before.

Open the terminal, type alsamixer and hit F6. This will give the list of ALSA devices besides the PulseAudio emulation, your real sound cards should display as well. Choose the right one and you should be given access to tweak the real card controls.

Note if you removed the PulseAudio paths, the PulseAudio controls wouldn't update when you tweak the hw controls.

Comments: 0

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License