quake@vaio /home/quake/files $ http_server.sh 8000 

And voila, the files in directory /home/quake/files are accessible via a web browser (or a wget command) on every computer in local network.

Some time ago I learned I can achieve the same effect using simple Python one-liner:

 python -m SimpleHTTPServer 

This work for standard Python 2.x installations, for Python 3.x even simpler:

 python -m http.server 

No need for custom scripts, netcat or other fancy stuff, you need just a standard Python installation and it works ;-).

But recently I faced a challenge of copying many gigabytes of files over network. Copying files over SSH was too slow (data is copied from an ARM machine, not really blazing fast at encryption stuff). I tried to copy files over FTP, but I failed at configuring read-write FTP server in limited time. I wanted to avoid configuring other fancy file servers like Samba/CIFS. I could use NFS, which is both simple to configure and fast enough, but I decided to go more fancy.

I took my old http_server.sh, tweaked it a bit (including replacing lame cat "$file" | wc -c - with stat -c %s to determine file size and replacing gawk with awk in the script) and then created a specialised version of it: tar_server.sh.

tar_server.sh is a HTTP server based on http_server.sh that shows you list of directories inside of directory it was run from and allows you to download the directories as tar files. It does the taring on the fly, so you don't waste the disk space.

It's as simple as:

 quake@vaio /home/quake/files $ tar_server.sh 8000 

Then you can see the list of tar files to download at http://your_ip:8000/ . Suppose you have a directory /home/quake/files/backup. You can download it on some other machine using:

 quake@other-machine /home/someuser/files $ wget http://your_ip:8000/backup.tar 

Or to unpack on the fly:

 quake@other-machine /home/someuser/files $ wget http://your_ip:8000/backup.tar -O - | tar -x 

This way you can mirror part of your filesystem with almost no dedicated tools. The script is quite OS-independent and requires only netcat, awk, tar and stat commands, that are likely to found in any Unix-like systems.

Also the script proves BASH is still very useful tool and adapting simple scripts is easy and FUN :-).

Remember, the scripts (just like original version) can handle only one client at once, so if you want to do parallel stuff, you need to launch more of them on different port each.

Here are the scripts:

http_server.sh:

 #!/bin/bash function debug { local severity="$1" shift local message="$@" echo -n "`date -u`" 1>&2 echo -ne '\t' 1>&2 echo -n "$severity" 1>&2 echo -ne '\t' 1>&2 echo "$message" 1>&2 } function fix_path { echo -n "$1" | head -n 1 | sed 's|^[/.]*||' | sed 's|/\.*|/|g' } function serve_dir { local dir="`fix_path "$1"`" if [ "$dir" = "" ]; then dir="./" fi echo 'HTTP/1.1 200 OK' echo 'Content-type: text/html;charset=UTF-8' echo echo LISTING "$dir" echo '<br/>' ls -p "$dir" | sed -e 's|^\(.*\)$|<a href="/'"$dir"'\1">\1</a><br/>|' } function serve_file { local file="`fix_path "$1"`" echo 'HTTP/1.1 200 OK' echo 'Content-type: application/x-download-this' echo 'Content-length: '"`stat -c %s "$file"`" echo debug INFO serving file "$file" cat "$file" } function process { local url="`awk '{print $2}' | head -n 1`" case "$url" in */) debug INFO Processing "$url" as dir serve_dir "$url" break ;; *) debug INFO Processing "$url" as file serve_file "$url" ;; esac } function serve { local port="$1" local sin="$2" local sout="$3" while debug INFO Running nc; do nc -l -p "$port" < "$sin" > "$sout" & pid="$!" debug INFO Server PID: "$pid" trap cleanup SIGINT head -n 1 "$sout" | process > "$sin" trap - SIGINT debug INFO Killing nc kill "$pid" done debug INFO Quiting server } function cleanup { debug INFO Caught signal, quitting... rm -Rf "$tmp_dir" exit } tmp_dir="`mktemp -d -t http_server.XXXXXXXXXX`" sin="$tmp_dir"/in sout="$tmp_dir"/out pid=0 port="$1" mkfifo "$sin" mkfifo "$sout" debug INFO Starting server on port "$port" serve "$port" "$sin" "$sout" 

tar_server.sh:

 #!/bin/bash function debug { local severity="$1" shift local message="$@" echo -n "`date -u`" 1>&2 echo -ne '\t' 1>&2 echo -n "$severity" 1>&2 echo -ne '\t' 1>&2 echo "$message" 1>&2 } function fix_path { echo -n "$1" | head -n 1 | sed 's|^[/.]*||' | sed 's|/\.*|/|g' } function serve_dir { echo 'HTTP/1.1 200 OK' echo 'Content-type: text/html;charset=UTF-8' echo find -type d -mindepth 1 -maxdepth 1 | sed -e 's|^\./||' -e 's|^\(.*\)$|<a href="/\1.tar">\1.tar</a><br/>|' } function serve_file { echo 'HTTP/1.1 200 OK' local file="`fix_path "$1"`" local dir="`echo "$file" | sed 's/\.tar$//'`" debug INFO serving tarred dir "$dir" echo 'Content-type: application/x-tar' echo tar -c "$dir" } function process { local url="`awk '{print $2}' | head -n 1`" case "$url" in /) debug INFO Processing "$url" as dir serve_dir "$url" break ;; *) debug INFO Processing "$url" as file serve_file "$url" ;; esac } function serve { local port="$1" local sin="$2" local sout="$3" while debug INFO Running nc; do nc -l -p "$port" < "$sin" > "$sout" & pid="$!" debug INFO Server PID: "$pid" trap cleanup SIGINT head -n 1 "$sout" | process > "$sin" trap - SIGINT debug INFO Killing nc kill "$pid" done debug INFO Quiting server } function cleanup { debug INFO Caught signal, quitting... rm -Rf "$tmp_dir" exit } tmp_dir="`mktemp -d -t http_server.XXXXXXXXXX`" sin="$tmp_dir"/in sout="$tmp_dir"/out pid=0 port="$1" mkfifo "$sin" mkfifo "$sout" debug INFO Starting server on port "$port" serve "$port" "$sin" "$sout" 

by Gabrys

This can be the option, when you send a bunch of files to an Internet service, that is generally fast, but the connection sequence is quite slow, so uploading 100 files one after one causes the script to wait 100 times to quickly upload a file.

Other situation could be when you have multi-core machine, for example you have eight processing units, but use only one in your script, and you have a bunch of files to compile or to process in some CPU-expensive manner.

We'll use only BASH to smartly parallelize the tasks and speed up the slow part of your script.

First of all you need to know how many jobs in parallel you want (if you have 8 cores and CPU-expensive part of script, having more than 8 jobs does not help, probably a number between 4 and 8 will do best in this case).

 #!/bin/bash PROC_NUM=4 

Generally, we'll ensure, than no more than PROC_NUM processes are forked into background and run another task. If there are PROC_NUM processes running in the background we'll wait a (fraction of) second and check again.

 #!/bin/bash PROC_NUM=4 function run_task() { # task to run # can be more than one-line # can take parameters $1, $2, ... } function run_parallel() { while [ `jobs | grep Running | wc -l` -ge $PROC_NUM ]; do sleep 0.25 done run_task "$@" & } 

run_task "$@" passes all the parameters passed to run_parallel to run_task. You can use "$@" in run_task to pass all the parameters to external command! The "$@" is the best choice when you have spaces, dollars and other special characters in parameters. It doesn't transform anything, it's completely safe (probably the only short way to pass all the parameters).

There are only two things left: invoking the run_parallel and synchronizing the tasks — you need to know when ALL the tasks ended, right?

 #!/bin/bash PROC_NUM=4 function run_task() { # task to run # can be more than one-line # can take parameters $1, $2, ... } function run_parallel() { while [ `jobs | grep Running | wc -l` -ge $PROC_NUM ]; do sleep 0.25 done run_task "$@" & } function end_parallel() { while [ `jobs | grep Running | wc -l` -gt 0 ]; do sleep 0.25 done } # script content cd /some/where/you/want # now the parallel operations # for example in some while find | while read file; do run_parallel "$file" done # now you want to continue when ALL parallel tasks ended end_parallel # the linear script code again cd /some/where/else make something 

That's all! Though, there is a different approach to this:

 #!/bin/bash function parallel() { local PROC_NUM="$1" local SLEEP_TIME="$2" shift; shift while [ `jobs | grep Running | wc -l` -ge $PROC_NUM ]; do sleep $SLEEP_TIME done "$@" & } 

This function acts as a wrapper to a non-parallel command and runs it in the background assuring that no more than PROC_NUM processes run at once. If there are PROC_NUM processes running in the background, the wrapper waits SLEEP_TIME to re-check the number of background jobs.

Invoking:

 parallel PROC_NUM SLEEP_TIME /usr/bin/some-command arguments ... 

so

 parallel 4 0.5 ls -R /tmp 

means: run ls -R /tmp in the background if there is no more than 3 processes already run in the background. Otherwise wait 0.5 seconds and try again. Then run ls -R /tmp if there is no more than 3 processes already run in the background. Otherwise wait 0.5 seconds and try again. Then run ls -R /tmp if …

Quite nice, isn't it?

by Gabrys

 echo $(($(printf '%d' "'`head -c 1 /dev/urandom | base64 | tr A D`")%4)) 

This prints a random number from 0 to 3. How it works?

Firstly the command below prints one random byte using special Unix /dev/urandom device:

 head -c 1 /dev/urandom 

Piping this to | base64 gives us base64 representation of the byte.

Base64 is a method of encoding 8-bit data as human-readable strings using only 64 visible characters (letters, digits and some !@ and stuff). Whitespace is always ignored when reading base64 string. The standard is really nice, because you can for example read a base64-encoded file with a phone or send a printout of it with a classic mail. Because all the characters are human-readable, one can enter them and decode the original message. The format is used widely for sending emails.

It is important for us, that running base64 on a random byte gives us 2 bytes that are only "normal" characters, ie don't have any special meaning in any context (like some white characters may have).

Now comes the tricky part:

 printf '%d' "'`head -c 1 /dev/urandom | base64`" 

printf '%d' "'a" would give us 97 — this is the ASCII code of letter a. Remember we get two bytes of data after base64? No problem, printf cares in this case only about the first character.

As you may notice the output of this is one of these:

• 43
• a number between 47 and 57
• a number between 65 and 90
• a number between 97 and 122

This gives us 64 possibilities. Great! 6 bits is just the same as 64.

This is when we get to BASH arithmetics. This is how it works:

 echo $((7*12)) 

This should print 84 obviously. Knowing that enclosing a string in $( and ) causes BASH to run the enclosed command and return its results (just like using the backticks: `command`) this is everything.

The % means modulo in BASH arithmetic (just like in C, Java, Python, PHP, …), so this:

 echo $(($(printf '%d' "'`head -c 1 /dev/urandom | base64`")%4)) 

prints the number we generated (one of 43, 47…57, 65…90, 97…122) modulo 4. This needs to be 0, 1, 2 or 3.

Let's now check what the probability of receiving each of the digits.

Suppose /dev/urandom prints every possible byte with equal probability. We'll now analyze each byte (possibly) generated by /dev/urandom, it's base64 representation, ASCII code of first byte of the representation and the modulo 4 of it. We'll use a table for it:

(Some of the symbols in the first column may appear not visible or otherwise look strange. This is normal, there are many characters in ASCII that has (or had) some special meaning.)

Let's count have many 0s, 1s, 2s and 3s did we get:

This is not ideal, because you get statistically slightly more 1s than 0s, but if you don't care too much, this is all!

If you care however, here is some solution. We need to get 4 results that give us 1 and convert them to some that gives 0:

 echo $(($(printf '%d' "'`head -c 1 /dev/urandom | base64 | tr A D`")%4)) 

Notice the tr A D. This changes each A to D in the base64 output. Thus four rows of table above (for 0 to 4 codes) should behave like the 12 to 15 codes, thus giving 0 at the end instead of 1.

We're done. Let's enclose the procedure in function clause and create a sample code that tells us how many 0s, 1s, 2s and 3s where hit within 100 shots.

 #!/bin/bash function random() { echo $(($(printf '%d' "'`head -c 1 /dev/urandom | base64 | tr A D`")%4)) } ret=`i=0; time while [ $i -lt 100 ]; do random ; i=$((i+1)); done` echo -n '0: ' ; echo "$ret" | grep 0 | wc -l echo -n '1: ' ; echo "$ret" | grep 1 | wc -l echo -n '2: ' ; echo "$ret" | grep 2 | wc -l echo -n '3: ' ; echo "$ret" | grep 3 | wc -l 

The program also shows how much time did it take to generate the numbers. Adjust the 100 to your needs ;).

UPDATE: having this article posted on reddit programming gave me actually much much better ways of doing a random function (see in comments). Thank you for your replies!

by Gabrys

Zatem, po pierwsze: iPhone rządzi, bo ma basha.

Po drugie: devcamp rządził :]. Chłopaki z www.socjum.pl też są spoko i zapraszają do Krakowa na imprezę, jaką długo spamiętamy.

Po trzecie: z pomocą Zająca odpaliłem na moim lapie Compiza w konfiguracji Dual Head na karcie graficznej Intela (a jak wszyscy wiedzą mam dwie karty graficzne, w moim kompie).

Niedługo będzie o pisaniu aplikacji do facebooka. A teraz sobie wymyślam jak powinien wyglądać serwis opiwo.com (czy raczej jak działać), żeby każdego chwycił za serce i spowodował efekt ŁAŁ.

Szczegóły wkrótce.

PS: Jeszcze chciałem przeprosić, że przynudzałem tak długo o tym JSON-RPC. Jak ktoś chętny: oto prezentacja.

by Gabrys