30 March 2016

Making charts from iostat output

I've decided to make some charts from the output of the iostat command. Just for fun =)

The idea is simple: a script named `report.sh` runs periodically in background(via `cron`) collecting `iostat` data into $REPORT_DIR/$param/$device files, where

  • $REPORT_DIR is a directory with the reports;
  • $param is the `iostat` parameter name;
  • $device is a device name from the same `iostat` output.

When we need to update the charts, we call `plot.sh`.

Required components

Scripts

util.sh
#!/bin/bash -

# $1 - Optional error message.
function die
{
  [ $# -gt 0 ] && echo >&2 $1
  exit 1
}

function warning
{
  echo >&2 "Warning: $1"
}


function getSafeFilename
{
  r=${1//\//_}
  echo ${r/\%/}
}

#vim: ts=2 sts=2 sw=2 et
report.sh
#!/bin/bash -
# Generates iostat data files for processing with the GNU plot utility.

HEADER=
TIME_FORMAT="%Y/%m/%d/%H:%M"
TIME=$(date "+$TIME_FORMAT")
DIR=$(cd $(dirname "$0"); pwd)
REPORT_DIR="${DIR}/r"


#####################################################################
# Functions

source $DIR/util.sh


# $1 - exit code
function usage
{
  echo "
Generates iostat data files for processing with the GNU plot utility(plot.sh).

Usage: $0 OPTIONS

OPTIONS:
-h, --help            Display help message
-t, --time-format     Time format. Default: ${TIME_FORMAT}.
-i, --report-dir      Directory with reports. Default: ${REPORT_DIR}.
"
  exit $1
}

#####################################################################
# Parsing CLI options

OPTS=$(getopt -o t:i:h -l time-format:,report-dir:,help -- "$@")
[ $? -eq 0 ] || usage 1
eval set -- "$OPTS"

while true
do
  case "$1" in
    -t|--time-format)
      TIME_FORMAT="$2"
      shift 2;;
    -i|--report-dir)
      REPORT_DIR="$2"
      shift 2;;
    -h|--help)
      usage 0
      shift;;
    --)
      shift
      break;;
    ?)
      usage 1;;
    *)
      die "Internal error: failed to parse CLI args"
  esac
done

echo TIME_FORMAT: $TIME_FORMAT REPORT_DIR: $REPORT_DIR

#####################################################################

mkdir -p $REPORT_DIR
[ $? -eq 0 ] || die "Report directory '$REPORT_DIR' is inaccessible"

iostat -dpxk | while read -r line
do
  if [[ -z $HEADER && $line == Device* ]]; then
    HEADER=( $line )
  elif [[ -n $HEADER ]]; then
    [[ -z $line ]] && continue

    columns=( $line )

    [[ ${#HEADER[@]} != ${#columns[@]} ]] && continue


    i=1
    for h in ${HEADER[@]:1}
    do
      param_report_dir="${REPORT_DIR}/"$(getSafeFilename $h)
      mkdir -p $param_report_dir
      [ $? -eq 0 ] || die "Parameter directory '$param_report_dir' is inaccessible"

      # $REPORT_DIR/param/device
      echo $TIME$'\t'${columns[$i]} >> $param_report_dir'/'${columns[0]}
      (( ++i ))
    done
  fi
done

echo 'Done'
plot.sh
#!/bin/bash -
# Creates visual representation of data files generated by report.sh

TIME_FORMAT="%Y/%m/%d/%H:%M"
# Report files location $REPORT_DIR/param/device
DIR=$(cd $(dirname "$0"); pwd)
REPORT_DIR="${DIR}/r"
OUTPUT_DIR="${DIR}/o"
XRANGE=
PLOT_TERMINAL="png"

#####################################################################
# Functions

source $DIR/util.sh

function getPlots
{
  param_path="$1"

  find $param_path/ -maxdepth 1 -mindepth 1 -type f | while read -r device_path; do
    # $device_path = /*/*/.../param/device
    device=$(basename $device_path)

    printf "'%s' using 1:2 t '%s' with lp pt 5, " \
      "$device_path" $device
    printf "'%s' using 1:2 notitle with impulses, " \
      "$device_path"
  done
}


# Returns default value for GNU plot xrange
function getXRange
{
  y=$(date +%Y)
  m=$(date +%m)

  (( m -= 1 ))

  if [ $m -lt 1 ]; then
    m=1
    (( y -= 1 ))
  fi

  d=$(date "+$TIME_FORMAT")

  echo "[ '$y/$m/1/00:00' : '$d' ]"
}


# $1 - exit code
function usage
{
  echo "
Creates charts from files generated by report.sh

Usage: $0 OPTIONS

OPTIONS:
-h, --help            Display help message
-x, --xrange          GNU Plot xrange value. Default: $(getXRange).
-t, --time-format     Time format. Default: ${TIME_FORMAT}.
-i, --report-dir      Directory with reports. Default: ${REPORT_DIR}.
-o, --output-dir      Output directory. Default: ${OUTPUT_DIR}.
-f, --term            GNU Plot terminal. Default: ${PLOT_TERMINAL}. See 'gnuplot set terminal'.
"
  exit $1
}

#####################################################################
# Parsing CLI options

OPTS=$(getopt -o x:t:i:o:f:h -l xrange:,time-format:,report-dir:,output-dir:,term:,help -- "$@")
[ $? -eq 0 ] || usage 1
eval set -- "$OPTS"

while true
do
  case "$1" in
    -x|--xrange)
        XRANGE="$2"
        shift 2;;
    -t|--time-format)
      TIME_FORMAT="$2"
      shift 2;;
    -i|--report-dir)
      REPORT_DIR="$2"
      shift 2;;
    -o|--output-dir)
      OUTPUT_DIR="$2"
      shift 2;;
    -f|--term)
      PLOT_TERMINAL="$2"
      shift 2;;
    -h|--help)
      usage 0
      shift;;
    --)
      shift
      break;;
    ?)
      usage 1;;
    *)
      die "Internal error: failed to parse CLI args"
  esac
done

echo XRANGE: $XRANGE TIME_FORMAT: $TIME_FORMAT OUTPUT_DIR: $OUTPUT_DIR REPORT_DIR: $REPORT_DIR

#####################################################################

[ -d $REPORT_DIR ] || die "$REPORT_DIR doesn't exist"

find $REPORT_DIR/ -maxdepth 1 -mindepth 1 -type d | while read -r param_path
do
  # $param_path = /*/*/.../param
  param=$(basename $param_path)
  plots=$(getPlots "$param_path")
  plots=${plots/%, /}

  mkdir -p $OUTPUT_DIR
  [ $? -eq 0 ] || die "Output directory '$OUTPUT_DIR' is inaccessible"

  outfile="${OUTPUT_DIR}/${param}.${PLOT_TERMINAL}"

  [[ -z $XRANGE ]] && XRANGE="$(getXRange)"

  gnuplot <<EOS
TIME_FORMAT = "$TIME_FORMAT"
set terminal $PLOT_TERMINAL
set grid nopolar
set xtics rotate
set xlabel "Date/Time"
set ylabel "Value"
set yrange [ 0 : ]
set format x TIME_FORMAT timedate
set timefmt TIME_FORMAT
set xdata time
set xrange $XRANGE
set output "$outfile"

plot $plots
EOS
done

echo 'Done'

Sample chart

Gnuplot Produced by GNUPLOT 5.0 patchlevel 1 (Gentoo revision r1) 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 2016/03/30/16:00 2016/03/30/16:30 2016/03/30/17:00 2016/03/30/17:30 2016/03/30/18:00 2016/03/30/18:30 Value Date/Time sda4 sda4 gnuplot_plot_2 sda3 sda3 gnuplot_plot_4 sda1 sda1 gnuplot_plot_6 loop1 loop1 gnuplot_plot_8 loop0 loop0 gnuplot_plot_10 sda sda gnuplot_plot_12 sda2 sda2 gnuplot_plot_14 loop2 loop2 gnuplot_plot_16

23 March 2016

Arch Linux ARMv8 VM on Gentoo AMD64

The post is about configuring Arch Linux ARMv8 guest virtual machine(VM) on Gentoo AMD64 host using QEMU.

VirtualBox

I'm used to making VMs in VirtualBox. It can handle a large variety of disk image formats. We can easily convert the RAW image to VDI:

$ VBoxManage convertfromraw disk.img arch.vdi --format VDI 
Converting from raw image file="disk.img" to file="arch.vdi"...
Creating dynamic image with size 4294967296 bytes (4096MB)...

However, at the time of writing I found no way to run the AARCH64 guest on AMD64 host. The boot process stuck on UEFI interactive shell which failed to detect the drive contents. All my attempts to boot from the shell had failed. I had also tried to fix it through VBoxInternal2/EfiBootArgs without success. Finally decided to give QEMU a try.

QEMU

QEMU proved to be more flexible. I had managed to boot the system by specifying initial ramdisk, firmware, kernel and kernel arguments. The following steps describe the whole process.

Install required packages

Prepare USE-flags:

$ sudo euse -p app-emulation/qemu -E qemu_softmmu_targets_aarch64 qemu_user_targets_aarch64 spice

Install QEMU as described in Gentoo Wiki.

Download firmware

Download QEMU_EFI.fd from snapshots.linaro.org.

Create disk image

$ qemu-img create -f raw disk.img 4G

Create partition table

$ parted disk.img mklabel msdos

Mount the disk image as a loopback device

$ sudo losetup -f disk.img
losetup -a
/dev/loop0: []: (/home/ruslan/aarch64-qemu/disk.img)

Create a partition

$ sudo fdisk /dev/loop0
fdisk> n
(choose defaults)
fdisk> a
fdisk> w

Detach the loop device

$ sudo losetup -d /dev/loop0

Create a loop device for the new partition

$ sudo losetup -f -P disk.img

$ losetup -a
/dev/loop0: []: (/home/ruslan/aarch64-qemu/disk.img)

Format the partition to ext4

$ sudo mke2fs -t ext4 /dev/loop0p1

Mount the new partition

$ mkdir -p mnt
$ sudo mount /dev/loop0p1 mnt

Extract the Arch files

Extract the Arch ARM Linux files from archive according to instructions:
$ sudo su -
# bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C mnt
# exit

Boot in fallback mode

The system(d) failed to detect the hard drive. So I had to fix initial ramdisk hooks order through fallback mode.

$ filename="disk.img"

$ qemu-system-aarch64 -m 2048 -cpu cortex-a57 \
  -smp 1 -M virt -bios QEMU_EFI.fd -serial stdio \
  -drive if=none,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -kernel mnt/boot/Image -initrd mnt/boot/initramfs-linux-fallback.img \
  -append "root=/dev/vda1"

Within the guest system change HOOKS in /etc/mkinitcpio.conf from

HOOKS="base udev block autodetect modconf filesystems keyboard fsck"
to:
HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

Remount / for read-write:

# mount -o remount,rw /

Then regenerate initramfs and shutdown the guest:

# mkinitcpio -p /etc/mkinitcpio.d/linux-aarch64.preset
# shutdown now

Remount the loop device

$ sudo umount /dev/loop0p1
$ sudo losetup -d /dev/loop0
$ sudo losetup -f -P disk.img
$ sudo mount /dev/loop0p1 mnt

Finally

Run the machine in normal mode:

$ filename="disk.img"
$ qemu-system-aarch64 \
  -m 512 \
  -cpu cortex-a57 -M virt \
  -drive if=none,format=raw,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -smp 1 \
  -bios QEMU_EFI.fd \
  -serial stdio \
  -kernel mnt/boot/Image \
  -initrd mnt/boot/initramfs-linux.img \
  -append "root=/dev/vda1 rootfstype=ext4 rw" \
  -netdev user,id=unet -device virtio-net-device,netdev=unet

The only drawback is that ICMP packets(ping) will not work in this mode. The network settings can be tweaked, though.

Helper scripts

To make further usage easier I've created scripts to mount and unmount the disk image, and to launch the VM.

util.sh

#!/bin/bash -

function usage()
{
  echo -e "Usage:\n$0 disk-image mount-point"
}

function error()
{
  [ $# -gt 0 ] && echo >&2 "!! $1"
}

function die()
{
  error $1
  exit 1
}

function notice()
{
  echo "* $1"
}

function checkimg()
{
  [ -e "$1" ] || die "$1 doesn't exist"
}

function checkmnt()
{
  [ -e "$1" ] || die "$1 doesn't exist"
}

mount.sh

#!/bin/bash -

source util.sh

if [ ! $# -gt 1 ]; then
  error 'Missing required arg(s)'
  usage
  die
fi

imgfile="$1"
mntdir="$2"

checkimg $imgfile
checkmnt $mntdir

free_loop_part=$(losetup --show --find --partscan "$imgfile")"p1"
[ $? -eq 0 ] || die 'Failed to create loop device'
notice "Created loop partition $free_loop_part"

mount -o loop,rw "$free_loop_part" "$mntdir"
[ $? -eq 0 ] || die "Failed to mount $imgfile to $mntdir"
notice "Associated $free_loop_part with $imgfile and mounted to $mntdir"

unmount.sh

#!/bin/bash -
# Unmounts and detaches loop devices associated with a disk image

source util.sh

if [ ! $# -gt 1 ]; then
  error 'Missing required arg(s)'
  usage
  die
fi

imgfile="$1"
mntdir="$2"

checkimg $imgfile
checkmnt $mntdir

notice "Unmounting $mntdir"
while true
do
  umount --detach-loop "$mntdir" >/dev/null 2>&1
  [ $? -eq 0 ] || break
done

losetup -j "$imgfile" -l --raw -O NAME -n | while read -r ld
do
  losetup -d "$ld"
  [ $? -eq 0 ] && notice "Detached loop device $ld"
done

run.sh


#!/bin/bash -

source util.sh

[ $# -gt 0 ] && filename="$1"
[ $# -gt 1 ] && mntdir="$2"

: ${filename:="disk.img"}
: ${mntdir:="mnt"}

checkimg $filename
checkmnt $mntdir

notice "Launching Arch aarch64, img:$filename mntdir $mntdir"

qemu-system-aarch64 \
  -m 512 \
  -cpu cortex-a57 -M virt \
  -drive if=none,format=raw,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -smp 1 \
  -bios QEMU_EFI.fd \
  -serial stdio \
  -kernel $mntdir/boot/Image \
  -initrd $mntdir/boot/initramfs-linux.img \
  -append "root=/dev/vda1 rootfstype=ext4 rw" \
  -netdev user,id=unet -device virtio-net-device,netdev=unet

Usage

With the help of these scripts we can start and stop the VM easily:

$ sudo ./mount.sh disk.img mnt
* Created loop partition /dev/loop2p1
* Associated /dev/loop2p1 with disk.img and mounted to mnt
$ df
...
/dev/loop3       4061888   1128524   2707316  30% /home/ruslan/aarch64-qemu/mnt
$ losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
...
/dev/loop2         0      0         0  0 /home/ruslan/aarch64-qemu/disk.img
/dev/loop3         0      0         1  0 /dev/loop2p1

$ ./run.sh
* Launching Arch aarch64, img:disk.img mntdir mnt
...
Arch Linux 4.5.0-1-ARCH (ttyAMA0)

alarm login: root
Password: 
Last login: Tue Mar 22 17:23:48 on ttyAMA0
[root@alarm ~]# arch
-bash: arch: command not found
[root@alarm ~]# uname -a
Linux alarm 4.5.0-1-ARCH #1 SMP Mon Mar 14 18:47:45 MDT 2016 aarch64 GNU/Linux
[root@alarm ~]# shutdown now
...
$ sudo ./unmount.sh disk.img mnt
* Unmounting mnt
* Detached loop device /dev/loop2

02 November 2015

PostgreSQL function for parsing JSON into Sphinx keywords

This post describes how to write a basic PostgreSQL function for parsing a JSON into Sphinx keywords(space-separated values of the JSON object).

Requirements

  • Perl core
  • Perl JSON module
  • PostgreSQL built with JSON support

Sample Problem

We have workers table:

CREATE TABLE workers(
  id integer,
  props text
);
where props keeps arbitrary worker properties in JSON format.

Let's insert a worker:

INSERT INTO workers VALUES(1, '{"x":"x test", "d":104, "o":{"o1":"o1 name"}, "a":["a1", "a2"]}');

We want to make the worker properties searchable through Sphinx. So we add corresponding PostgreSQL source and an index in our sphinx.conf:

source src_workers: pgsql
{
  sql_query_pre = set work_mem = '100MB'
  sql_query = select id, props from workers
  sql_field_string = props
}

index idx_workers
{
  source = src_testing
  path = /home/ruslan/sphinxsearch/idx_workers
  min_prefix_len = 3
  docinfo = extern
}

With the current sql_query the props field will be cluttered with JSON keys and specific characters. We'll write a PostgreSQL function which will give us a single line of space-separated values:

$ createlang plperlu t # < "t" is a database name
$ psql -d t
create or replace function json2sphkw(text,text) returns text as $$
use JSON qw( decode_json );

my ($in) = @_;
$_SHARED{read_sub} = sub {
  my ($in) = @_[0];

  if (ref($in) eq "HASH") {
    my @res, $read_sub = $_SHARED{read_sub};
    foreach $v (values %$in) {
      push @res, &$read_sub($v);
    }
    return join(' ', @res);
  } elsif (ref($in) eq "ARRAY") {
    my @res, $read_sub = $_SHARED{read_sub};
    foreach $v (values @$in) { push @res, &$read_sub($v); }
    return join(' ', @res);
  }
  return $in;
};

my $h = decode_json $in;
my $read_sub = $_SHARED{read_sub};

return &$read_sub($h);
$$ LANGUAGE plperlu;

Now we can use it:

SELECT json2sphkw(props, ' ') FROM workers WHERE id = 1;
        json2sphkw
--------------------------
 o1 name a1 a2 x test 104
(1 row)
And the sphinx.conf can be changed as follows:
@@ -1,7 +1,7 @@
 source src_workers: pgsql
 {
   sql_query_pre = set work_mem = '100MB'
-  sql_query = select id, props from workers
+  sql_query = select id, json2sphkw(props, ' ') from workers
   sql_field_string = props
 }

References

18 December 2014

Randomly changing the desktop wallpaper via crontab

Here is a simple way to organize a random desktop wallpaper by means of crontab.

We'll use feh application here. Let's put the following code into ~/bin/wallpaper-change file:

#!/bin/bash -
# Pick and set a random wallpaper

: ${DISPLAY:=":0"}

if [ $# -lt 1 ]; then
  echo >&2 "No source directory specified"
  exit 1
fi

dir="$1"
if [[ ! -d "$dir" ]]; then
  echo >&2 "Directory $dir doesn't exist"
  exit 1
fi

find $dir/ -maxdepth 1 -type f | sort -R | tail -1 | while read f
do
  feh --bg-fill "$f" &
done
(Don't forget to make it executable.)

Now we can adjust the crontab, e.g.:

DISPLAY=:0
@hourly ~/bin/wallpaper-change ~/Pictures/wallpapers

Note, setting the DISPLAY environment variable is crucial.

26 November 2014

Changing keyboard layout in system console

The post is about configuring a Unicode keyboard layout for the system console on Gentoo. However, the steps should be similar on other systems.

Enable Unicode in /etc/rc.conf:

unicode="yes"

Install a font for UTF-8 consoles:

# emerge -av media-fonts/terminus-font

Change /etc/conf.d/consolefont:

consolefont=LatArCyrHeb-16
# consolefont="ter-v16b"
consoletranslation=""
Make sure that /etc/init.d/consolefont is enabled:
# rc-update add consolefont boot

Set default console keymap in /etc/conf.d/keymaps:

keymap="us"

Add /etc/init.d/keymaps to the boot run level:
# rc-update add keymaps boot

Configure the shell to send switch-to-unicode escape sequence at each login. For Bash add the following to your ~/.bash_profile:

if test -t 1 -a -t 2 ; then
        echo -n -e '\033%G'
fi

To make the switch to Unicode global for all users the above-mentioned snippet should be added to /etc/profile.

Reboot the system. Now Unicode should be available on all system console logins.

To load different keyboard layout use the loadkeys command (which is shipped with sys-apps/kbd package):

# loadkeys /usr/share/keymaps/{YOUR_ARCH}/{PATH_TO_KEYMAP}
For instance, to load Russian Qwerty layout for i386 (will work on amd64):
# loadkeys /usr/share/keymaps/i386/qwerty/ru.map.gz

One might want to make a shortcut for this. Use Ctrl-Shift shortcut for switching between the keymaps.

28 June 2014

Gentoo configure error: C compiler cannot create executables

If configure phase failes with this message, then most likely the dev-libs/mpc package is removed. I removed it accidentally. Afterwards I was unable to emerge anything. Here is a simple solution for the issue.

Download stage3 tar ball from a Gentoo mirror, unpack it and copy usr/lib/libmpc.so* over /usr/lib. Then run ldconfig and try to build something.

Hopefully, the post will help another gentoer like me who will also wonder what's going wrong with the C compiler.

21 June 2014

How to capture audio stream using Pulseaudio and VLC

Some Internet radio stations offer URLs for streaming with popular software, some don't. I'll show a way to capture currently runnig audio stream to file using Pulseaudio and VLC.

Step 1. Pick an output

$ pactl list short | grep RUNNING | awk '{print $2}'
alsa_output.pci-0000_00_1b.0.analog-stereo
alsa_output.pci-0000_00_1b.0.analog-stereo.monitor
(The list of devices also be obtained via VLC GUI: View - Playlist - Audio capture.) Let's pick the second one.

Step 2. Write a Bash script

In the following script replace the value of pulseaudio_stream variable with 'pulse://YOUR_OUTPUT', where YOUR_OUTPUT is the output chosen in the first step.

#!/bin/bash -
set -o nounset

pulseaudio_stream='pulse://alsa_output.pci-0000_00_1b.0.analog-stereo.monitor'

usage() {
  IFS= read -r errstr <<-errstr
  Captures current pulseaudio stream $pulseaudio_stream and saves it to a file
  Usage: $0 <output-path>
  Example: $0 ~/Music/radio.mp3
errstr

  echo >&2 "$errstr"
  exit 1
}

if [[ $# < 1 ]]; then
  usage
fi

output="$1"

nvlc "$pulseaudio_stream" ':sout=#transcode{vcodec=none,acodec=mp3,ab=128,channels=2,samplerate=44100}:duplicate{dst=std{access=file,mux=raw,dst='$output'}}'
  

Save it to ~/scripts/bash/rec-audio.sh and make it executable.

Step 3. Use it

~/scripts/bash/rec-audio.sh ~/Music/radio.mp3

Currently running audio will be recorded to the file until you quit VLC.