16 August 2010

Compiling Debian package for MySQL 5.1 with SphinxSE

I'd share experiense of compiling MySQL with SphinxSE Debian-way, i.e. making a DEB package from MySQL and SphinxSE sources. I pick an easy way using checkinstall. Checkinstall implies quick-and-dirty solution. If you want package complied Debian policy, you are not on the right spot(go here; this gives an overview). Ubuntu standard packages don't always provide latest releases. At least it regards to both sphinxsearch and mysql-server. If you already have MySQL installed, remove it:
$ sudo apt-get remove --purge mysql-server-5.1
UPDATE 19.08.2010 Otherwise, you should create a group and a user for MySQL as described here After that which mysql command should return an empty reply. Download Sphinx source tarball and untar it. We'll need just it's mysqlse folder. For instance, I'm checking out 1.10 beta release now. So I refer to sphinx-1.10-beta folder below. Download MySQL source for generic Linux platform and untar it:
$ tar -xf mysql-5.1.49.tar.gz
Currently 5.1.49 is the latest version. So I refer to mysql-5.1.49 folder below. Copy Sphinx mysqlse folder to mysql-5.1.49/storage/sphinx/:
$ cp -R sphinx-1.10-beta/mysqlse/ mysql-5.1.49/storage/sphinx/
Configure and precompile MySQL:
$ cd mysql-5.1.49
$ sh BUILD/autorun.sh
$ ./configure \
--prefix=/usr/local/mysql \
--with-plugins=sphinx \
--enable-assembler \
--with-mysqld-ldflags=-all-static \
--with-server-suffix=' SphinxSE 1.10 beta' \
--with-charset=utf8 
$ make && sudo make install 
Instead of sudo make install:
$ sudo checkinstall -D \
--pkgname=mysql-server-5.1.49_sphinxse-1.10b \
--pkgversion=1.0
You should receive output like
...
======================== Installation successful ==========================

Copying documentation directory...
./
./README
./ChangeLog
./Docs/
./Docs/Makefile
./Docs/Makefile.am
./Docs/INSTALL-BINARY
./Docs/mysql.info
./Docs/Makefile.in
./INSTALL-SOURCE
./COPYING
./INSTALL-WIN-SOURCE
grep: /var/tmp/tmp.FFdtipwofv/newfile: No such file or directory

Copying files to the temporary directory...OK

Stripping ELF binaries and libraries...OK

Compressing man pages...OK

Building file list...OK

Building Debian package...OK

Installing Debian package...OK

Erasing temporary files...OK

Writing backup package...OK

Deleting temp dir...OK


**********************************************************************

 Done. The new package has been installed and saved to

 /home/ruslan/now/mysql-5.1.49/mysql-server-5.1.49-sphinxse-1.10b_1.0-1_i386.deb

 You can remove it from your system anytime using: 

      dpkg -r mysql-server-5.1.49-sphinxse-1.10b

**********************************************************************
Save mysql-server-5.1.49-sphinxse-1.10b somewhere, e.g. /usr/src. Now you can install it:
$ sudo dpkg -i mysql-server-5.1.49-sphinxse-1.10b
and see it in the package list:
$ dpkg --list mysql-server-5.1.49-sphinxse-1.10b 
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version                     Description
+++-===========================-===========================-======================================================================
ii  mysql-server-5.1.49-sphinxs 1.0-1                       mysql-5.1.49 + SphinxSE 1.10 beta
Launch mysql server at startup:
$ sudo cp support-files/mysql.server /etc/init.d/mysql
$ sudo chmod +x /etc/init.d/mysql
$ sudo update-rc.d mysql defaults
Copy configuration:
$ sudo cp support-files/my-medium.cnf /etc/my.cnf
If you had MySQL installed before, probably /etc/mysql/my.cnf file still exists with old configuration. In such case overwrite it:
$ sudo cp -f support-files/my-medium.cnf /etc/mysql/my.cnf
or make appropriate changes in it. Make MySQL binaries visible from everywhere. E.g. append the following line to ~/.bashrc:
export PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/sbin
Now start MySQL daemon:
$ sudo /etc/init.d/mysql start
or, directly,
$ sudo /usr/local/mysql/bin mysqld_safe --user=mysql &
basedir, datadir etc. could be overriden whether in command line arguments, or in /etc/init.d/mysql itself. Now SPHINX should be in the engine list:
mysql> show engines \G
*************************** 1. row ***************************
      Engine: CSV
     Support: YES
     Comment: CSV storage engine
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 2. row ***************************
      Engine: SPHINX
     Support: YES
     Comment: Sphinx storage engine 0.9.9 ($Revision: 2391 $)
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 3. row ***************************
      Engine: MEMORY
     Support: YES
     Comment: Hash based, stored in memory, useful for temporary tables
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 4. row ***************************
      Engine: MyISAM
     Support: DEFAULT
     Comment: Default engine as of MySQL 3.23 with great performance
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 5. row ***************************
      Engine: MRG_MYISAM
     Support: YES
     Comment: Collection of identical MyISAM tables
Transactions: NO
          XA: NO
  Savepoints: NO
5 rows in set (0.00 sec)

UPDATE 18 AUGUST 2010

BTW, Ubuntu's standard mysql-server-5.1 package has a bug in \s (status) command: it doesn't show current database info(charset, name etc.). There is no such bug after installing from original source. And I'm still figuring out how to fix bug with DELETE key(it prints tilde("~") instead of removing a character). Occasionally, on the first compile a had got rid of it. But in the next install it appeared again. I suspect mysql-common(5.1.41-3ubuntu12.3) package. I believe, it'll be fixed, if I recompile mysql-server, but I've no spare time now.

04 August 2010

How to parse CLI arguments in bash

Here are some ideas how to parse command line arguments in a bash script.

Using getopt

#!/bin/bash
echo "Before getopt"
for i
do
    echo $i
    done
    args=`getopt abc:d $*`
    set -- $args
    echo "After getopt"
for i
do
  echo "-->$i"
done
The code is from here.

Using getopts

#!/usr/bin/env bash
# cookbook filename: getopts_example
#
# using getopts
#
aflag=
bflag=
while getopts 'ab:' OPTION
do
  case $OPTION in
    a) aflag=1
    ;;
    b) bflag=1
    bval="$OPTARG"
    ;;
    ?) printf "Usage: %s: [-a] [-b value] args\n" $(basename $0) >&2
    exit 2
    ;;
  esac
done
shift $(($OPTIND - 1))

if [ "$aflag" ]
then
  printf "Option -a specified\n"
fi
if [ "$bflag" ]
then
  printf 'Option -b "%s" specified\n' "$bval"
fi
printf "Remaining arguments are: %s\n" "$*"
Source

Using sed

Here is how to parse using a regular expression.
#!/bin/bash 

# For string like part1=part2
# returns part2 
function my_arg_val()
{
  echo $1 | sed 's/\([-a-zA-Z0-9_]*=\)\|\(-[a-z]\)//'
}

# Get args
for i in $*
do
  case $i in
    --option-a=*)
      option_a=`my_arg_val $i` 
      ;;
    --option-b=*)
      option_b=`my_arg_val $i`
      ;;
    --default)
      is_default=YES
      ;;
    *)
      # unknown option
      exit 2
      ;;
  esac
done

echo " 
option_a=$option_a
option_b=$option_b
is_default=$is_default
"

Using $IFS

Here is mine script I finally decided to use:
#!/bin/bash 

# Splits the first argument by delimeter specified by the 2nd parameter 
# E.g. my_split_str $string $delim
# If delimeter is not specified, the func defaults to '='
function my_split_str()
{
  if (( $# <=1 )) 
  then 
    IFS='=' 
  else 
    IFS=$2 
  fi 
  set -- $1
  echo $*
}

# Get args
for i in $*
do
  o=(`my_split_str $i '='`)
  case ${o[0]} in
    --option-a)
      option_a=${o[1]}
      ;;
    --option-b)
      option_b=${o[1]}
      ;;
    --default)
      is_default=YES
      ;;
    *)
      # unknown option
      echo "Unknown option ${o[0]}"
      exit 2
      ;;
  esac
done

echo " 
option_a=$option_a
option_b=$option_b
is_default=$is_default
"

02 August 2010

Colorful grep

This simple article describes how to make grep command highlight results with custom color without using --color option. 1. In ~/.bashrc find the following lines and uncomment them: if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases fi 2. In ~/.bash_aliases add line: alias grep='grep --color' 3. Append the following line in ~/.bashrc: export GREP_COLOR=';33m' Here ';33m' is yellow foreground color for matched symbols in grep results. Now you can invoke grep without --color option, and results are highlighted with custom color. The color codes are available e.g. here You can compose a bash script like #!/bin/bash # # This file echoes a bunch of color codes to the # terminal to demonstrate what's available. Each # line is the color code of one forground color, # out of 17 (default + 16 escapes), followed by a # test use of that color on all nine background # colors (default + 8 escapes). # T='gYw' # The test text echo -e "\n 40m 41m 42m 43m\ 44m 45m 46m 47m"; for FGs in ' m' ' 1m' ' 30m' '1;30m' ' 31m' '1;31m' ' 32m' \ '1;32m' ' 33m' '1;33m' ' 34m' '1;34m' ' 35m' '1;35m' \ ' 36m' '1;36m' ' 37m' '1;37m'; do FG=${FGs// /} echo -en " $FGs \033[$FG $T " for BG in 40m 41m 42m 43m 44m 45m 46m 47m; do echo -en "$EINS \033[$FG\033[$BG $T \033[0m"; done echo; done echo for reference.

Related info

"Colorizing" Scripts: http://tldp.org/LDP/abs/html/colorizing.html