Software Engineer's Notes

29 July 2011

Mapping special characters in Sphinx configuration

Sphinx sometimes assumes some characters as word separators. For instance, letters 'e' and 'ё' are kinda similar in Russian. Some print issues even replace the latter with the former. However, Sphinx assumes 'ё' a word separator. To prevent it, one should search for Unicode code points and append mapping to charset_table index config:

charset_type  = utf-8
charset_table  = 0..9, A..Z->a..z, _, a..z, U+A8->U+E5, U+B8->U+E5, U+410..U+42F->U+430..U+44F, U+430..U+44F, \
U+0451->U+0435

Here U+0451->U+0435 maps 'ё'(U+0451) to 'e'(U+0435). The Unicode code points could be found here.

References

16 June 2011

How to create a Sphinx wordform dictionary

To build a wordform file for Sphinx you may use some kind of spellchecker dictionary like myspell, ispell, pspell, aspell. Let's make a wordform file for Russian language from myspell-russian package in openSUSE.

To install myspell-russian:

$ sudo zypper in myspell-russian
$ rpm -ql myspell-russian
/usr/share/doc/packages/myspell-russian
/usr/share/doc/packages/myspell-russian/descr_en.txt
/usr/share/doc/packages/myspell-russian/descr_ru.txt
/usr/share/doc/packages/myspell-russian/description.xml
/usr/share/doc/packages/myspell-russian/dictionaries.xcu
/usr/share/doc/packages/myspell-russian/icon.png
/usr/share/doc/packages/myspell-russian/licence.txt
/usr/share/myspell
/usr/share/myspell/ru_RU.aff
/usr/share/myspell/ru_RU.dic

Build wordforms:

$ spelldump /usr/share/myspell/ru_RU.dic /usr/share/myspell/ru_RU.aff wordforms_myspell_ru_RU.txt
spelldump, an ispell dictionary dumper

Loading dictionary...
Loading affix file...
Using MySpell affix file format
Dictionary words processed: 146265

Detect encoding:

$ cat wordforms_myspell_ru_RU.txt | enca -L ru 
KOI8-R Cyrillic
  LF line terminators

Конвертируем в UTF-8:

$ iconv -f KOI8-R -t UTF-8 -o wordforms_myspell_ru_RU_UTF8.txt wordforms_myspell_ru_RU.txt

Finally, in sphinx.conf set:

wordforms     = /path/to/wordforms_ru_RU_UTF8.txt

23 April 2011

Strange C sizeof(struct) results?

If you wonder, why sizeof operator returns values bigger than you expect, you probably have to learn about compiler data structure alignment(as I had to:). Here I post a simple program demonstrating the case.

main.c

#include <stdio.h>
#include <string.h>

typedef struct 
#ifdef PACKED 
__attribute__ ((packed)) 
#endif
{
    char *str;      // 8
    char ch ;       // 1
    int a;          // 4
    int b;          // 4
    int c;          // 4
    // Packed total size should be: 21
    // Aligned total size should be: 24
} test_t;

static void print_sizes(test_t *t)
{
    printf("DATA TYPE SIZES\n");
    printf("char*: %ld\n", sizeof(char*));
    printf("char: %ld\n", sizeof(char));
    printf("int: %ld\n", sizeof(int));

    printf("\nSIZE OF *t: %ld\n", sizeof(*t));
}

int main (int argc, char const* argv[]) {
    test_t t;
    t.str = "test string"; 

    print_sizes(&t);

    return 0;
}

makefile

ifeq ($(PACKED), true)
    PACK=-DPACKED
else
    PACK=
endif

ifdef OUT
    OUTFILE=$(OUT)
else
    OUTFILE=test
endif

CC=gcc
CFLAGS=-Wall -c $(PACK)

all: test

test: main.o
    $(CC) main.o -o $(OUT) 

main.o: main.c
    $(CC) $(CFLAGS) main.c -o main.o

clean: 
    rm -f *.o *~

Compile versions with packed and padded memory:

$ make PACKED=true OUT=test_packed
$ make OUT=test_padded

and test them:

$ ./test_packed
DATA TYPE SIZES
char*: 8
char: 1
int: 4

SIZE OF *t: 21

$ ./test_padded
DATA TYPE SIZES
char*: 8
char: 1
int: 4

SIZE OF *t: 24

So one should consider a space-time tradeoff for every particular program.

I'd be glad, if it helps someone.

19 April 2011

How to add new highlight mode in GEdit

Here I show common steps to create custom language for GtkSourceView 2.0(GEdit uses it to display code). Then we'll create MBox language and special highlighting for it.

Common steps

1. Make directories for language definition and for new MIME:

$ mkdir -p ~/.local/share/gtksourceview-2.0/language-specs/
$ mkdir -p  ~/.local/share/mime/packages/

2. Create new MIME definition in ~/.local/share/mime/packages/

3. Create .lang file in ~/.local/share/gtksourceview-2.0/language-specs/

4. Update MIME database

$ update-mime-database ~/.local/share/mime

New language for MBox

STEP - 2: New MIME
Actually it should be already in /usr/share/mime/application/mbox.xml. But one would probably override it:

$ cat > ~/.local/share/mime/packages/mbox.xml
<!--
Author:     Ruslan Osmanov
Date:       Tue Apr 19 05:12:58 UTC 2011
Version:    1.0
-->
<mime-info xmlns='http://www.freedesktop.org/standards/shared-mime-info'>
    <mime-type type="application/mbox">
        <comment>MBox Source</comment>
        <magic priority="50">
            <match type="string" offset="0" value="&gt;"/>
        </magic>
        <glob pattern="*.mbox"/>
    </mime-type>
</mime-info>
^D

^D stands for Ctrl + D shortcut.

STEP - 3: New Language Definition

The GNOME Language Definition Reference 2.0[1] and Language Definition v2.0 Tutorial [2] are good start points.
Syntax highlighting styles could be found in /usr/share/gtksourceview-2.0/styles/.

This info is quite enought to implement a new language:

$ cat > ~/.local/share/gtksourceview-2.0/language-specs/mbox.lang
<?xml version="1.0" encoding="UTF-8"?>
<!--
Author: Ruslan Osmanov 
Date:   Mon Apr 18 17:08:52 UTC 2011
Version:   1.0
MBox language spec
-->
<language id="mbox" _name="MBox" version="2.0" _section="Email">
  <metadata>
 <property name="mimetypes">text/x-mbox;application/mbox</property>
 <property name="globs">*.mbox</property>
 <property name="line-comment-start">&gt;</property>
  </metadata>
  <styles>
 <style id="keyword" _name="Keyword" map-to="def:keyword"/>
 <style id="underlined" _name="Underlined" map-to="def:underlined"/>
 <style id="comment" _name="Comment" map-to="def:comment"/>
 <style id="string" _name="A string" map-to="def:string"/>
  </styles>
  <definitions>
 <context id="mbox">
   <include>
  <context id="keyword" style-ref="keyword">
    <prefix>^</prefix>
    <suffix>( |\:)</suffix>
    <keyword>(From|To|Subject)</keyword>
    <!-- Russian -->
    <keyword>(От|Кому|Тема)</keyword>
  </context>
  <context id="link" style-ref="underlined">
   <keyword>\b(ftp|http|https)\:\/\/.*\b</keyword> 
   <keyword>mailto\:.+\@[a-z]+</keyword> 
  </context>
  <context id="comment" style-ref="comment">
    <start>^&gt;</start>
    <end>$</end>
    <include>
   <context ref="link"/>
    </include>
  </context>
  <context id="comment-multiline" style-ref="comment">
    <start>^--</start>
    <include>
   <context ref="def:in-comment"/>
    </include>
  </context>
  <context id="string" end-at-line-end="true" style-ref="string">
    <start>"</start>
    <end>"</end>
  </context>

   </include>
 </context>
  </definitions>
</language>
<!--vim: set ft=xml:-->

^D

Note, ^D stands for Ctrl + D shortcut.

STEP - 4. Update MIME database

$ update-mime-database ~/.local/share/mime

Voila! You now have new MBox language in View - Highlight Mode - Email section.

References

08 April 2011

Change indentation: spaces to tabs

To convert space indentation in significant number of files, you may use the
following script.

#!/bin/bash - 
#===============================================================================
#          FILE:  fixtab.sh
#         USAGE:  ./fixtab.sh <directory>
#   DESCRIPTION:  Convert expanded tabs into tabs with 4 space width
#       OPTIONS: $1 directory
#        AUTHOR: Ruslan Osmanov 
#       CREATED: 04/08/2011 03:26:00 PM UZT
#===============================================================================
set -o nounset                              # Treat unset variables as an error

DIR=$1

function usage(){
 echo "bash $0 <directory>"
}

[[ ! -d $DIR ]] && usage

# Correct existing modelines I used to include modelines beginning from set
# expandtab, so this worked for me. If you don't use modelines, comment it out 
find $DIR -name *.php -exec \
 sed -i -e 's%# vim: set expandtab:%# vim: set noet:%g' {} \;

# Retab & format source code
find $DIR -name *.php -exec \
 vim -u NONE \
 -c 'set ft=php' \
 -c 'set shiftwidth=4' \
 -c 'set tabstop=4' \
 -c 'set noexpandtab!' \
 -c 'set noet' \
 -c 'retab!' \
 -c 'bufdo! "execute normal gg=G"' \
 -c wq \
 {} \;

For example:

$ bash ./fixtab.sh /srv/www/myproject

I would be glad, if this helped somebody.

21 March 2011

When middle click doesn't work in Vim

I mean the Linux feature to use mouse middle click to paste selected text. In vim(not gvim) it sometimes doesn't work. Here I post how to fix it.

In ~/.vimrc find line like

set mouse=a

and change it to

set mouse=r

That's it!

20 March 2011

RPM for nethogs in openSUSE

Nethogs is a 'top' for network connections. It seems, RPMs are no longer supported(At least homepage links for RPM don't work). I had some troubles making an RPM for in in openSUSE. So I'd share a workaround for it.

Download lastest(for now) release tarball and uncompress it:

$ cd /tmp && wget 'http://sourceforge.net/projects/nethogs/files/nethogs/0.7/nethogs-0.7.0.tar.gz/download'
$ tar -xzvf nethogs-0.7.0.tar.gz

Now make

$ cd nethogs
$ make

You may try running

$ sudo checkinstall -R

but if you are at openSUSE box as me, you'd probably get error that says you didn't specify filename in
'install -d -m 755'... If so, edit ./Makefile:
1) find line:

sbin  := $(DESTDIR)/sbin

2) append the folliwing line after that:

bin  := $(DESTDIR)/bin

That's it! Yoy may now run

$ sudo checkinstall -R

and follow checkinstall instuctions. Everything should be OK now.

Good luck.

UPDATE
You might have to install checkinstall:

$ sudo zipper in checkinstall

Subscribe to: Posts ( Atom )

29 July 2011

References

16 June 2011

See also

23 April 2011

main.c

makefile

19 April 2011

Common steps

New language for MBox

References

08 April 2011

21 March 2011

20 March 2011