I said I'd never blog

DeforaOS, NetBSD, reverse-engineering and stuff

Older stuff...

Cross-compiling Debian packages for hackable:1
Fri Apr 17 02:00:36 CEST 2009

I was finally able to create my first cross-compiled Debian package this evening. This wasn't without a lot of pain, and some issues remain. Still, it's worth mentioning how to reproduce it, along with the problems that I have more or less solved along the way.

Requirements

It's somewhat easier to start from hackable:1's cross-compiling environment. In fact there are still a few glitches to get it truly working, as always just create tickets if you get stuck (or solve something yourself).

pkg-config 0.23

In contrast with the current autoconf/automake/libtool's nightmare, pkg-config has managed to make life easier. This is particularly true since its 0.23 release and fix for the PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_LIBDIR environment variables. In our case, they are set as follows:

export PKG_CONFIG_LIBDIR="/usr/arm-linux-gnueabi/usr/lib/pkgconfig:/usr/arm-linux-gnueabi/usr/share/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR="/usr/arm-linux-gnueabi"

This does two important things:

  • prefix the relevant CFLAGS and LDFLAGS flags with the path to the cross-compiled include and library files;
  • forces pkg-config to only look at the definition files for packages installed in the cross-compilation environment.

Unfortunately, Debian does not include pkg-config 0.23 yet. I have explained how to install it from source, but even better I have packaged the patched 0.23 release in hackable:1. It will appear from the upcoming build bot soon.

libmokoui2

Unlike jbl2024 with neod, I have settled on libmokoui2. Here's a summary of the issues I ran into.

$ dpkg-buildpackage -rfakeroot -us -uc -aarmel
[...]
checking for C compiler default output file name... 
configure: error: C compiler cannot create executables
See `config.log' for more details.
make: *** [config.status] Error 77
dpkg-buildpackage: failure: debian/rules build gave error exit status 2

Here's what said in config.log:

configure:2950: arm-linux-gnueabi-gcc -g -O2  -Wl,-z,defs -lX11 conftest.c  >&5
/usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/arm-linux-gnueabi/bin/../../lib/libX11.so when searching for -lX11
/usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/arm-linux-gnueabi/bin/../../lib/libX11.a when searching for -lX11
/usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/bin/ld: cannot find -lX11
collect2: ld returned 1 exit status

The solution here is to force the LDFLAGS, as follows:

ifneq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))
CROSS= --build $(DEB_BUILD_GNU_TYPE) --host $(DEB_HOST_GNU_TYPE)
LDFLAGS=-L/usr/$(DEB_HOST_GNU_TYPE)/lib -L/usr/$(DEB_HOST_GNU_TYPE)/usr/lib
else
[...]
./configure $(CROSS) --prefix=/usr --mandir=\$${prefix}/share/man --infodir=\$${prefix}/share/info CFLAGS="$(CFLAGS)" LDFLAGS="-Wl,-z,defs $(LDFLAGS)"

This works much better. We even got rid of -lX11, which should not be necessary: it is already required by the other libraries which will be used.

libtool nightmare

This is what drove me nuts again. I am not the only one:

« You might play with compiler and linker flags, try to hack and understand the libtool shell script (best of luck if you can decipher it), and end up cursing the ancestors of those who ever came up with such a dumb system. »

...and so I read it, and cursed endlessly. Thousands of lines of shell stress-testing code. To put it shortly, libtool is re-generated everytime you run ./configure, which appends ltmain.sh to some configuration variables.

At this point, depending on which LDFLAGS you're using, libtool will have a life of its own, sometimes replacing -lname by /usr/lib/libname.so, sometimes not (and ignoring most other flags). This got wrong in our case; in fact, libtool was correct in picking up *.la files (libtool's own library description format) in /usr/arm-linux-gnueabi/usr/lib, but it was blindly trusting them as they mention the libraries are stored in /usr/lib instead.

The solution should have been as "simple" as using -inst-prefix-dir (which I did not know about, and found by reading the code). Unfortunately it wasn't and I had to patch ltmain.sh to obtain what I feel is the correct fix:

test -f "$inst_prefix_dir$add" && add="$inst_prefix_dir$add"

This checks if the library is in the staging directory, which takes then precedence over the normal path.

dh_strip and dh_shlibdeps

I was totally expecting them to break: dh_strip hardcodes strip as the executable to run (instead of $(DEB_HOST_GNU_TYPE)-strip). Likewise, dh_shlibdeps will have to be modified to support cross-compilation. So I temporarily disabled them both.

And the winner is...

It feels good when it works:

dpkg-deb: building package `libmokoui2-dev' in `../libmokoui2-dev_0.3+svn4878-1_armel.deb'.
dpkg-deb: building package `libmokoui2' in `../libmokoui2_0.3+svn4878-1_armel.deb'.
 dpkg-genchanges  >../libmokoui2_0.3+svn4878-1_armel.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload (original source is included)

It's not so bad, I learned some :)

pkg_install is segfaulting
Sat Apr 4 22:28:38 CEST 2009

Currently analyzing this:

Core was generated by `pkg_info'.
Program terminated with signal 11, Segmentation fault.
#0  var_get_memory (buf=0x0, variable=0x7f7ffffffed9 "OBJECT_FMT") at var.c:161
161             for (; *buf; buf = next) {
(gdb) bt
#0  var_get_memory (buf=0x0, variable=0x7f7ffffffed9 "OBJECT_FMT") at var.c:161
#1  0x000000000040524b in show_var (buf=0x7f7ffffffed9 "OBJECT_FMT", 
    variable=0x7f7ffd209b25 "[ $wrapper_re") at show.c:137
#2  0x00000000004043d2 in pkg_do (pkg=0x7f7ffd205080 "nasm-2.05.01")
    at perform.c:371
#3  0x00000000004048fb in pkg_perform (pkghead=0x51dcc0) at perform.c:624
#4  0x0000000000403de2 in main (argc=<value optimized out>, 
    argv=<value optimized out>) at main.c:352
(gdb) print buf
$1 = 0x0

First, it doesn't look like 30276.

The calls to malloc and realloc might gain being checked for errors:

In pkg_install-20081013/lib/var.c, function var_get_memory:

147 char *
148 var_get_memory(const char *buf, const char *variable)
149 {
[...]
173                 if (value) {
174                         value = realloc(value, valuelen+thislen+2);
175                         value[valuelen++] = '\n';
176                 }
177                 else {
178                         value = malloc(thislen+1);
179                 }
180                 sprintf(value + valuelen, "%.*s", (int)thislen, data);

(they were replaced by xmalloc and xrealloc in the last version, not sure if they abort in case of error or whatever)

This code is only called when using pkg_info -Q (from reading info/main.c and info/perform.c). I get this:

# /usr/pkg/sbin/pkg_info -Q OBJECT_FMT nasm                                    
Memory fault (core dumped)
# gdb /usr/pkg/sbin/pkg_info pkg_info.core
[...]
(gdb) set print elements 0
(gdb) break var_get_memory
(gdb) run -Q OBJECT_FMT nasm
[...]
(gdb) x/s buf
[...]
(gdb) next
[...]
(gdb) break 166

The code goes all the way until the end of the string, and crashes when it can't find the desired value (shortly after hitting line 166). But even after modifying the code for clarity, it gets really weird:

161             while (buf && *buf) {
(gdb) print buf
$2 = 0x7f7ffd207b25 "[ $wrapper_re"
(gdb) step

Program received signal SIGSEGV, Segmentation fault.
0x000000000040def7 in var_get_memory (buf=0x0, 
    variable=0x7f7fffffff02 "OBJECT_FMT") at var.c:161
161             while (buf && *buf) {
(gdb) print buf
$3 = 0x0

I feel just like losing time... :(

hackable:1 as a Xen guest
Fri Apr 3 01:16:42 CEST 2009

As probably already pretty obvious here, my main Operating System is NetBSD. Over the past few weeks, I've been trying virtualization, using Xen with NetBSD as the native host. This excellent howto describes the whole procedure better than I could do it.

This wasn't without a few issues though. Here's what was important in my case:

Xen device files

Don't forget to create the device files:

# cd /dev && MAKEDEV xen

The bootloader

For 64-bit hosts, or simply if your / partition is "big" (somewhere between 512MB and 4096MB in my experience), then you must use NetBSD's native bootloader. It requires NetBSD 5.0 (or -current, and you can also get it to work on 4.0 and below).

First, don't forget to copy /usr/mdec/boot to /boot when updating the bootloader (symptom is "unknown command"). Then, copy and modify /boot.cfg from /usr/src/etc/etc.amd64/boot.cfg:

menu=Boot normally:boot /netbsd.gz
menu=Boot Xen:modules enabled;load /netbsd.xen0 bootdev=wd0a ro console=pc;multiboot /xen.gz dom0_mem=1024M
menu=Boot single user:boot /netbsd.gz -s
menu=Disable ACPI:boot /netbsd.gz -2
menu=Disable ACPI and SMP:/boot /netbsd.gz -12
menu=Drop to boot prompt:prompt
default=1
timeout=10

Create the virtual machine

You can use a plain file as the guest, to be able to switch between Xen and qemu at will. I have detailed my procedure below.

Create and partition a 20GB file

I'm not sure if this blocksize is optimal. You may also feel free to set swap space at this stage, I chose to proceed without any:

# dd if=/dev/zero of=ldeb bs=1024*1024 count=20480
# vnconfig -c vnd0 ldeb
# fdisk -u vnd0
[...]
Do you want to change our idea of what BIOS thinks? [n] n

Partition table:
0: <UNUSED>
1: <UNUSED>
2: <UNUSED>
3: <UNUSED>
Bootselector disabled.
No active partition.
Which partition do you want to change?: [none] 0
The data for partition 0 is:
<UNUSED>
sysid: [0..255 default: 169] 131
start: [0..xcyl default: 63, 0cyl, 0MB] 63
size: [0..xcyl default: x, xcyl, xMB] $
bootmenu: []

Partition table:
0: Linux native (sysid 131)
    start 63, size 41942977 (20480 MB, Cyls 0-2610/212/34), Active
        PBR is not bootable: All bytes are identical (0x00)
[...]
Which partition do you want to change?: [none]

We haven't written the MBR back to disk yet.  This is your last chance.
Partition table:
[...]
Should we write new partition table? [n] y

NetBSD also needs a disklabel to identify the different partitions:

# disklabel -Ii vnd0
partition> a
Filesystem type [?] [4.2BSD]: Linux Ext2
Start offset ('x' to start after partition 'x') [0c, 0s, 0M]: 63
Partition size ('$' for all remaining) [xc, ys, zM]: $
partition> W
Label disk [n]? y
Label written
partition> Q

Format and mount the partition

I force a block size of 4KB for compatibility with NetBSD, and 128 bytes per inode for performance:

# mke2fs -b 4096 -I 128 /dev/vnd0a
# mkdir /mnt/ldeb
# mount /dev/vnd0a /mnt/ldeb

Download and extract hackable:1

# wget http://build.hackable1.org/Hackable1-Openmoko-Freerunner-cross-daily.tar.gz
# tar -xzvpf Hackable1-Openmoko-Freerunner-cross-daily.tar.gz -C /mnt/ldeb

Download and install the kernel modules

We're using Xen 3.0.4 now to make sure it is well supported as of today. The kernel itself doesn't need to be on the partition, and will be installed later:

# wget http://bits.xensource.com/oss-xen/release/3.0.4-1/bin.tgz/xen-3.0.4_1-install-x86_64.tgz
# tar xzvf xen-3.0.4_1-install-x86_64.tgz dist/install/lib/modules \
		dist/install/boot
# mkdir -p /mnt/ldeb/lib/modules
# mv dist/install/lib/modules/2.6.16.33-xen /mnt/ldeb/lib/modules

Unmount the partition

And take care of the vnd pseudo-device as well:

# umount /mnt/ldeb
# vnconfig -u vnd0

Install the kernel image

Copy the kernel next to the image:

# cp dist/install/boot/vmlinuz-2.6.16.33-xen /home/xen/ldeb.xenU

Configure the virtual machine

This is my configuration file, with the comments stripped off:

# cat > /usr/pkg/etc/xen/ldeb << EOF
kernel = "/home/xen/ldeb.xenU"
memory = 768
name = "ldeb"
cpu = 1
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]
disk = [ 'file:/home/xen/ldeb,0x300,w' ]
root = "hda1"
extra = "root=/dev/hda1 ro"
EOF

Boot the machine

Finally, if all is well:

# xm create -c /usr/pkg/etc/xen/ldeb

You should then be able to login as root. It's recommended to install libc6-xen, some other tweaks may be useful too. I'll be all ears :)

Update:

  • the disklabel is better now:
    • using "Linux Ext2" instead of "4.2BSD"
    • starting at offset 63
  • the flag to mke2fs was wrong (it's -I to specify the bytes per inode)
  • it's probably vnd0a instead of vnd0e
Come back...
Creative Commons License