The story of how qemu met MIPS and created netcat

Earlier this week I found myself in a predicament when I was reversing a stripped down MIPS embedded device. The device had minimal available memory and the only real executables on it were an even more stripped down busybox executable, tftp, and tcpdump. My goal was to obtain tcpdump logs being captured on device, but due to the lack of NFS support and the minimal available memory, I had very few options.

Initially I attempted to write the tcpdump logs to a file, and tftp them up to my local box. This became an issue when the log file became too large and all of the running processes started crashing. Trying to get my timing correct for the tftp command to execute, before everything went haywire, became a serious frustration. This is when I decided to go down a different route and try using everyone’s friend netcat. Unfortunately, I could not find a working netcat binary for the device, so I had to compile my own…

First, I downloaded every MIPS netcat I could find and tried to run them on the device. None worked – the output when trying to run them looked like:

# ./netcat
./netcat: line 1: syntax error: "(" unexpected

Which is extremely frustrating because that output looks like the shell is trying to run the netcat binary as a shell script. Anyway, after a few hours of failing, this is when I decided that I needed to compile my own netcat. This day long endeavor took much longer than I anticipated. I am writing this up in the hope that I can save someone else this frustration.

I knew I was going to build a Debian based MIPS system using qemu on Ubuntu 11.04. I found this tutorial that helped greatly:

Mistake: I assumed which architecture I needed. Make SURE you know which architecture and endianness you are going to be building against. There is a very easy trick to understand this. Pull a file of of the device, and run the ‘file’ command against it. In my case it looked like this:

$ file some_elf_binary
some_elf_binary: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), dynamically linked (uses shared libs), stripped

The “LSB” in that line stands for Least Significat Bit and means you should be building a MIPSEL virtual machine. If it says “MSB,” or Most Significant Bit you should build a MIPS virtual machine. I did not take the time to understand this, and now have a MIPS and MIPSEL virtual machine.

To reiterate – run ‘file’ on an existing executable
LSB = Create MIPSEL virtual machine
MSB = Create MIPS virtual machine

Creating the virtual machine with qemu is pretty easy, but time consuming.

First you must install the correct packages on Ubuntu (the ‘extra’ package provides the MIPS architecture):

sudo aptitude install qemu qemu-kvm-extras

Now you need to pull the initrd and vmlinux files from Debian for your virtual machine type. The initrd is the initial ramdisk and is used to load a temporary filesystem in memory. The vmlinux is the kernel that will be loaded into memory.

At the time of writing this, these links work for downloading the files for the respective architecture:
http://ftp.de.debian.org/debian/dists/stable/main/installer-mipsel/current/images/malta/netboot/
http://ftp.de.debian.org/debian/dists/stable/main/installer-mips/current/images/malta/netboot/

The next step is creating a file that will be your virtual hard disk.

qemu-img create -f raw hda.img 1G

Simple enough. Now to boot the initrd/vmlinux combination which will get you to the Debian installer.

NOTE: The ‘-M malta’ that relates to the ‘malta’ Debian images I downloaded. The tutorial I mentioned above uses ‘-M mips’ – this was a hanging point for me. When booting with ‘-M mips’ it just hung, no output, nothing.

MIPSEL:

qemu-system-mipsel -M malta -kernel vmlinux-2.6.26-2-4kc-malta -initrd initrd.gz -hda hda.img -append "root=/dev/ram console=ttyS0" -nographic

qemu-system-mips -M malta -kernel vmlinux-2.6.26-2-4kc-malta -initrd initrd.gz -hda hda.img -append "root=/dev/ram console=ttyS0" -nographic

Follow the instructions in the installer, and at the end it will tell you everything is completed and to remove the CD rom or boot media. At this point I waited for the VM to reboot into the installer again, and then had to kill the host’s ‘qemu-system’ process.

NOTE: There is a dialog that tells you that there is no boot loader, and you will have to append some arguments to the kernel. Make sure you take note what device it tells you (sometimes it will be /dev/hda1, others /dev/sda1)

If all went well, you should now be able to boot into the MIPS or MIPSEL OS. As explained above, my two installations had different root partitions that I needed to boot into. Another hanging point when creating my second MIPSEL VM.

MIPSEL:qemu-system-mipsel -M malta -kernel vmlinux-2.6.32-5-4kc-malta -hda hda.img -append root=/dev/sda1 -nographic
MIPS:qemu-system-mips -M malta -kernel vmlinux-2.6.32-5-4kc-malta -hda hda.img -append root=/dev/hda1 -nographic

Finally we have a MIPS VM shell. I will use netcat as an example of how I built a binary that worked on the end MIPS embedded system. First step is to obviously download the source.

Using wget to pull it down on the MIPS VM, I realized I had to install a compiler.

apt-get install gcc

worked just fine.

Mistake: I did not think about how I had to compile these applications. The end device is very stripped down, and who knows what libraries it even has on it. So after some trial and error I realized that I had to compile with static library linking. It has been a while since I have compiled something like this, so I found this that explains that all you need to do is add the ‘-static’ flag to the compiler arguments, prior to finding that link I asked in the IG chat room, and got a similar response:

[10/11/11 5:10:46 PM] wuntee: how do you force automake/autoconf/gcc whatever to do static librarys, vs dynamic?
[10/11/11 5:11:38 PM] Jeremy: -static
[10/11/11 5:11:39 PM] Sid: -static
[10/11/11 5:11:46 PM] Sid: ^ that
[10/11/11 5:11:46 PM] Jeremy: Zach, go on… say it
[10/11/11 5:11:49 PM] Zach twitches
[10/11/11 5:11:52 PM] Jeremy: -static
[10/11/11 5:11:57 PM] Jeremy: -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static -static
[10/11/11 5:11:57 PM] Sid: -static
[10/11/11 5:12:36 PM] Zach: “./configure –help”

RTFM… Thanks guys.

Mistake: There are MANY options you can pass to the compiler when compiling something for MIPS:. The output of the ‘file’ command on a working executable can be helpful here. In my case it says “ELF 32-bit LSB executable, MIPS, MIPS-II” which meant I wanted to also pass ‘-mips2′ to the compiler.

Configuring and making netcat was easy enough after figuring out all of the options:

CPPFLAGS="-mips2 -static" CFLAGS="-mips2 -static" ./configure && make

Running file on this looks close enough to the working executable, and it is statically linked like I wanted:

some_elf_binary: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), dynamically linked (uses shared libs), stripped
netcat: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), statically linked, for GNU/Linux 2.6.18, with unknown capability 0xf41 = 0x756e6700, with unknown capability 0x70100 = 0x1040000, stripped

You can then put the executable on the device, and it hopefully will run. One last thing you can do to the binary in order to conserve space is strip it. This can be done with the strip command.

After about a day of getting this working, the end netcat file was only 750k, and allowed me to obtain tcpdump output. Hope this helps someone.

wuntee

Post a comment or leave a trackback: Trackback URL.