Testing PAM modules and PAM-aware applications in the Matrix

Flattr this!

Jakub Hrozek and I are proud to announce the first release of pam_wrapper. This tool allows you to either simplify testing PAM modules or your application using PAM to authenticate users. PAM (Pluggable Authentication Modules) is a layer of abstraction on top of Unix authentication.

For testing PAM-aware applications we have written a simple PAM module called pam_matrix. If you plan to test a PAM module you can use the pamtest library we have implemented. It simplifies testing of modules. You can combine it with the cmocka unit testing framework or you can use the provided Python bindings to write tests for your module in Python.

Jakub and I have written an article for LWN.net to provide more details how to use it. You can find it here.

Now start testing your PAM module or application!

libssh is running in the Matrix now

Flattr this!

Since I joined the libssh project we started to write tests to find regression and make development easier. This has been achieved using the a unit testing framework called cmocka which I maintain and develop. The problem is that to run these tests you need to modify the sshd configuration and setup a test user so that the tests can be successfully executed. This is something contributors normally don’t do so we need to rely on our testing infrastructure.

In 2013 I’ve started the cwrap project. cwrap is a set of tool to make full network server/client testing made easy. These tools are used to make it possible to run the Samba Testsuite easily on every machine without setting anything up. Some time ago I’ve started to use cwrap for libssh testing. Finally I found the time to finish the task.

libssh in the Matrix

Now a libssh client tests sets up an artificial test environment. We have a passwd, shadow and group file so we can use two users to authenticate (nss_wrapper). sshd is running as the user starting the testcase but as it is part of the Matrix it thinks it is root (uid_wrapper). The client and server think they communicate on a real network (socket_wrapper) but it is again the Matrix!

It took me a while to get it working and I needed to implement new feature to the wrapper libraries of cwrap. socket_wrapper needed support to report TCP_NODELAY in getsockopt(). nss_wrapper needed shadow file support for password authentication so I had to add support for getspnam(). And as sshd is paranoid uid_wrapper needed checks if if is privileged to actually change to the user. After it drops privileged it checks if it really can’t go back.

With all of this implemented and new releases of the wrappers, which I’m preparing at the moment, all you have to do is to install cmocka, socket_wrapper, nss_wrapper and uid_wrapper and run ‘make test’. The Matrix will be created and libssh tested. You can find the cwrap libssh branch here.

There is one test for a feature missing right now. We do not test keyboard-interactive authentication, but the cwrap project is working on a new wrapper to fix this. Stay tuned!

Hunting down a fd closing bug in Samba

Flattr this!

In Samba I had a failing test suite. I have nss_wrapper compiled with debug messages turned on, so it showed me the following line:

NWRAP_ERROR(23052) - nwrap_he_parse_line: 3 Invalid line[TDB]: 'DB'

The file should parse a hosts file like /etc/hosts, but the debug line showed that it tried to parse a TDB (Trivial Database) file, Samba database backend. I’ve started to investigate it and wondered what was going on. This morning I called Michael Adam and we looked into the issue together. It was obvious that something closed the file descriptor for the hosts file of nss_wrapper and it was by Samba to open other files. The big question was, what the heck closes the fd. As socket_wrapper was loaded and it wraps the open() and close() call we started to add debug to the socket_wrapper code.

So first we added debug statements to the open() and close() calls to see when the fd was opened and closed. After that we wanted to see a stacktrace at the close() call to see what is the code path were it happens. Here is the code how to do this:

commit 6c632a4419b6712f975db390145419b008442865
Author:     Andreas Schneider 
AuthorDate: Thu Mar 26 11:07:38 2015 +0100
Commit:     Andreas Schneider 
CommitDate: Thu Mar 26 11:07:59 2015 +0100

    DEBUG stacktrace
---
 src/socket_wrapper.c | 37 +++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/src/socket_wrapper.c b/src/socket_wrapper.c
index 1188c4e..cb73cf2 100644
--- a/src/socket_wrapper.c
+++ b/src/socket_wrapper.c
@@ -80,6 +80,8 @@
 #include <rpc/rpc.h>
 #endif
 
+#include <execinfo.h>
+
 enum swrap_dbglvl_e {
 	SWRAP_LOG_ERROR = 0,
 	SWRAP_LOG_WARN,
@@ -303,8 +305,8 @@ static void swrap_log(enum swrap_dbglvl_e dbglvl,
 		switch (dbglvl) {
 			case SWRAP_LOG_ERROR:
 				fprintf(stderr,
-					"SWRAP_ERROR(%d) - %s: %s\n",
-					(int)getpid(), func, buffer);
+					"SWRAP_ERROR(ppid=%d,pid=%d) - %s: %s\n",
+					(int)getppid(), (int)getpid(), func, buffer);
 				break;
 			case SWRAP_LOG_WARN:
 				fprintf(stderr,
@@ -565,10 +567,35 @@ static int libc_bind(int sockfd,
 	return swrap.fns.libc_bind(sockfd, addr, addrlen);
 }
 
+#define BACKTRACE_STACK_SIZE 64
 static int libc_close(int fd)
 {
 	swrap_load_lib_function(SWRAP_LIBC, close);
 
+	if (fd == 21) {
+		void *backtrace_stack[BACKTRACE_STACK_SIZE];
+		size_t backtrace_size;
+		char **backtrace_strings;
+
+		SWRAP_LOG(SWRAP_LOG_ERROR, "fd=%d", fd);
+
+		backtrace_size = backtrace(backtrace_stack,BACKTRACE_STACK_SIZE);
+		backtrace_strings = backtrace_symbols(backtrace_stack, backtrace_size);
+
+		SWRAP_LOG(SWRAP_LOG_ERROR,
+			  "BACKTRACE %lu stackframes",
+			  (unsigned long)backtrace_size);
+
+		if (backtrace_strings) {
+			size_t i;
+
+			for (i = 0; i < backtrace_size; i++) {
+				SWRAP_LOG(SWRAP_LOG_ERROR,
+					" #%lu %s", i, backtrace_strings[i]);
+			}
+		}
+	}
+
 	return swrap.fns.libc_close(fd);
 }
 
@@ -704,6 +731,8 @@ static int libc_vopen(const char *pathname, int flags, va_list ap)
 
 	fd = swrap.fns.libc_open(pathname, flags, (mode_t)mode);
 
+	SWRAP_LOG(SWRAP_LOG_ERROR, "path=%s, fd=%d", pathname, fd);
+
 	return fd;
 }
 

We found out that the code responsible for this created a pipe() to communitcate with the child and then forked. The child called close() on the second pipe file descriptor. So when another fork happend in the child, the close() on the pipe file descriptor was called again and we closed a fd of the process to a tdb, connection or something like that. So initializing the pipe fd array with -1 and only calling close() if we have a file description which is not -1, fixed the problem.

If you need a better stacktrace you should use libunwind. However socket_wrapper can be a nice little helper to find bugs with file descriptors 😉

BUG: Samba standard process model closes random files when forking more than once

cmocka 1.0

Flattr this!

At the beginning of February I attended devconf.cz in Brno and the days before I had a hack week with Jakub Hrozek on cmocka. cmocka is a unit testing framework for C with support for mock objects.

We already rewrote the test runner last year and it was time to finish it and add support for several different message output formats. You are able to switch between cmocka standard output, Subunit, Test Anything Protocol and jUnit XML reports. In addition we we have a skip() function and test_realloc() to detect buffer overflows and memory leaks now.

You can find all other required information on the overhauled shiny new website: http://cmocka.org

New uid_wrapper with full threading support.

Flattr this!

Today I’ve released a new version of uid_wrapper (1.1.0) with full threading support. Robin Hack a colleague of mine spent a lot of time improving the code and writing tests for it. It now survives funny things like forking in a thread. We also added two missing functions and fixed several bugs. uid_wrapper is a tool to help you writing tests for your application.

If you don’t know uid_wrapper and wonder what you can do with it, here is an example:

$ id
uid=1000(asn) gid=100(users) groups=100(users),478(docker)
$ LD_PRELOAD=libuid_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 id
uid=0(root) gid=0(root) groups=0(root)

More details about uid_wrapper can be found on the cwrap project website, here.

Taking your bike on a plane

Flattr this!

Here is a totally computer unrelated post! Several people asked me how do I protect my bike to transport it safely on a plane. One possibility is to use a bike box, but the issue with a box is that airline personal likes big boxes, because they can use it to pile a lot of suitcases on it. I prefer to just wrap it with cardboard! Normally I go to a supermarket and ask if they have some spare cardboard for me. Normally they are happy to get rid of some. What you need bring from home in addition is a piece of rope, zip ties, duct tape, an old towel and a multitool or cutter.

I prepare everything at the supermarket. Cut the cardboard for the different pieces of the bike, put holes in the cardboard for the zip ties (first put duct tape on the cardboard then make the hole trough the duct tape and the cardboard!). Make sure you can still push the bike, the wheels should spin. In the end I have a small package like this:

Bike on a plane, cardboard collection

This small package is easy to transport. Either on the back of the bike or on your back 😉

At the airport you remove the pedals and fix the crank. Put the towel over the saddle and fix it with duct tape or a piece of rope. Tie a rope from the saddle to the handle bar so you can carry it. This makes it also easier for the airport personal to lift it. Then cover the bike with cardboard. Some parts are fixed on the bike with zip ties so they can’t move. In the end it normally looks like this:

Bike on a plane, protected with cardboard

If you’re on a bike trip you normally have 4 bike panniers with you, but the airline only allows to carry one luggage. Either the airport offers to wrap them or you go to a grocery store and buy plastfoil for food. It is very thin so you need 60m. It is not really Eco-friendly but normally you get it everywhere and it is cheap.

First start to wrap the two biggest panniers:

bike on a plane, panniers.

I use a rope to connect them and make sure not to loose one. After the two are in a good shape (ca. 25m) put the smaller panniers on the side and start to wrap the whole thing:

bike on a plane, wrapped panniers

Another possibility is to use 2 Ikea bags.

Have a safe trip!

resolv_wrapper 1.0.0 – the new cwrap tool

Flattr this!

I’ve released a new preloadable wrapper named resolv_wrapper which can be used for nameserver redirection or DNS response faking. It can be used in testing environment to route DNS queries to a real nameserver separate from resolv.conf or fake one with simple config file. We tested it on Linux, FreeBSD and Solaris. It should work on other UNIX flavors too.

resolv_wrapper

You can download resolv_wrapper here.

A talk about cwrap at LinuxCon Europe

Flattr this!

Next week is the LinuxCon Europe in Düsseldorf, Germany. I will be there and give a talk about cwrap, a set of tools to make client/server testing easy on a single machine. Testing network applications correctly is hard. This talk will demonstrate how to create a fully isolated network environment for client and server testing on a single host, complete with synthetic account information, hostname resolution, and privilege separation.

I hope you will attend my talk if you are there. If you can’t attend the LinuxCon Europe, but you’re going to the Linux Plumbers Conference then say hello and lets talk about cwrap there!

At the LinuxCon Europe I will announce new cool stuff and the website will be updated. So you should check

http://cwrap.org/

next week!

cwrap talk