Writing and reading code

Posted on 28th March 2013 in Development, KDE, Linux, Samba

flattr this!

You’ve probably heard that a developer of an established software project writes an average of 100 lines of code (loc) a day. I can say that this applies to me. So if you write 100 loc per day, how many do you read? I would estimate that the amount of time you spend on reading and understanding code is significantly more than on writing code. You probably also spend quite some time debugging code.

If you spend so much more time on reading and debugging code than writing code, shouldn’t you put more effort in writing clean and debuggable code? The Samba codebase is pretty old, more than 15 years now. I would say we have some experience with bad code and we have started to write much better and cleaner code, because we have wasted so much time trying to understand and debug code. However there is still room for improvement. Lets take a look at the following C code snippet.

if (!a) {
    return;
}

if (!b) {
    return;
}

if (!c) {
    return;
}

if (!*d) {
    return;
}

Can you guess from the code above what types of variables a, b, c and d are? The answer is no? Ok, lets take a look at the following code:

if (a == NULL) {
    return;
}

if (b == 0) {
    return;
}

if (c == false) {
    return;
}

if (d[0] == '\0') {
    return;
}

If you look at the code now, you can probably guess what types they are. Well not exactly which type, but in which superset they are. ‘a’ is a pointer, ‘b’ is an integer type, ‘c’ is a bool and ‘d’ is a string (char array). If you write code the way shown above, you don’t have to scroll up to find out as which type the variable is defined. Most of the time it is enough to know what type of superset you are checking and why.

Think about this: If you spend just a bit more time on writing clean code now, you will spend less time on reading, understanding and debugging the code later if you have to find a bug.

Lets look at some more best practices we do in the Samba code:

bool ok;
int rc;

/* bool return codes should always have the name 'ok' or start with 'is_' or 'do_' */
ok = fn_returning_a_bool();
if (!ok) {
    return;
}

/* We use rc or ret for an integer return code */
rc = do_something();
if (rc < 0) {
    return;
}

You can see that we have variables for the return codes and check them with an if-clause. The reason is that it is easy on the eyes and in a debugger you can simply print the return code variable. If you write it like this: if (do_something() < 0) You have a hard time in the debugger to find out the actual return code. You have to step into and through the function to get it. We allow the !ok syntax for bool types, cause ok is by definition in our code a bool.

To be continued ...

comments: 13 »

vim modelines in git config

Posted on 26th February 2013 in Development, KDE, Linux

flattr this!

I’m working on different Open Source projects and most of them have different coding style guidelines. Mostly spaces or tabs or different tabwidth. The easiest thing would be to store these information in the git config of the project. So here is a easy and secure way to have modelines in the git config.

So first I set the modelines (here for the Samba project):
git config --add vim.modeline "tabstop=8 shiftwidth=8 noexpandtab cindent"

Then copy this plugin into ~/.vim/plugin folder.

The modeline you defined in your git config will be appended to the :set command of vim. It only allows a limited set of set commands to be used. It will not execute any arbitrary code and you probably are the only person modifying the git config. You can change the allowed commands by adding the following to your ~/.vimrc file

    let g:git_modelines_allowed_items = [
                \ "textwidth",   "tw",
                \ "softtabstop", "sts",
                \ "tabstop",     "ts",
                \ "shiftwidth",  "sw",
                \ "expandtab",   "et",   "noexpandtab", "noet",
                \ "filetype",    "ft",
                \ "foldmethod",  "fdm",
                \ "readonly",    "ro",   "noreadonly", "noro",
                \ "rightleft",   "rl",   "norightleft", "norl",
                \ "cindent",     "cin",  "nocindent", "nocin",
                \ "smartindent", "si",   "nosmartindent", "nosi",
                \ "autoindent",  "ai",   "noautoindent", "noai",
                \ "spell",
                \ "spelllang"
                \ ]

UPDATE:
* New script which only allows a specified list
* Use sandbox command for set
* Added git repository
* Set only locally when reading the buffer

comments: 8 » tags: ,

cmocka – a unit testing framework for C

Posted on 14th January 2013 in Development, KDE, Linux

flattr this!

I’m a big fan of unit testing frameworks. When I developed csync, a bidirectional file synchronizer, I used check to write unit tests from the start. check was ok, but it were running valgrind on your testcases to find memleaks in your code the mode reports were about check. So I needed to add valgrind suppressions to get rid of them. When I started to work on libssh, a library implementing the SSH protocol, I wrote unit tests with check too. libssh is multi platform and also works on Windows and with Visual Studio. So we needed a new unit testing framework which is platform independent and has better code quality. I stumbled upon cmockery, a unit testing framework from Google. It was easy to use, the code looked good and it worked with Visual Studio. The build system sucked, so I added CMake support to produce a NSIS installer for Windows. I sent all my patches upstream but nothing happened. I fixed more bugs and added all patches people posted in their bug tracking system. I tried to talk with friends at Google, but in the end I needed to fork it.

cmocka is a fork and the successor of cmockery. I started to fix a lot of bugs, got all examples working and wrote API documentation with doxygen. The result is this first release version 0.2.0.

cmocka is a great unit testing framework with support for mock objects. Mock objects are simulated objects that mimic the behavior of real objects in a controlled way. Instead of calling the real objects, the tested object calls a mock object that merely asserts that the correct methods were called, with the expected parameters, in the correct order. It is really easy to write a unit test, take a look at the API an get started.

Example:

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>

/* A test case that does nothing and succeeds. */
static void null_test_success(void **state) {
(void) state; /* unused */
}
int main(void) {
const UnitTest tests[] = {
unit_test(null_test_success),
};
return run_tests(tests);
}

comments: 10 » tags: ,

Finally it is out, Samba 4.0.0!

Posted on 11th December 2012 in Uncategorized

flattr this!

We just released version 4.0.0 of Samba. About 10 years in development this is a huge milestone. It is the first version that can be a domain controller in an Active Directory domain!

comments: 0 »

Understanding Winbind

Posted on 8th November 2012 in Development, Linux, Samba

flattr this!

I recently fixed a bug resolving Domain Local groups in Winbind. I was asked how to reproduce it with a more complex setup, so I had to dig through the Winbind code to understand everything in more detail. I have documented my findings here, in order to retain what I’ve learned and to help others understand how Winbind works.

The Setup

We have a forest with two AD domains: level1.discworld.site and level2.discworld.site. The two domains have a two way trust. User accounts are created on LEVEL1, groups and machine accounts are on LEVEL2. We have a Linux machine named ‘linux’, with Winbind joined to LEVEL2. I will describe everything from the perspective of Winbind, so LEVEL2 inside of Winbind is also referred to as ‘own domain’.

Users:
LEVEL1\ab
LEVEL1\asn
LEVEL1\gd

Groups:
LEVEL2\samba (members: LEVEL1\ab, LEVEL2\asn, LEVEL2\gd)

Machine Accounts:
LEVEL2\linux$

Winbind Startup

Lets assume we have successfully joined the machine ‘linux’ to LEVEL2 and then start Winbind. There is a parent Winbind process which delegates work to Winbind children. The parent forks a child for each logical domain, so in this setup there are 4 domain child processes: LEVEL1, LEVEL2, BUILTIN and SAMBA (local SAM). LEVEL1 and LEVEL2 will connect to their corresponding AD domain controllers.

Querying information from AD domain controllers

If we want to obtain information about users or groups we have to query a Domain Controller. There are two ways to lookup this information. If the corresponding user is not logged in, then we queries for this information using the machine account. The machine account has limited permissions to query information, especially on Domain Controllers of trusted domains, so most of the time this information is incomplete, as we cannot provide more than what the AD domain controllers allow us to read. Often these queries are expensive, so caching is important to reduce the load on the domain controllers. Correct information about e.g. group memberships for a user is obtained when we authenticate as this user. The domain controller will then collect the information with the token of the user and send it to Winbind. In Winbind we cache this information. We have an issue here. If you get the information about a user with the machine account and cache it. Then authenticate as the user and get the
information again, we still return the information from the cache. Even if there is new or additional information, you will not see it until the cache expires and the authenticated user collects it again.

Authentication

If Winbind authenticates a user there are normally two routes. It could do a normal samlogon or a samlogon with kerberos. If you want to authenticate a user using samlogon you can do this using ‘wbinfo -a ‘, with kerberos ‘wbinfo -K ‘.

So if a login is initiated, the main Winbind process gets a pam authentication request. Depending to which domain the user belongs the auth is sent to the child handling the domain. So if LEVEL1+asn is trying to login, the Winbind child handling LEVEL1 will do a LogonSamLogon using the Netlogon PIPE to the domain controller. The domain controller is responsible for collecting all required information about the user and will return all information about group memberships in the info3 structure of the LogonSamLogon response.
If Kerberos is involved the Winbind child handling LEVEL1 will authenticate the user talking to the KDC of the domain controller. All information will be stored in the PAC (Privilege Attribute Certificate) of the Kerberos ticket (which is similar to the info3 structure in the LogonSamLogon response).

id and getent

Information retrieval for ‘id’ or ‘getent’ will take a special route if we are only able to gather it using the machine account. The results collected with the machine account can differ from the results obtained during user authentication. Normally the information sent back from the Domain Controller during authentication is much more detailed and complete. It is possible the results differ between querying information as a user and as a machine account. With the limited resources of a machine account we can only try to get the basic information from the domain controller that the user is a member of. We will not contact trusted domains as enumeration is expensive and often not allowed with a machine account.

Lets look which functions will be called in Winbind if a user runs the command ‘id LEVEL1+asn’. For ‘id’ to be working we assume that nsswitch has been correctly configured to talk to Winbind. There is a libnss_winbind.so module which talks to the parent Winbind process over a unix pipe. The parent Winbind process handles all nsswitch function calls (POSIX functions) coming over the pipe asynchronously. We not discuss id mapping here, it will get too complex, we will just look on the flow of information.

‘id LEVEL1+asn’ will calls several POSIX functions which are sent over the UNIX pipe to the main Winbind process. These functions are getpwnam, getgrgid and getgroups. We assume that we have cold caches and need to handle these requests using machine account privileges.

getpwnam

The first thing ‘id’ calls is the getpwnam function. This will retrieve basic information about the user like the primary group id, the home directory and shell. The main Winbind process sends three queries to the LEVEL1 child for this. lookupname to get the SID of the user, a second lookupname to translate the SID to the username for verification, and finally a QueryUser call to get the basic information (primary gid, …). The first lookupname is a lsa_LookupNames call to the DC’s LSA server. The second is a LookupSids call to translate the SID to a name again. The QueryUser command is a LDAP query. Normally we always try to get the information with the fastest method and fall back to slower mechanisms if that fails.
After we collect all information and also store them in the cache (the child processes are responsible for the caches) we return the information to nsswitch. Now the ‘id’ command needs to get the name for the primary group and calls ‘getgrgid 1000000′ (1000000 being the gid of the primary group).

getgrgid 1000000

The parent Winbind will connect to the LEVEL1 domain child and call lookupname. LEVEL1 will then connect over RPC to the LSA pipe and call LookupSids3 to translate the SID to the name (the idmapping knows about the SID for the gid, the details are left out here).
As we have the important user information it is time to ask for additional group memberships of the user. This results in the following call:

getgroups LEVEL1+asn

The request is received by the main Winbind process, which needs to resolve the groups on three domains. It is always the same even if the machine is joined to a different domain the user is a member of.

a) The domain the user is a member of (LEVEL1)
b) The local SAM Authority (SAMBA)
c) The BUILTIN domain

We will need the SID of the user first so we ask the LEVEL1 child to resolve the name to a SID. Then for each of the domains we ask the corresponding child to do a LookupUserGroups. The LEVEL1 child will do a LDAP search to get a list of SIDs the user is a member of. Then it will talk to the DC LSA Server and call LookupSids3() to translate the SID into a name for each group. The information is sent back to the parent which will ask the local domain (SAMBA) if there are any aliases that the user is a member of. It will send a LookupUserAliases to the SAMBA child which will lookup the information using pdb. The final step is to talk to the BUILTIN domain for user aliases.

After all of the above POSIX calls were successful id will print the information it collected.

If you login as the user using kerberos first, then the information about the user are cached by the domain child serving the users domain. If you now call getpwnam then query will be filled with the information stored in the cache. Only the SID to name translation requires a LookupSids3 LSA call to the DC if it is not cached yet. The same for get getgrgid or getgroups call. We already got the information from the DC in the PAC which groups the user is a member of. We just need to translate the SIDs to names.

To be continued …

comments: 4 » tags:

How to create a SuplRootCert for supl.google.com

Posted on 24th July 2012 in Android, Linux

flattr this!

Back to these bad GPS fixes for Android spread all over the net. This time I will describe how to find out which is the correct SSL root certificate you need and how to create it. So first we need know the root certificate the Google SUPL server has been signed. There are several ways but we use the easiest. Connect with the openssl binary to the SUPL server. This can be done by the following command:


$ openssl s_client -connect supl.google.com:7275

The output you will see will include the following relevant part at the beginning:


CONNECTED(00000003)
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=supl.google.com
i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
2 s:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---

This means the server certificate of supl.google.com is signed by the “Thawte Premium Server CA” root certificate. So you need to go and find that. Each SSL certificate issuer has a site with all it root certificates available. You can find the root certificates for Thawte here:

https://www.thawte.com/roots/

Now download the “Root 2 Thawte Premium Server CA” certificate. You will get the certificate in the PEM format. The SuplRootCert file on Android is stored in the DER format so we need to convert it. This can be done with the following command:

openssl x509 -inform PEM -in thawte_Premium_Server_CA.pem -outform DER -out SuplRootCert

Now you have the correct SuplRootCert you can put in /system/etc and use with the gps.conf. If you have a SuplRootCert and you want to find out which root certificate it is you can do it with the following command:

openssl x509 -inform DER -in SuplRootCert -text

REMEMBER: This works only on vendor ROMs which provide SUPL TLS support. I didn’t see any CyanogenMod ROM with TLS SUPL support yet!!!

comments: 5 »

Android, CyanogenMod and the GPS fix

Posted on 12th July 2012 in Android, Linux

flattr this!

There are a lot of people around providing gps fixes for Android. Most of them don’t really help at all. I want to explain here what variables the code recognizes and what they do. The other thing is SuplCert which is the certificate to verify the ssl connection for the supl server. Most of the time poeple provide the wrong certificate.

gps.conf

There are two locations in the code which read the gps.conf file. GpsLocationProvider.java [1] GpsXtraDownloader.java [2] and loc_eng_cfg.cpp [3]. The GpsLocationProvider reads the following variables from the config:

    SUPL_HOST
    SUPL_PORT

    C2K_HOST
    C2K_PORT

GpsXtraDownloader.java

    XTRA_SERVER_1
    XTRA_SERVER_2
    XTRA_SERVER_3

This is pretty obvious. It then calls a function from libloc_api called loc_eng_set_server() to set the agps server. libloc_api itself reads also the gps.conf to get the values for the following variables:

    INTERMEDIATE_POS (Turn on intermediate position)
    ACCURACY_THRES   (Turn-off  intermediate positions outside required accuracy)
    ENABLE_WIPER     (The code only reads this variable but it doesn't do anything)
    DEBUG_LEVEL      (0-5, default: 4)

SuplCert

In most of the flashable gps fixes you find a SuplCert. Most of them I looked at have the wrong certificate. They include a Verisign certificate but supl.google.com is signed by Thawte.

supl_google_com_request

This is a SSL encrypted supl request to supl.google.com on port 7275.

However as you can see above. There is no support for a TLS supl request in the code. It doesn’t make sense to add the host, port and certificate in CyanogenMod.

Supl Protocol

For supl you need to have a cell connection. You ask the radio for the cell parameters and send it to the supl server. The supl server will return the gps coordinates of the tower or an estimation of your location.

You can get the cell id information with the following AT commands:

       $ AT+COP?
       +COPS: 0,2,"24405",2
       OK
       $ AT+CREG=2
       OK
       $ AT+CREG?
       +CREG: 2,1,"59E2","31B0"
       OK
       $ AT+CREG=0
       OK

       This results in:

       MCC = 244     (Mobile Country Code)
       MCN = 05      (Mobile Country Network)
       LAC = 0x59e2  (Local Area Code)
       CI  = 0x31b0  (Cell Identifier)

Conclusion

The default gps.conf by CyanogenMod provides all needed variables to have supl support for a faster GPS fix.

With DEBUG_LEVEL=5 you normally see the following lines:

    D/libloc  (  169): loc_eng_init created client, id = 1
    D/libloc  (  169): loc_eng_ni_init: entered.
    D/libloc  (  169): loc_eng_set_server, type = 1, hostname = supl.google.com, port = 7276
    D/libloc  (  169): loc_eng_set_server, addr = supl.google.com:7276
    D/libloc  (  169): loc_eng_ioctl called: client = 1, ioctl_type = RPC_LOC_IOCTL_SET_UMTS_SLP_SERVER_ADDR
    D/RPC     (  169): 3000008c:00020002 call success.
    D/RPC     (  169): 3000008c:00020002 sending call (XID 2).
    D/RPC     (  169): 3000008c:00020002 received REPLY (XID 2), grabbing mutex to wake up client.

[1] frameworks/base/services/java/com/android/server/location/GpsLocationProvider.java
[2] frameworks/base/services/java/com/android/server/location/GpsXtraDownloader.java
[2] hardware/qcom/gps/loc_api/libloc_api/loc_eng_cfg.cpp

comments: 12 »

CyanogenMod 9 for HTC Wildfire S (ALPHA3)

Posted on 21st June 2012 in Android, Linux

flattr this!

Maybe you already know that I worked since January 2012 on porting CM9 to the HTC Wildfire S (Marvel). It took quite a while to figure out how Android is working and how to get the the OS correctly talking with the hardware. Today I’ve released ALPHA3 of my work which is a pretty stable version. The Wildfire S is not a supported Android 4.0 device, so you have to workaround a lot of things and use old binary blobs. This also means there will probably never be an official CM9 release for the Wildfire S. However most of the stuff is working pretty well and you can live with the known bugs.

Known bugs:

  • Camera isn’t fully working, you can take pictures without problems.
  • OMX codecs aren’t fully working cause they are too old. We asked Quallcom to release new OMX codecs for ARMv6 but they said it will not be possible to get it working with QDSP5.
  • The device reboots if you turn blootooth on sometimes and bluetooth drains the battery.

You can download ALPHA3 of the ROM here.

Have fun.

comments: 24 » tags: ,

CM9 (Android 4.0 ICS) and deep sleep

Posted on 6th May 2012 in Linux

flattr this!

I’ve had the problem that the device didn’t want to switch into deep sleep mode if radio was on. What is deep sleep? To make it simple we break it down. Your device has 3 modes. The fisrst is “Screen On and Awake”, “Awake” and “Deep Sleep”. If you use your device it you’re in the first mode and you need a obviously a lot of battery. The second “Awake” means it is doing some background work. Checking for calls, checking Emails, syncing contacts. The last one means it goes for some time into a mode were it uses almost no battery, and this is Deep Sleep. If you don’t do anything and your phone is in your pocket you want that it is in the Deep Sleep mode most of the time.

My HTC Wildfire S didn’t want to go into the “Deep Sleep” mode at all if radio was turned on. It worked with Airplane mode. I thought this has something todo with RIL but I was wrong. Actually it was a bluetooth wakelock. The wakelock “msm_serial_hs_dma” was held all the time. The problem is that the msm7227 platform doesn’t supports quick switch-on/off of the bluetooth module and you need to deactivate it with an overlay else ICS always tries to trigger it.

So adding

<bool name="config_bluetooth_adapter_quick_switch">false</bool>

to overlay/frameworks/base/core/res/res/values/config.xml fixed the problem and the wakelock was gone.

comments: 1 » tags: , ,

libhtc_ril.so and segfaults

Posted on 23rd April 2012 in Android, Linux

flattr this!

If you try to get a new Android version, in this case CyanogenMod9, working on your old phone you have to deal with binary blobs. One of these blobs is the library talking to the radio, libhtc_ril.so.

I wanted to document what I learned about libhtc_ril.so. I’ve wanted to get the library version matching my baseband version working with cm9. This resulted it several segfaults. So I’ve started to strace the rild process to find what’s going wrong, which permissions are missing etc. The library doesn’t check return values so it segfaults. One of these segfaults was a missing kernel interface called usb_function_switch. The file should be in /sys/devices/platform/msm_hsusb/usb_function_switch. I’ve implemented that function in the kernel and it still segfaulted and I had no idea what to do now. Today I analyzed the RADIO logs and stumpled upon:

D/RILJ    (  328): [0100]> SCREEN_STATE: false
D/HTC_RIL ( 1360): ril_func_screen_state_notified():called
D/HTC_RIL ( 1360): ril_func_screen_state_notified():Not found 'ether:' in USB_STATE_PATH

As it segfaulted directly after closing /sys/devices/platform/msm_hsusb/usb_function_switch it smelled like it expeced to have something like:

ether:disable

I’ve dived into the code and found out that in my kernel tree it was called rndis and in the htc kernel tree it was called ether. So I’ve fixed that and added the other values of /sys/devices/platform/msm_hsusb/usb_function_switch it started to work just fine. I hope this post will help other developers with similar problems.

This is the full set of the usb_function_switch:

ether:disable
accessory:disable
usb_mass_storage:enable
adb:enable
cdc_ethernet:disable
diag:disable
modem:disable
serial:disable
comments: 0 » tags: , ,
« Previous