This FAQ has not been updated for a while and may not be 100% up to date.
The glibc mailing lists
often contain more up-to-date information.
Also note that LinuxThreads is no longer developed and has been replaced
by NPTL,
the Native POSIX Threads Library for Linux, which fixes many
shortcomings of LinuxThreads and offers better performance.
LinuxThreads is a Linux library for multi-threaded programming.
It implements the Posix 1003.1c API (Application Programming
Interface) for threads. It runs on any Linux system with kernel 2.0.0
or more recent, and a suitable C library (see section C).
A thread is a sequential flow of control through a program.
Multi-threaded programming is, thus, a form of parallel programming
where several threads of control are executing concurrently in the
program. All threads execute in the same memory space, and can
therefore work concurrently on shared data.
Multi-threaded programming differs from using multiple Unix processes in
that all threads share the same memory space (and a few other system
resources, such as file descriptors), instead of running in their own
memory space as is the case with Unix processes.
Threads are useful for several reasons. First, they allow a program to
exploit multi-processor machines: the threads can run in parallel on
several processors, allowing a single program to divide its work
between several processors, thus running faster than a single-threaded
program, which runs on only one processor at a time.
Second, even on uniprocessor machines, threads allow overlapping I/O
and computations in a simple way.
Last, some programs are best expressed as several threads of control
that communicate together, rather than as one big monolithic
sequential program. Examples include server programs, overlapping
asynchronous I/O, and graphical user interfaces.
It's an API for multi-threaded programming standardized by IEEE as
part of the POSIX standards. Most Unix vendors have endorsed the
POSIX 1003.1c standard. Implementations of the 1003.1c API are
already available under Sun Solaris 2.5, Digital Unix 4.0,
Silicon Graphics IRIX 6, and should soon be available from other
vendors such as IBM and HP. More generally, the 1003.1c API is
replacing relatively quickly the proprietary threads library that were
developed previously under Unix, such as Mach cthreads, Solaris
threads, and IRIX sprocs. Thus, multithreaded programs using the
1003.1c API are likely to run unchanged on a wide variety of Unix
platforms.
LinuxThreads is currently distributed and maintained as part of glibc,
but no longer actively developed. Development continues on NPTL, which is a new
implementation of POSIX threads for Linux with many advantages over
LinuxThreads. It is expected that LinuxThreads will stop being maintained
starting with version 2.4 of glibc.
LinuxThreads implements almost all of Posix 1003.1c, as well as a few
extensions. The only part of LinuxThreads that does not conform to
Posix is signal handling (see section J). NPTL
improves significantly over LinuxThreads both on standard conformance
and on performance.
I post LinuxThreads-related announcements on the newsgroup
comp.os.linux.announce,
and also on the mailing list
linux-threads@magenet.com.
You can subscribe to the latter by writing
majordomo@magenet.com.
For questions about programming with POSIX threads in general, use
the newsgroup
comp.programming.threads.
Be sure you read the
FAQ
for this group before you post.
If you're using glibc 2, the best way by far is to use the
glibcbug script to mail a bug report to the glibc
maintainers.
If you're using an older libc, or don't have the glibcbug
script on your machine, then e-mail me directly
(Xavier.Leroy@inria.fr).
In both cases, before sending the bug report, make sure that it is not
addressed already in this FAQ. Also, try to send a short program that
reproduces the weird behavior you observed.
Unfortunately, no. POSIX standards are copyrighted by IEEE, and
IEEE does not distribute them freely. You can buy paper copies from
IEEE, but the price is fairly high ($120 or so). If you disagree with
this policy and you're an IEEE member, be sure to let them know.
On the other hand, you probably don't want to read the standard. It's
very hard to read, written in standard-ese, and targeted to
implementors who already know threads inside-out. A good book on
POSIX threads provides the same information in a much more readable form.
I can personally recommend Dave Butenhof's book, Programming
with POSIX threads (Addison-Wesley). Butenhof was part of the
POSIX committee and also designed the Digital Unix implementations of
POSIX threads, and it shows.
Another good source of information is the X/Open Group Single Unix
specification which is available both
on-line
and as a
book and CD/ROM.
That specification includes pretty much all the POSIX standards,
including 1003.1c, with some extensions and clarifications.
The best choice by far is glibc 2, a.k.a. libc 6. It offers very good
support for multi-threading, and LinuxThreads has been closely
integrated with glibc 2. The glibc 2 distribution contains the
sources of a specially adapted version of LinuxThreads.
glibc 2 comes preinstalled as the default C library on most Linux
distributions nowadays, such as RedHat 5 and up, and Debian 2 and up.
Those distributions include the version of LinuxThreads matching
glibc 2.
Yes, but you're likely to run into some problems, as libc 5 only
offers minimal support for threads and contains some bugs that affect
multithreaded programs.
The versions of libc 5 that work best with LinuxThreads are
libc 5.2.18 on the one hand, and libc 5.4.12 or later on the other hand.
Avoid 5.3.12 and 5.4.7: these have problems with the per-thread errno
variable.
I'd recommend you switch to glibc 2. Even for single-threaded
programs, glibc 2 is more solid and more standard-conformant than libc
5. And the shortcomings of libc 5 almost preclude any serious
multi-threaded programming.
Switching an already installed
system from libc 5 to glibc 2 is not completely straightforward.
See the Glibc2
HOWTO for more information. Much easier is (re-)installing a
Linux distribution based on glibc 2, such as RedHat 6.
You must transfer the whole glibc sources, then drop the LinuxThreads
sources in the linuxthreads/ subdirectory, then recompile
glibc as a whole. There are now too many inter-dependencies between
LinuxThreads and glibc 2 to allow separate re-compilation of LinuxThreads.
libc_r/dirent.c:94: structure has no member named `dd_lock'
I haven't actually seen this problem, but several users reported it.
My understanding is that something is wrong in the include files of
your Linux installation (/usr/include/*). Make sure
you're using a supported version of the libc 5 library. (See question C.2).
Yes, /usr/include/sched.h that comes with libc 5.3.12 is broken.
Replace it with the sched.h file contained in the
LinuxThreads distribution. But really you should not be using libc
5.3.12 with LinuxThreads! (See question C.1.)
Due to the general "one process per thread" model, there's one process
for the initial thread and N processes for the threads it created
using pthread_create. That leaves one process
unaccounted for. That extra process corresponds to the "thread
manager" thread, a thread created internally by LinuxThreads to handle
thread creation and thread termination. This extra thread is asleep
most of the time.
That behavior has mostly disappeared in recent releases of
LinuxThreads (version 0.8 and up). It was fairly common in older
releases, though.
What happens in LinuxThreads 0.7 and before is the following: when a
thread unlocks a mutex, all other threads that were waiting on the
mutex are sent a signal which makes them runnable. However, the
kernel scheduler may or may not restart them immediately. If the
thread that unlocked the mutex tries to lock it again immediately
afterwards, it is likely that it will succeed, because the threads
haven't yet restarted. This results in an apparently very unfair
behavior, when the same thread repeatedly locks and unlocks the mutex,
while other threads can't lock the mutex.
In LinuxThreads 0.8 and up, pthread_unlock restarts only
one waiting thread, and pre-assign the mutex to that thread. Hence,
if the thread that unlocked the mutex tries to lock it again
immediately, it will block until other waiting threads have had a
chance to lock and unlock the mutex. This results in much fairer
scheduling.
Notice however that even the old "unfair" behavior is perfectly
acceptable with respect to the POSIX standard: for the default
scheduling policy, POSIX makes no guarantees of fairness, such as "the
thread waiting for the mutex for the longest time always acquires it
first". Properly written multithreaded code avoids that kind of heavy
contention on mutexes, and does not run into fairness problems. If
you need scheduling guarantees, you should consider using the
real-time scheduling policies SCHED_RR and
SCHED_FIFO, which have precisely defined scheduling
behaviors.
Again, this behavior is characteristic of old releases of LinuxThreads
(0.7 and before); more recent versions (0.8 and up) should not exhibit
this behavior.
The reason for this behavior is explained in
question D.6 above: printf() performs
locking on stdout, and thus your two threads contend very
heavily for the mutex associated with stdout. But if you
do some real work between two calls to printf(), you'll
see that scheduling becomes much smoother.
Nope. That's the way it should be. The closing brace is provided by
the pthread_cleanup_pop macro. The POSIX standard
requires pthread_cleanup_push and
pthread_cleanup_pop to be used in matching pairs, at the
same level of brace nesting. This allows
pthread_cleanup_push to open a block in order to
stack-allocate some data structure, and
pthread_cleanup_pop to close that block. It's ugly, but
it's the standard way of implementing cleanup handlers.
Versions of LinuxThreads prior to 0.8 are susceptible to ``livelocks''
(one thread loops, consuming 100% of the CPU time) in conjunction with
real-time scheduling. Since real-time threads and processes have
higher priority than normal Linux processes, all other processes on
the machine, including the shell, the X server, etc, cannot run at all
and the machine appears frozen.
Each thread, from the kernel's standpoint, is one process. Stock
Linux kernels (version 2.2 and earlier) are limited to at most 512
processes for the super-user, and half this number for regular users.
This can be changed by changing NR_TASKS in
include/linux/tasks.h and recompiling the kernel. On the
x86 processors at least, architectural constraints seem to limit
NR_TASKS to 4090 at most. (It seems that 2.4 kernels
have higher limits, though.)
LinuxThreads contains a table of all active threads. This table
has room for 1024 threads at most. To increase this limit, you must
change PTHREAD_THREADS_MAX in the LinuxThreads/glibc sources
and recompile.
By default, each thread reserves 2M of virtual memory space for
its stack. This space is just reserved; actual memory is allocated
for the stack on demand. But still, on a 32-bit processor, the total
virtual memory space available for the stacks is on the order of 1G,
meaning that more than 500 threads will have a hard time fitting in.
You can overcome this limitation by moving to a 64-bit platform, or by
allocating smaller stacks yourself using the setstackaddr
attribute.
Finally, the Linux kernel contains many algorithms that run in
time proportional to the number of process table entries. Increasing
this number drastically will slow down the kernel operations
noticeably.
(Other POSIX threads libraries have similar limitations, by the way.)
For all these reasons, you'd better restructure your application so
that it doesn't need more than, say, 100 threads. For instance,
in the case of a multithreaded server, instead of creating a new
thread for each connection, maintain a fixed-size pool of worker
threads that pick incoming connection requests from a queue.
Because it's not part of the (final) POSIX 1003.1c standard.
Several drafts of the standard contained pthread_yield(),
but then the POSIX guys discovered it was redundant with
sched_yield() and dropped it. So, just use
sched_yield() instead.
No, I didn't. What you're describing is draft 4 of the POSIX
standard, which is used in OSF DCE threads. LinuxThreads conforms to the
final standard. Even though the functions have the same names as in
draft 4 and DCE, their calling conventions are slightly different. In
particular, attributes are passed by reference, not by value, and
default attributes are denoted by the NULL pointer. Since draft 4/DCE
will eventually disappear, you'd better port your program to use the
standard interface.
POSIX did it. The thr_* functions correspond to Solaris
threads, an older thread interface that you'll find only under
Solaris. The pthread_* functions correspond to POSIX
threads, an international standard available for many, many platforms.
Even Solaris 2.5 and later support the POSIX threads interface. So,
do yourself a favor and rewrite your code to use POSIX threads: this
way, it will run unchanged under Linux, Solaris, and quite a lot of
other platforms.
The POSIX standard provides no mechanism by which a thread A can
suspend the execution of another thread B, without cooperation from B.
The only way to implement a suspend/restart mechanism is to have B
check periodically some global variable for a suspend request
and then suspend itself on a condition variable, which another thread
can signal later to restart B.
Notice that thr_suspend() is inherently dangerous and
prone to race conditions. For one thing, there is no control on where
the target thread stops: it can very well be stopped in the middle of
a critical section, while holding mutexes. Also, there is no
guarantee on when the target thread will actually stop. For these
reasons, you'd be much better off using mutexes and conditions
instead. The only situations that really require the ability to
suspend a thread are debuggers and some kind of garbage collectors.
If you really must suspend a thread in LinuxThreads, you can send it a
SIGSTOP signal with pthread_kill. Send
SIGCONT for restarting it.
Beware, this is specific to LinuxThreads and entirely non-portable.
Indeed, a truly conforming POSIX threads implementation will stop all
threads when one thread receives the SIGSTOP signal!
One day, LinuxThreads will implement that behavior, and the
non-portable hack with SIGSTOP won't work anymore.
These optional functions are provided in recent versions of
LinuxThreads (0.8 and up). Earlier releases did not provide these
optional components of the POSIX standard.
Even if pthread_attr_setstacksize() and
pthread_attr_setstackaddr() are now provided, we still
recommend that you do not use them unless you really have strong
reasons for doing so. The default stack allocation strategy for
LinuxThreads is nearly optimal: stacks start small (4k) and
automatically grow on demand to a fairly large limit (2M).
Moreover, there is no portable way to estimate the stack requirements
of a thread, so setting the stack size yourself makes your program
less reliable and non-portable.
With a "one-to-one" model, as in LinuxThreads (one kernel execution
context per thread), there is only one scheduler for all processes and
all threads on the system. So, there is no way to obtain the behavior of
PTHREAD_SCOPE_PROCESS.
This is another optional component of the POSIX standard. Portable
applications should test _POSIX_THREAD_PROCESS_SHARED
before using this facility.
The goal of this extension is to allow different processes (with
different address spaces) to synchronize through mutexes, conditions
or semaphores allocated in shared memory (either SVR4 shared memory
segments or mmap()ed files).
The reason why this does not work in LinuxThreads is that mutexes,
conditions, and semaphores are not self-contained: their waiting
queues contain pointers to linked lists of thread descriptors, and
these pointers are meaningful only in one address space.
Matt Messier and I spent a significant amount of time trying to design a
suitable mechanism for sharing waiting queues between processes. We
came up with several solutions that combined two of the following
three desirable features, but none that combines all three:
allow sharing between processes having different UIDs
supports cancellation
supports pthread_cond_timedwait
We concluded that kernel support is required to share mutexes,
conditions and semaphores between processes. That's one place where
Linus Torvalds's intuition that "all we need in the kernel is
clone()" fails.
Until suitable kernel support is available, you'd better use
traditional interprocess communications to synchronize different
processes: System V semaphores and message queues, or pipes, or sockets.
Douglas Schmidt's ACE library contains, among a lot of other
things, C++ wrappers for LinuxThreads and quite a number of other
thread libraries. Check out
http://www.cs.wustl.edu/~schmidt/ACE.html
You're probably trying to pass a class member function or some
other C++ thing as third argument to pthread_create().
Recall that pthread_create() is a C function, and it must
be passed a C function as third argument.
From what I understand, thread support in libg++ is completely broken,
especially with respect to locking of iostreams. H.J.Lu wrote:
If you want to use thread, I can only suggest egcs and glibc. You
can find egcs at
http://www.cygnus.com/egcs.
egcs has libsdtc++, which is MT safe under glibc 2. If you really
want to use the libg++, I have a libg++ add-on for egcs.
Yes, but not with the stock gdb 4.17. You need a specially patched
version of gdb 4.17 developed by Eric Paire and colleages at The Open
Group, Grenoble. The patches against gdb 4.17 are available at
http://pauillac.inria.fr/~xleroy/linuxthreads/gdb-4.17-debug-threads.patch.gz.
H.J.Lu also develops patches for gdb 4.17 that include thread support
and more; those are available at
ftp://ftp.valinux.com/pub/support/hjl/gdb/.
More recent versions of gdb seem to include LinuxThreads support.
Some Linux distributions provide an already-patched version of gdb;
others don't. For instance, the gdb in RedHat 5.2 and RedHat 6.2 is
thread-aware,
but apparently not the one in RedHat 6.0. Just ask (politely) the
makers of your Linux distributions to please make sure that they apply
the correct patches to gdb.
Not very well. Generally, the core file does not correspond to the
thread that crashed. The reason is that the kernel will not dump core
for a process that shares its memory with other processes, such as the
other threads of your program. So, the thread that crashes silently
disappears without generating a core file. Then, all other threads of
your program die on the same signal that killed the crashing thread.
(This is required behavior according to the POSIX standard.) The last
one that dies is no longer sharing its memory with anyone else, so the
kernel generates a core file for that thread. Unfortunately, that's
not the thread you are interested in.
Assertions and printf() are your best friends. Try to debug
sequential parts in a single-threaded program first. Then, put
printf() statements all over the place to get execution traces.
Also, check invariants often with the assert() macro. In truth,
there is no other effective way (save for a full formal proof of your
program) to track down concurrency bugs. Debuggers are not really
effective for subtle concurrency problems, because they disrupt
program execution too much.
The include files define prototypes for the reentrant variants of
some of the standard library functions,
e.g. gethostbyname_r() as a reentrant equivalent to
gethostbyname().
If _REENTRANT is defined, some
<stdio.h> functions are no longer defined as macros,
e.g. getc() and putc(). In a multithreaded
program, stdio functions require additional locking, which the macros
don't perform, so we must call functions instead.
More importantly, <errno.h> redefines errno when
_REENTRANT is
defined, so that errno refers to the thread-specific errno location
rather than the global errno variable. This is achieved by the
following #define in <errno.h>:
#define errno (*(__errno_location()))
which causes each reference to errno to call the
__errno_location() function for obtaining the location
where error codes are stored. libc provides a default definition of
__errno_location() that always returns
&errno (the address of the global errno variable). Thus,
for programs not linked with LinuxThreads, defining
_REENTRANT makes no difference w.r.t. errno processing.
But LinuxThreads redefines __errno_location() to return a
location in the thread descriptor reserved for holding the current
value of errno for the calling thread. Thus, each thread operates on
a different errno location.
If all threads were to store error codes in the same, global errno
variable, then the value of errno after a system call or library
function returns would be unpredictable: between the time a system
call stores its error code in the global errno and your code inspects
errno to see which error occurred, another thread might have stored
another error code in the same errno location.
Lots of trouble. If the code uses getc() or
putc(), it will perform I/O without proper interlocking
of the stdio buffers; this can cause lost output, duplicate output, or
just crash other stdio functions. If the code consults errno, it will
get back the wrong error code. The following code fragment is a
typical example:
do {
r = read(fd, buf, n);
if (r == -1) {
if (errno == EINTR) /* an error we can handle */
continue;
else { /* other errors are fatal */
perror("read failed");
exit(100);
}
}
} while (...);
Assume this code is not compiled with -D_REENTRANT, and
linked with LinuxThreads. At run-time, read() is
interrupted. Since the C library was compiled with
-D_REENTRANT, read() stores its error code
in the location pointed to by __errno_location(), which
is the thread-local errno variable. Then, the code above sees that
read() returns -1 and looks up errno. Since
_REENTRANT is not defined, the reference to errno
accesses the global errno variable, which is most likely 0. Hence the
code concludes that it cannot handle the error and stops.
The short answer is: because the Linux kernel you're using does not
support realtime signals.
LinuxThreads needs two signals for its internal operation.
One is used to suspend and restart threads blocked on mutex, condition
or semaphore operations. The other is used for thread
cancellation.
On ``old'' kernels (2.0 and early 2.1 kernels), there are only 32
signals available and the kernel reserves all of them but two:
SIGUSR1 and SIGUSR2. So, LinuxThreads has
no choice but use those two signals.
On recent kernels (2.2 and up), more than 32 signals are provided in
the form of realtime signals. When run on one of those kernels,
LinuxThreads uses two reserved realtime signals for its internal
operation, thus leaving SIGUSR1 and SIGUSR2
free for user code. (This works only with glibc, not with libc 5.)
Yes, you can -- if you're very careful. The stacks are indeed visible
from all threads in the system. Some non-POSIX thread libraries seem
to map the stacks for all threads at the same virtual addresses and
change the memory mapping when they switch from one thread to
another. But this is not the case for LinuxThreads, as it would make
context switching between threads more expensive, and at any rate
might not conform to the POSIX standard.
So, you can take the address of an "auto" variable and pass it to
other threads via shared data structures. However, you need to make
absolutely sure that the function doing this will not return as long
as other threads need to access this address. It's the usual mistake
of returning the address of an "auto" variable, only made much worse
because of concurrency. It's much, much safer to systematically
heap-allocate all shared data structures.
That's a prime example of the errno problem described in question H.2. The binaries for Xlib you're using have not been
compiled with -D_REENTRANT. It happens Xlib contains a
piece of code very much like the one in question H.2. So, your Xlib fetches the error code from the
wrong errno location and concludes that an error it cannot handle
occurred.
The best solution is to use X libraries that have been compiled with
multithreading options set. Linux distributions that come with glibc
2 as the main C library generally provide thread-safe X libraries.
At least, that seems to be the case for RedHat 5 and later.
You can try to recompile yourself the X libraries with multithreading
options set. They contain optional support for multithreading; it's
just that the binaries provided by your Linux distribution were built
without this support. See the file README.Xfree3.3 in
the LinuxThreads distribution for patches and info on how to compile
thread-safe X libraries from the Xfree3.3 distribution. The Xfree3.3
sources are readily available in most Linux distributions, e.g. as a
source RPM for RedHat. Be warned, however, that X Windows is a huge
system, and recompiling even just the libraries takes a lot of time
and disk space.
Another, less involving solution is to call X functions only from the
main thread of your program. Even if all threads have their own errno
location, the main thread uses the global errno variable for its errno
location. Thus, code not compiled with -D_REENTRANT
still "sees" the right error values if it executes in the main thread
only.
Most libraries cannot be used "as is" in a multithreaded program.
For one thing, they are not necessarily thread-safe: calling
simultaneously two functions of the library from two threads might not
work, due to internal use of global variables and the like. Second,
the libraries must have been compiled with -D_REENTRANT to avoid
the errno problems explained in question H.2.
This avoids problems with the library not being thread-safe. But
you're still vulnerable to errno problems. At the very least, a
recompile of the library with -D_REENTRANT is needed.
That might actually work. As explained in question I.1,
the main thread uses the global errno variable, and can therefore
execute code not compiled with -D_REENTRANT.
With a recent kernel (2.2 or later) and the glibc version of LinuxThreads,
there should be no problems. With older kernels or LinuxThreads version,
both LinuxThreads and SVGAlib use the signals
SIGUSR1 and SIGUSR2. See question H.4.
Signal handlers are shared between all threads: when a thread calls
sigaction(), it sets how the signal is handled not only
for itself, but for all other threads in the program as well.
On the other hand, signal masks are per-thread: each thread chooses
which signals it blocks independently of others. At thread creation
time, the newly created thread inherits the signal mask of the thread
calling pthread_create(). But afterwards, the new thread
can modify its signal mask independently of its creator thread.
That's how it should be. The POSIX standard mandates that all threads
should terminate when the process (i.e. the collection of all threads
running the program) receives a signal whose effect is to
terminate the process (such as SIGKILL or SIGINT
when no handler is installed on that signal). This behavior makes a
lot of sense: when you type "ctrl-C" at the keyboard, or when a thread
crashes on a division by zero or a segmentation fault, you really want
all threads to stop immediately, not just the one that caused the
segmentation violation or that got the SIGINT signal.
(This assumes default behavior for those signals; see question
J.3 if you install handlers for those signals.)
If you're trying to terminate a thread without bringing the whole
process down, use pthread_cancel().
If the signal is generated by a thread during its execution (e.g. a
thread executes a division by zero and thus generates a
SIGFPE signal), then the handler is executed by that
thread. This also applies to signals generated by
raise().
If the signal is sent to a particular thread using
pthread_kill(), then that thread executes the handler.
If the signal is sent via kill() or the tty interface
(e.g. by pressing ctrl-C), then the POSIX specs say that the handler
is executed by any thread in the process that does not currently block
the signal. In other terms, POSIX considers that the signal is sent
to the process (the collection of all threads) as a whole, and any
thread that is not blocking this signal can then handle it.
The latter case is where LinuxThreads departs from the POSIX specs.
In LinuxThreads, there is no real notion of ``the process as a whole'':
in the kernel, each thread is really a distinct process with a
distinct PID, and signals sent to the PID of a thread can only be
handled by that thread. As long as no thread is blocking the signal,
the behavior conforms to the standard: one (unspecified) thread of the
program handles the signal. But if the thread to which PID the signal
is sent blocks the signal, and some other thread does not block the
signal, then LinuxThreads will simply queue in
that thread and execute the handler only when that thread unblocks
the signal, instead of executing the handler immediately in the other
thread that does not block the signal.
This is to be viewed as a LinuxThreads bug, but I currently don't see
any way to implement the POSIX behavior without kernel support.
The less you mix them, the better. Notice that all
pthread_* functions are not async-signal safe, meaning
that you should not call them from signal handlers. This
recommendation is not to be taken lightly: your program can deadlock
if you call a pthread_* function from a signal handler!
The only sensible things you can do from a signal handler is set a
global flag, or call sem_post on a semaphore, to record
the delivery of the signal. The remainder of the program can then
either poll the global flag, or use sem_wait() and
sem_trywait() on the semaphore.
Another option is to do nothing in the signal handler, and dedicate
one thread (preferably the initial thread) to wait synchronously for
signals, using sigwait(), and send messages to the other
threads accordingly.
It's an unfortunate consequence of how LinuxThreads implements
sigwait(). Basically, it installs signal handlers on all
signals waited for, in order to record which signal was received.
Since signal handlers are shared with the other threads, this
temporarily deactivates any signal handlers you might have previously
installed on these signals.
Though surprising, this behavior actually seems to conform to the
POSIX standard. According to POSIX, sigwait() is
guaranteed to work as expected only if all other threads in the
program block the signals waited for (otherwise, the signals could be
delivered to other threads than the one doing sigwait(),
which would make sigwait() useless). In this particular
case, the problem described in this question does not appear.
One day, sigwait() will be implemented in the kernel,
along with others POSIX 1003.1b extensions, and sigwait()
will have a more natural behavior (as well as better performances).
LinuxThreads follows the so-called "one-to-one" model: each thread is
actually a separate process in the kernel. The kernel scheduler takes
care of scheduling the threads, just like it schedules regular
processes. The threads are created with the Linux
clone() system call, which is a generalization of
fork() allowing the new process to share the memory
space, file descriptors, and signal handlers of the parent.
Advantages of the "one-to-one" model include:
minimal overhead on CPU-intensive multiprocessing (with
about one thread per processor);
minimal overhead on I/O operations;
a simple and robust implementation (the kernel scheduler does
most of the hard work for us).
The main disadvantage is more expensive context switches on mutex and
condition operations, which must go through the kernel. This is
mitigated by the fact that context switches in the Linux kernel are
pretty efficient.
There are basically two other models. The "many-to-one" model
relies on a user-level scheduler that context-switches between the
threads entirely in user code; viewed from the kernel, there is only
one process running. This model is completely out of the question for
me, since it does not take advantage of multiprocessors, and require
unholy magic to handle blocking I/O operations properly. There are
several user-level thread libraries available for Linux, but I found
all of them deficient in functionality, performance, and/or robustness.
The "many-to-many" model combines both kernel-level and user-level
scheduling: several kernel-level threads run concurrently, each
executing a user-level scheduler that selects between user threads.
Most commercial Unix systems (Solaris, Digital Unix, IRIX) implement
POSIX threads this way. This model combines the advantages of both
the "many-to-one" and the "one-to-one" model, and is attractive
because it avoids the worst-case behaviors of both models --
especially on kernels where context switches are expensive, such as
Digital Unix. Unfortunately, it is pretty complex to implement, and
requires kernel support which Linux does not provide. Linus Torvalds
and other Linux kernel developers have always been pushing the
"one-to-one" model in the name of overall simplicity, and are doing a
pretty good job of making kernel-level context switches between
threads efficient. LinuxThreads is just following the general
direction they set.
LinuxThreads Frequently Asked Questions
(with answers)
[For LinuxThreads version 0.8]
Warning
This FAQ has not been updated for a while and may not be 100% up to date. The glibc mailing lists often contain more up-to-date information.Also note that LinuxThreads is no longer developed and has been replaced by NPTL, the Native POSIX Threads Library for Linux, which fixes many shortcomings of LinuxThreads and offers better performance.
A. The big picture
B. Getting more information
C. Issues related to the C library
D. Problems, weird behaviors, potential bugs
E. Missing functions, wrong types, etc
F. C++ issues
G. Debugging LinuxThreads programs
H. Compiling multithreaded code; errno madness
I. X-Windows and other libraries
J. Signals and threads
K. Internals of LinuxThreads
A. The big picture
A.1: What is LinuxThreads?
LinuxThreads is a Linux library for multi-threaded programming. It implements the Posix 1003.1c API (Application Programming Interface) for threads. It runs on any Linux system with kernel 2.0.0 or more recent, and a suitable C library (see section C).A.2: What are threads?
A thread is a sequential flow of control through a program. Multi-threaded programming is, thus, a form of parallel programming where several threads of control are executing concurrently in the program. All threads execute in the same memory space, and can therefore work concurrently on shared data.Multi-threaded programming differs from using multiple Unix processes in that all threads share the same memory space (and a few other system resources, such as file descriptors), instead of running in their own memory space as is the case with Unix processes.
Threads are useful for several reasons. First, they allow a program to exploit multi-processor machines: the threads can run in parallel on several processors, allowing a single program to divide its work between several processors, thus running faster than a single-threaded program, which runs on only one processor at a time. Second, even on uniprocessor machines, threads allow overlapping I/O and computations in a simple way. Last, some programs are best expressed as several threads of control that communicate together, rather than as one big monolithic sequential program. Examples include server programs, overlapping asynchronous I/O, and graphical user interfaces.
A.3: What is POSIX 1003.1c?
It's an API for multi-threaded programming standardized by IEEE as part of the POSIX standards. Most Unix vendors have endorsed the POSIX 1003.1c standard. Implementations of the 1003.1c API are already available under Sun Solaris 2.5, Digital Unix 4.0, Silicon Graphics IRIX 6, and should soon be available from other vendors such as IBM and HP. More generally, the 1003.1c API is replacing relatively quickly the proprietary threads library that were developed previously under Unix, such as Mach cthreads, Solaris threads, and IRIX sprocs. Thus, multithreaded programs using the 1003.1c API are likely to run unchanged on a wide variety of Unix platforms.A.4: What is the status of LinuxThreads?
LinuxThreads is currently distributed and maintained as part of glibc, but no longer actively developed. Development continues on NPTL, which is a new implementation of POSIX threads for Linux with many advantages over LinuxThreads. It is expected that LinuxThreads will stop being maintained starting with version 2.4 of glibc.LinuxThreads implements almost all of Posix 1003.1c, as well as a few extensions. The only part of LinuxThreads that does not conform to Posix is signal handling (see section J). NPTL improves significantly over LinuxThreads both on standard conformance and on performance.
B. Getting more information
B.1: What are good books and other sources of information on POSIX threads?
The FAQ for comp.programming.threads lists several books: http://www.serpentine.com/~bos/threads-faq/.There are also some online tutorials. Follow the links from the LinuxThreads web page: http://pauillac.inria.fr/~xleroy/linuxthreads.
B.2: I'd like to be informed of future developments on LinuxThreads. Is there a mailing list for this purpose?
I post LinuxThreads-related announcements on the newsgroup comp.os.linux.announce, and also on the mailing listlinux-threads@magenet.com
. You can subscribe to the latter by writing majordomo@magenet.com.B.3: What are good places for discussing LinuxThreads?
For questions about programming with POSIX threads in general, use the newsgroup comp.programming.threads. Be sure you read the FAQ for this group before you post.For Linux-specific questions, use comp.os.linux.development.apps and comp.os.linux.development.kernel. The latter is especially appropriate for questions relative to the interface between the kernel and LinuxThreads.
B.4: How should I report a possible bug in LinuxThreads?
If you're using glibc 2, the best way by far is to use theglibcbug
script to mail a bug report to the glibc maintainers.If you're using an older libc, or don't have the
glibcbug
script on your machine, then e-mail me directly (Xavier.Leroy@inria.fr
).In both cases, before sending the bug report, make sure that it is not addressed already in this FAQ. Also, try to send a short program that reproduces the weird behavior you observed.
B.5: I'd like to read the POSIX 1003.1c standard. Is it available online?
Unfortunately, no. POSIX standards are copyrighted by IEEE, and IEEE does not distribute them freely. You can buy paper copies from IEEE, but the price is fairly high ($120 or so). If you disagree with this policy and you're an IEEE member, be sure to let them know.On the other hand, you probably don't want to read the standard. It's very hard to read, written in standard-ese, and targeted to implementors who already know threads inside-out. A good book on POSIX threads provides the same information in a much more readable form. I can personally recommend Dave Butenhof's book, Programming with POSIX threads (Addison-Wesley). Butenhof was part of the POSIX committee and also designed the Digital Unix implementations of POSIX threads, and it shows.
Another good source of information is the X/Open Group Single Unix specification which is available both on-line and as a book and CD/ROM. That specification includes pretty much all the POSIX standards, including 1003.1c, with some extensions and clarifications.
C. Issues related to the C library
C.1: Which version of the C library should I use with LinuxThreads?
The best choice by far is glibc 2, a.k.a. libc 6. It offers very good support for multi-threading, and LinuxThreads has been closely integrated with glibc 2. The glibc 2 distribution contains the sources of a specially adapted version of LinuxThreads.glibc 2 comes preinstalled as the default C library on most Linux distributions nowadays, such as RedHat 5 and up, and Debian 2 and up. Those distributions include the version of LinuxThreads matching glibc 2.
C.2: My system has libc 5 preinstalled, not glibc 2. Can I still use LinuxThreads?
Yes, but you're likely to run into some problems, as libc 5 only offers minimal support for threads and contains some bugs that affect multithreaded programs.The versions of libc 5 that work best with LinuxThreads are libc 5.2.18 on the one hand, and libc 5.4.12 or later on the other hand. Avoid 5.3.12 and 5.4.7: these have problems with the per-thread errno variable.
C.3: So, should I switch to glibc 2, or stay with a recent libc 5?
I'd recommend you switch to glibc 2. Even for single-threaded programs, glibc 2 is more solid and more standard-conformant than libc 5. And the shortcomings of libc 5 almost preclude any serious multi-threaded programming.Switching an already installed system from libc 5 to glibc 2 is not completely straightforward. See the Glibc2 HOWTO for more information. Much easier is (re-)installing a Linux distribution based on glibc 2, such as RedHat 6.
C.4: Where can I find glibc 2 and the version of LinuxThreads that goes with it?
Both glibc 2 and the associated LinuxThreads distribution can be found on Cygnus' Sourceware collection, and also on any FTP site that mirrors GNU software (the Cygnus site is sometimes more up-to-date than the GNU sites).C.5: Where can I find libc 5 and the version of LinuxThreads that goes with it?
For libc 5, seeftp://sunsite.unc.edu/pub/Linux/devel/GCC/
.For the libc 5 version of LinuxThreads, see ftp://ftp.inria.fr/INRIA/Projects/cristal/Xavier.Leroy/linuxthreads/.
C.6: How can I recompile the glibc 2 version of the LinuxThreads sources?
You must transfer the whole glibc sources, then drop the LinuxThreads sources in thelinuxthreads/
subdirectory, then recompile glibc as a whole. There are now too many inter-dependencies between LinuxThreads and glibc 2 to allow separate re-compilation of LinuxThreads.C.7: What is the correspondence between LinuxThreads version numbers, libc version numbers, and RedHat version numbers?
Here is a summary. (Information on Linux distributions other than RedHat are welcome.)D. Problems, weird behaviors, potential bugs
D.1: When I compile LinuxThreads, I run into problems in file
You probably mean: I haven't actually seen this problem, but several users reported it. My understanding is that something is wrong in the include files of your Linux installation (libc_r/dirent.c
/usr/include/*
). Make sure you're using a supported version of the libc 5 library. (See question C.2).D.2: When I compile LinuxThreads, I run into problems with
Yes,/usr/include/sched.h
: there are several occurrences of_p
that the C compiler does not understand/usr/include/sched.h
that comes with libc 5.3.12 is broken. Replace it with thesched.h
file contained in the LinuxThreads distribution. But really you should not be using libc 5.3.12 with LinuxThreads! (See question C.1.)D.3: My program does
You're using one of the buggy versions of libc (5.3.12, 5.4.7., etc). See question C.1 above.fdopen()
on a file descriptor opened on a pipe. When I link it with LinuxThreads,fdopen()
always returns NULL!D.4: My program creates a lot of threads, and after a while
This is known bug in the version of LinuxThreads that comes with glibc 2.1.1. An upgrade to 2.1.2 is recommended.pthread_create()
no longer returns!D.5: When I'm running a program that creates N threads,
Due to the general "one process per thread" model, there's one process for the initial thread and N processes for the threads it created usingtop
orps
display N+2 processes that are running my program. What do all these processes correspond to?pthread_create
. That leaves one process unaccounted for. That extra process corresponds to the "thread manager" thread, a thread created internally by LinuxThreads to handle thread creation and thread termination. This extra thread is asleep most of the time.D.6: Scheduling seems to be very unfair when there is strong contention on a mutex: instead of giving the mutex to each thread in turn, it seems that it's almost always the same thread that gets the mutex. Isn't this completely broken behavior?
That behavior has mostly disappeared in recent releases of LinuxThreads (version 0.8 and up). It was fairly common in older releases, though. What happens in LinuxThreads 0.7 and before is the following: when a thread unlocks a mutex, all other threads that were waiting on the mutex are sent a signal which makes them runnable. However, the kernel scheduler may or may not restart them immediately. If the thread that unlocked the mutex tries to lock it again immediately afterwards, it is likely that it will succeed, because the threads haven't yet restarted. This results in an apparently very unfair behavior, when the same thread repeatedly locks and unlocks the mutex, while other threads can't lock the mutex.In LinuxThreads 0.8 and up,
pthread_unlock
restarts only one waiting thread, and pre-assign the mutex to that thread. Hence, if the thread that unlocked the mutex tries to lock it again immediately, it will block until other waiting threads have had a chance to lock and unlock the mutex. This results in much fairer scheduling.Notice however that even the old "unfair" behavior is perfectly acceptable with respect to the POSIX standard: for the default scheduling policy, POSIX makes no guarantees of fairness, such as "the thread waiting for the mutex for the longest time always acquires it first". Properly written multithreaded code avoids that kind of heavy contention on mutexes, and does not run into fairness problems. If you need scheduling guarantees, you should consider using the real-time scheduling policies
SCHED_RR
andSCHED_FIFO
, which have precisely defined scheduling behaviors.D.7: I have a simple test program with two threads that do nothing but
Again, this behavior is characteristic of old releases of LinuxThreads (0.7 and before); more recent versions (0.8 and up) should not exhibit this behavior.printf()
in tight loops, and from the printout it seems that only one thread is running, the other doesn't print anything!The reason for this behavior is explained in question D.6 above:
printf()
performs locking onstdout
, and thus your two threads contend very heavily for the mutex associated withstdout
. But if you do some real work between two calls toprintf()
, you'll see that scheduling becomes much smoother.D.8: I've looked at
Nope. That's the way it should be. The closing brace is provided by the<pthread.h>
and there seems to be a gross error in thepthread_cleanup_push
macro: it opens a block with{
but does not close it! Surely you forgot a}
at the end of the macro, right?pthread_cleanup_pop
macro. The POSIX standard requirespthread_cleanup_push
andpthread_cleanup_pop
to be used in matching pairs, at the same level of brace nesting. This allowspthread_cleanup_push
to open a block in order to stack-allocate some data structure, andpthread_cleanup_pop
to close that block. It's ugly, but it's the standard way of implementing cleanup handlers.D.9: I tried to use real-time threads and my program loops like crazy and freezes the whole machine!
Versions of LinuxThreads prior to 0.8 are susceptible to ``livelocks'' (one thread loops, consuming 100% of the CPU time) in conjunction with real-time scheduling. Since real-time threads and processes have higher priority than normal Linux processes, all other processes on the machine, including the shell, the X server, etc, cannot run at all and the machine appears frozen.The problem is fixed in LinuxThreads 0.8.
D.10: My application needs to create thousands of threads, or maybe even more. Can I do this with LinuxThreads?
No. You're going to run into several hard limits:NR_TASKS
ininclude/linux/tasks.h
and recompiling the kernel. On the x86 processors at least, architectural constraints seem to limitNR_TASKS
to 4090 at most. (It seems that 2.4 kernels have higher limits, though.)PTHREAD_THREADS_MAX
in the LinuxThreads/glibc sources and recompile.setstackaddr
attribute.E. Missing functions, wrong types, etc
E.1: Where is
Because it's not part of the (final) POSIX 1003.1c standard. Several drafts of the standard containedpthread_yield()
? How comes LinuxThreads does not implement it?pthread_yield()
, but then the POSIX guys discovered it was redundant withsched_yield()
and dropped it. So, just usesched_yield()
instead.E.2: I've found some type errors in
No, I didn't. What you're describing is draft 4 of the POSIX standard, which is used in OSF DCE threads. LinuxThreads conforms to the final standard. Even though the functions have the same names as in draft 4 and DCE, their calling conventions are slightly different. In particular, attributes are passed by reference, not by value, and default attributes are denoted by the NULL pointer. Since draft 4/DCE will eventually disappear, you'd better port your program to use the standard interface.<pthread.h>
. For instance, the second argument topthread_create()
should be apthread_attr_t
, not apthread_attr_t *
. Also, didn't you forget to declarepthread_attr_default
?E.3: I'm porting an application from Solaris and I have to rename all thread functions from
POSIX did it. Thethr_blah
topthread_blah
. This is very annoying. Why did you change all the function names?thr_*
functions correspond to Solaris threads, an older thread interface that you'll find only under Solaris. Thepthread_*
functions correspond to POSIX threads, an international standard available for many, many platforms. Even Solaris 2.5 and later support the POSIX threads interface. So, do yourself a favor and rewrite your code to use POSIX threads: this way, it will run unchanged under Linux, Solaris, and quite a lot of other platforms.E.4: How can I suspend and resume a thread from another thread? Solaris has the
The POSIX standard provides no mechanism by which a thread A can suspend the execution of another thread B, without cooperation from B. The only way to implement a suspend/restart mechanism is to have B check periodically some global variable for a suspend request and then suspend itself on a condition variable, which another thread can signal later to restart B.thr_suspend()
andthr_resume()
functions to do that; why don't you?Notice that
thr_suspend()
is inherently dangerous and prone to race conditions. For one thing, there is no control on where the target thread stops: it can very well be stopped in the middle of a critical section, while holding mutexes. Also, there is no guarantee on when the target thread will actually stop. For these reasons, you'd be much better off using mutexes and conditions instead. The only situations that really require the ability to suspend a thread are debuggers and some kind of garbage collectors.If you really must suspend a thread in LinuxThreads, you can send it a
SIGSTOP
signal withpthread_kill
. SendSIGCONT
for restarting it. Beware, this is specific to LinuxThreads and entirely non-portable. Indeed, a truly conforming POSIX threads implementation will stop all threads when one thread receives theSIGSTOP
signal! One day, LinuxThreads will implement that behavior, and the non-portable hack withSIGSTOP
won't work anymore.E.5: Does LinuxThreads implement
These optional functions are provided in recent versions of LinuxThreads (0.8 and up). Earlier releases did not provide these optional components of the POSIX standard.pthread_attr_setstacksize()
andpthread_attr_setstackaddr()
?Even if
pthread_attr_setstacksize()
andpthread_attr_setstackaddr()
are now provided, we still recommend that you do not use them unless you really have strong reasons for doing so. The default stack allocation strategy for LinuxThreads is nearly optimal: stacks start small (4k) and automatically grow on demand to a fairly large limit (2M). Moreover, there is no portable way to estimate the stack requirements of a thread, so setting the stack size yourself makes your program less reliable and non-portable.E.6: LinuxThreads does not support the
With a "one-to-one" model, as in LinuxThreads (one kernel execution context per thread), there is only one scheduler for all processes and all threads on the system. So, there is no way to obtain the behavior ofPTHREAD_SCOPE_PROCESS
value of the "contentionscope" attribute. Why?PTHREAD_SCOPE_PROCESS
.E.7: LinuxThreads does not implement process-shared mutexes, conditions, and semaphores. Why?
This is another optional component of the POSIX standard. Portable applications should test_POSIX_THREAD_PROCESS_SHARED
before using this facility.The goal of this extension is to allow different processes (with different address spaces) to synchronize through mutexes, conditions or semaphores allocated in shared memory (either SVR4 shared memory segments or
mmap()
ed files).The reason why this does not work in LinuxThreads is that mutexes, conditions, and semaphores are not self-contained: their waiting queues contain pointers to linked lists of thread descriptors, and these pointers are meaningful only in one address space.
Matt Messier and I spent a significant amount of time trying to design a suitable mechanism for sharing waiting queues between processes. We came up with several solutions that combined two of the following three desirable features, but none that combines all three:
pthread_cond_timedwait
clone()
" fails.Until suitable kernel support is available, you'd better use traditional interprocess communications to synchronize different processes: System V semaphores and message queues, or pipes, or sockets.
F. C++ issues
F.1: Are there C++ wrappers for LinuxThreads?
Douglas Schmidt's ACE library contains, among a lot of other things, C++ wrappers for LinuxThreads and quite a number of other thread libraries. Check out http://www.cs.wustl.edu/~schmidt/ACE.htmlF.2: I'm trying to use LinuxThreads from a C++ program, and the compiler complains about the third argument to
You're probably trying to pass a class member function or some other C++ thing as third argument topthread_create()
!pthread_create()
. Recall thatpthread_create()
is a C function, and it must be passed a C function as third argument.F.3: I'm trying to use LinuxThreads in conjunction with libg++, and I'm having all sorts of trouble.
From what I understand, thread support in libg++ is completely broken, especially with respect to locking of iostreams. H.J.Lu wrote:G. Debugging LinuxThreads programs
G.1: Can I debug LinuxThreads program using gdb?
Yes, but not with the stock gdb 4.17. You need a specially patched version of gdb 4.17 developed by Eric Paire and colleages at The Open Group, Grenoble. The patches against gdb 4.17 are available athttp://pauillac.inria.fr/~xleroy/linuxthreads/gdb-4.17-debug-threads.patch.gz
. H.J.Lu also develops patches for gdb 4.17 that include thread support and more; those are available atftp://ftp.valinux.com/pub/support/hjl/gdb/
. More recent versions of gdb seem to include LinuxThreads support.Some Linux distributions provide an already-patched version of gdb; others don't. For instance, the gdb in RedHat 5.2 and RedHat 6.2 is thread-aware, but apparently not the one in RedHat 6.0. Just ask (politely) the makers of your Linux distributions to please make sure that they apply the correct patches to gdb.
G.2: Does it work with post-mortem debugging?
Not very well. Generally, the core file does not correspond to the thread that crashed. The reason is that the kernel will not dump core for a process that shares its memory with other processes, such as the other threads of your program. So, the thread that crashes silently disappears without generating a core file. Then, all other threads of your program die on the same signal that killed the crashing thread. (This is required behavior according to the POSIX standard.) The last one that dies is no longer sharing its memory with anyone else, so the kernel generates a core file for that thread. Unfortunately, that's not the thread you are interested in.G.3: Any other ways to debug multithreaded programs, then?
Assertions andprintf()
are your best friends. Try to debug sequential parts in a single-threaded program first. Then, putprintf()
statements all over the place to get execution traces. Also, check invariants often with theassert()
macro. In truth, there is no other effective way (save for a full formal proof of your program) to track down concurrency bugs. Debuggers are not really effective for subtle concurrency problems, because they disrupt program execution too much.H. Compiling multithreaded code; errno madness
H.1: You say all multithreaded code must be compiled with
It affects include files in three ways:_REENTRANT
defined. What difference does it make?gethostbyname_r()
as a reentrant equivalent togethostbyname()
._REENTRANT
is defined, some<stdio.h>
functions are no longer defined as macros, e.g.getc()
andputc()
. In a multithreaded program, stdio functions require additional locking, which the macros don't perform, so we must call functions instead.<errno.h>
redefines errno when_REENTRANT
is defined, so that errno refers to the thread-specific errno location rather than the global errno variable. This is achieved by the following#define
in<errno.h>
: which causes each reference to errno to call the__errno_location()
function for obtaining the location where error codes are stored. libc provides a default definition of__errno_location()
that always returns&errno
(the address of the global errno variable). Thus, for programs not linked with LinuxThreads, defining_REENTRANT
makes no difference w.r.t. errno processing. But LinuxThreads redefines__errno_location()
to return a location in the thread descriptor reserved for holding the current value of errno for the calling thread. Thus, each thread operates on a different errno location.H.2: Why is it so important that each thread has its own errno variable?
If all threads were to store error codes in the same, global errno variable, then the value of errno after a system call or library function returns would be unpredictable: between the time a system call stores its error code in the global errno and your code inspects errno to see which error occurred, another thread might have stored another error code in the same errno location.H.3: What happens if I link LinuxThreads with code not compiled with
Lots of trouble. If the code uses-D_REENTRANT
?getc()
orputc()
, it will perform I/O without proper interlocking of the stdio buffers; this can cause lost output, duplicate output, or just crash other stdio functions. If the code consults errno, it will get back the wrong error code. The following code fragment is a typical example: Assume this code is not compiled with-D_REENTRANT
, and linked with LinuxThreads. At run-time,read()
is interrupted. Since the C library was compiled with-D_REENTRANT
,read()
stores its error code in the location pointed to by__errno_location()
, which is the thread-local errno variable. Then, the code above sees thatread()
returns -1 and looks up errno. Since_REENTRANT
is not defined, the reference to errno accesses the global errno variable, which is most likely 0. Hence the code concludes that it cannot handle the error and stops.H.4: With LinuxThreads, I can no longer use the signals
The short answer is: because the Linux kernel you're using does not support realtime signals.SIGUSR1
andSIGUSR2
in my programs! Why?LinuxThreads needs two signals for its internal operation. One is used to suspend and restart threads blocked on mutex, condition or semaphore operations. The other is used for thread cancellation.
On ``old'' kernels (2.0 and early 2.1 kernels), there are only 32 signals available and the kernel reserves all of them but two:
SIGUSR1
andSIGUSR2
. So, LinuxThreads has no choice but use those two signals.On recent kernels (2.2 and up), more than 32 signals are provided in the form of realtime signals. When run on one of those kernels, LinuxThreads uses two reserved realtime signals for its internal operation, thus leaving
SIGUSR1
andSIGUSR2
free for user code. (This works only with glibc, not with libc 5.)H.5: Is the stack of one thread visible from the other threads? Can I pass a pointer into my stack to other threads?
Yes, you can -- if you're very careful. The stacks are indeed visible from all threads in the system. Some non-POSIX thread libraries seem to map the stacks for all threads at the same virtual addresses and change the memory mapping when they switch from one thread to another. But this is not the case for LinuxThreads, as it would make context switching between threads more expensive, and at any rate might not conform to the POSIX standard.So, you can take the address of an "auto" variable and pass it to other threads via shared data structures. However, you need to make absolutely sure that the function doing this will not return as long as other threads need to access this address. It's the usual mistake of returning the address of an "auto" variable, only made much worse because of concurrency. It's much, much safer to systematically heap-allocate all shared data structures.
I. X-Windows and other libraries
I.1: My program uses both Xlib and LinuxThreads. It stops very early with an "Xlib: unknown 0 error" message. What does this mean?
That's a prime example of the errno problem described in question H.2. The binaries for Xlib you're using have not been compiled with-D_REENTRANT
. It happens Xlib contains a piece of code very much like the one in question H.2. So, your Xlib fetches the error code from the wrong errno location and concludes that an error it cannot handle occurred.I.2: So, what can I do to build a multithreaded X Windows client?
The best solution is to use X libraries that have been compiled with multithreading options set. Linux distributions that come with glibc 2 as the main C library generally provide thread-safe X libraries. At least, that seems to be the case for RedHat 5 and later.You can try to recompile yourself the X libraries with multithreading options set. They contain optional support for multithreading; it's just that the binaries provided by your Linux distribution were built without this support. See the file
README.Xfree3.3
in the LinuxThreads distribution for patches and info on how to compile thread-safe X libraries from the Xfree3.3 distribution. The Xfree3.3 sources are readily available in most Linux distributions, e.g. as a source RPM for RedHat. Be warned, however, that X Windows is a huge system, and recompiling even just the libraries takes a lot of time and disk space.Another, less involving solution is to call X functions only from the main thread of your program. Even if all threads have their own errno location, the main thread uses the global errno variable for its errno location. Thus, code not compiled with
-D_REENTRANT
still "sees" the right error values if it executes in the main thread only.This is a lot of work. Don't you have precompiled thread-safe X libraries that you could distribute?
No, I don't. Sorry. But consider installing a Linux distribution that comes with thread-safe X libraries, such as RedHat 6.I.3: Can I use library FOO in a multithreaded program?
Most libraries cannot be used "as is" in a multithreaded program. For one thing, they are not necessarily thread-safe: calling simultaneously two functions of the library from two threads might not work, due to internal use of global variables and the like. Second, the libraries must have been compiled with-D_REENTRANT
to avoid the errno problems explained in question H.2.I.4: What if I make sure that only one thread calls functions in these libraries?
This avoids problems with the library not being thread-safe. But you're still vulnerable to errno problems. At the very least, a recompile of the library with-D_REENTRANT
is needed.I.5: What if I make sure that only the main thread calls functions in these libraries?
That might actually work. As explained in question I.1, the main thread uses the global errno variable, and can therefore execute code not compiled with-D_REENTRANT
.I.6: SVGAlib doesn't work with LinuxThreads. Why?
With a recent kernel (2.2 or later) and the glibc version of LinuxThreads, there should be no problems. With older kernels or LinuxThreads version, both LinuxThreads and SVGAlib use the signalsSIGUSR1
andSIGUSR2
. See question H.4.J. Signals and threads
J.1: When it comes to signals, what is shared between threads and what isn't?
Signal handlers are shared between all threads: when a thread callssigaction()
, it sets how the signal is handled not only for itself, but for all other threads in the program as well.On the other hand, signal masks are per-thread: each thread chooses which signals it blocks independently of others. At thread creation time, the newly created thread inherits the signal mask of the thread calling
pthread_create()
. But afterwards, the new thread can modify its signal mask independently of its creator thread.J.2: When I send a
That's how it should be. The POSIX standard mandates that all threads should terminate when the process (i.e. the collection of all threads running the program) receives a signal whose effect is to terminate the process (such asSIGKILL
to a particular thread usingpthread_kill
, all my threads are killed!SIGKILL
orSIGINT
when no handler is installed on that signal). This behavior makes a lot of sense: when you type "ctrl-C" at the keyboard, or when a thread crashes on a division by zero or a segmentation fault, you really want all threads to stop immediately, not just the one that caused the segmentation violation or that got theSIGINT
signal. (This assumes default behavior for those signals; see question J.3 if you install handlers for those signals.)If you're trying to terminate a thread without bringing the whole process down, use
pthread_cancel()
.J.3: I've installed a handler on a signal. Which thread executes the handler when the signal is received?
If the signal is generated by a thread during its execution (e.g. a thread executes a division by zero and thus generates aSIGFPE
signal), then the handler is executed by that thread. This also applies to signals generated byraise()
.If the signal is sent to a particular thread using
pthread_kill()
, then that thread executes the handler.If the signal is sent via
kill()
or the tty interface (e.g. by pressing ctrl-C), then the POSIX specs say that the handler is executed by any thread in the process that does not currently block the signal. In other terms, POSIX considers that the signal is sent to the process (the collection of all threads) as a whole, and any thread that is not blocking this signal can then handle it.The latter case is where LinuxThreads departs from the POSIX specs. In LinuxThreads, there is no real notion of ``the process as a whole'': in the kernel, each thread is really a distinct process with a distinct PID, and signals sent to the PID of a thread can only be handled by that thread. As long as no thread is blocking the signal, the behavior conforms to the standard: one (unspecified) thread of the program handles the signal. But if the thread to which PID the signal is sent blocks the signal, and some other thread does not block the signal, then LinuxThreads will simply queue in that thread and execute the handler only when that thread unblocks the signal, instead of executing the handler immediately in the other thread that does not block the signal.
This is to be viewed as a LinuxThreads bug, but I currently don't see any way to implement the POSIX behavior without kernel support.
J.3: How shall I go about mixing signals and threads in my program?
The less you mix them, the better. Notice that allpthread_*
functions are not async-signal safe, meaning that you should not call them from signal handlers. This recommendation is not to be taken lightly: your program can deadlock if you call apthread_*
function from a signal handler!The only sensible things you can do from a signal handler is set a global flag, or call
sem_post
on a semaphore, to record the delivery of the signal. The remainder of the program can then either poll the global flag, or usesem_wait()
andsem_trywait()
on the semaphore.Another option is to do nothing in the signal handler, and dedicate one thread (preferably the initial thread) to wait synchronously for signals, using
sigwait()
, and send messages to the other threads accordingly.J.4: When one thread is blocked in
It's an unfortunate consequence of how LinuxThreads implementssigwait()
, other threads no longer receive the signalssigwait()
is waiting for! What happens?sigwait()
. Basically, it installs signal handlers on all signals waited for, in order to record which signal was received. Since signal handlers are shared with the other threads, this temporarily deactivates any signal handlers you might have previously installed on these signals.Though surprising, this behavior actually seems to conform to the POSIX standard. According to POSIX,
sigwait()
is guaranteed to work as expected only if all other threads in the program block the signals waited for (otherwise, the signals could be delivered to other threads than the one doingsigwait()
, which would makesigwait()
useless). In this particular case, the problem described in this question does not appear.One day,
sigwait()
will be implemented in the kernel, along with others POSIX 1003.1b extensions, andsigwait()
will have a more natural behavior (as well as better performances).K. Internals of LinuxThreads
K.1: What is the implementation model for LinuxThreads?
LinuxThreads follows the so-called "one-to-one" model: each thread is actually a separate process in the kernel. The kernel scheduler takes care of scheduling the threads, just like it schedules regular processes. The threads are created with the Linuxclone()
system call, which is a generalization offork()
allowing the new process to share the memory space, file descriptors, and signal handlers of the parent.Advantages of the "one-to-one" model include:
K.2: Have you considered other implementation models?
There are basically two other models. The "many-to-one" model relies on a user-level scheduler that context-switches between the threads entirely in user code; viewed from the kernel, there is only one process running. This model is completely out of the question for me, since it does not take advantage of multiprocessors, and require unholy magic to handle blocking I/O operations properly. There are several user-level thread libraries available for Linux, but I found all of them deficient in functionality, performance, and/or robustness.The "many-to-many" model combines both kernel-level and user-level scheduling: several kernel-level threads run concurrently, each executing a user-level scheduler that selects between user threads. Most commercial Unix systems (Solaris, Digital Unix, IRIX) implement POSIX threads this way. This model combines the advantages of both the "many-to-one" and the "one-to-one" model, and is attractive because it avoids the worst-case behaviors of both models -- especially on kernels where context switches are expensive, such as Digital Unix. Unfortunately, it is pretty complex to implement, and requires kernel support which Linux does not provide. Linus Torvalds and other Linux kernel developers have always been pushing the "one-to-one" model in the name of overall simplicity, and are doing a pretty good job of making kernel-level context switches between threads efficient. LinuxThreads is just following the general direction they set.
Xavier.Leroy@inria.fr
Recent Posts
Archive Posts
Tags