Those are threads. Linux native threads are scheduled by the same scheduler that deals with processes, and under kernels up to 2.4, anyway, ps shows them as looking just like processes. But you can actually tell ps to display them as a "tree" of threads using the -f or --forest switch (this depends on your version of "ps" of course) which shows how the threads are linked to processes.
On the newest (2.6 kernel) systems, threads don't seem to have their own process numbers anymore.
The continuing history of Linux threading is IMO an interesting one, and should answer your question, so here's a potted history so far.
The original linux thread implementation, called LinuxThreads, was built on a model where each thread was a process in its own right, with a manager thread to control creation and destruction of "threads". However, it didn't scale well, there were bottlenecks, some posix thread features could not be implemented, and the overheads were high. Each thread had its own PID.
Your linux system most likely uses LinuxThreads.
In the rush to improve threading on Linux, the state of which has been one of the main obstacles to Linux being adopted on an Enterprise scale, two main projects worked in parallel (and in competition) hoping to be adopted as the Linux threading model. They are the NGPT project and the NPTL project.
In late 2.4 kernels the NGPT (next generation posix threads) work can be back-ported to the kernel. In this threading model the actual user threads (eg. threads that you spawn in your Java proggie) are different from the kernel threads and scheduled separately. This is generally called the M:N threading model, where there are M threads in the process, and N underlying kernel threads to handle the actual processing. However, this had problems too, with thread priorities, scheduling, and again some posix features could not be implemented.
The threading model in 2.6 kernels is based on the work of the NPTL (Native POSIX Thread Library) project, which implements a 1:1 threading model, ie. one kernel thread for one user thread. This should mean that for each thread that you spawn in your java proggie there will be a kernel thread spawned. The kernel's internal threading infrastructure has also been rewritten to allow the Native POSIX Thread Library (pthreads) to run on top of it (but since Java threads don't exactly match POSIX threads if your interest is in java thread programming this may be irrelevant for you).
It seems that the NPTL project won the day and has secured its place in Linux's future. Hence its appearance in the 2.6 kernel.
Where threading is concerned, the kernel can move each kernel thread from one processor to another in an SMP (symmetric multiprocessing) configuration. This is not possible with independent user threads. The important difference between Linux kernels 2.4 and 2.6 is that with a 1:1 threading model (2.6 kernels) you should get far better performance given a multi-processor machine or a machine with a processor that implements hyperthreading (eg. the Intel Xeon).
This has been good news for Enterprise-class Linux.
HTH. [ July 08, 2004: Message edited by: George Brown ]
Joined: Sep 26, 2000
Incidentally, the NPTL thread library is designed to be compatible with the old LinuxThreads implementation, but since the older implementation did not strictly follow the POSIX standard some changes to existing code may be required, and one of the changes that is conformant with the POSIX standard is that getpid() now returns the same value in all threads. As you correctly point out, in the LinuxThreads implementation each thread has its own PID, so each different thread's call to getpid() will return a different PID.
NPTL was backported into the Linux 2.4 kernel in RedHat 9 and maybe 8. Since Fedora is for all practical purposes still Red Hat, I'm pretty sure it, too uses NPTL.
Customer surveys are for companies who didn't pay proper attention to begin with.
Joined: Aug 14, 2004
At my company we are developing a large multi-threaded Java app that runs on a Redhat 7.3 Linux that they customized for us. One of the app's functions is to collect real-time cardiac signal samples and display them as "traces" on a sweeping screen display (like the ECG monitors you see on TV).
The trace-display feature has been challenging because the Sun JVM for Linux does not honor user-requested Java thread priorities.
Redhat has indicated that they can patch our distribution to replace LinuxThreads with NPTL. I was wondering if anyone knew what effect NPTL might have on setting thread priorities via the JVM. My guess is none; I doubt if the JVM even knows or cares about the threading model of the underlying OS.
Thoughts? (John Brown?)
Joined: Mar 04, 2004
George, thanks very much for that explanation.
Do you happen to know how Linux threading compares with the threading on other major OSes - in particular Windows and BSD Unix?