---
metadata:
- name: generator
content: Diplodoc Platform v5.50.6
alternate:
- https://perforator.tech/docs/en/reference/language-support/python/threadstate.md
---
> **Documentation Index:** Fetch the complete configuration index at https://perforator.tech/docs/en/llms.txt
# Python Current Thread State Collection
Each native thread is mapped to one `PyThreadState` structure which contains information about the corresponding Python thread.
Perforator utilizes multiple ways to obtain current `*PyThreadState` in eBPF context - reading from Thread Local Storage (TLS) or Thread Specific Storage (TSS) and extracting from global variables - such as `_PyRuntime` or others. The combination of these approaches and caching helps to improve the accuracy of the `*PyThreadState` collection.
## Reading `*PyThreadState` from TLS
In Python 3.12+, a pointer to current thread's `PyThreadState` is stored in a Thread Local Storage variable called `_PyThreadState_Current`.
In an eBPF program, the pointer to Thread Control Block (TCB) can be retrieved by reading `thread.fsbase` from the `task_struct` structure. This structure can be obtained with the `bpf_get_current_task()` helper. The Thread Local Image will be to the left of the pointer stored in `thread.fsbase`.
The exact offset of thread local variable `_PyThreadState_Current` in Thread Local Image is unknown yet. Therefore, the disassembler is used to find the offset of `_PyThreadState_Current`.
### Parsing offset of `_PyThreadState_Current` in Thread Local Image
`_PyThreadState_GetCurrent` is a simple getter function which returns the pointer from `_PyThreadState_Current` thread local variable and looks somewhat like this:
**Optimized build**:
```
000000000028a0b0 <_PyThreadState_GetCurrent@@Base>:
28a0b0: f3 0f 1e fa endbr64
28a0b4: 64 48 8b 04 25 f8 ff mov %fs:0xfffffffffffffff8,%rax
28a0bb: ff ff
28a0bd: c3 ret
28a0be: 66 90 xchg %ax,%ax
```
**Debug build**:
```
0000000001dad910 <_PyThreadState_GetCurrent>:
1dad910: 55 push %rbp
1dad911: 48 89 e5 mov %rsp,%rbp
1dad914: 48 8d 3d 15 6e 65 00 lea 0x656e15(%rip),%rdi # 2404730 <_PyRuntime>
1dad91b: e8 10 00 00 00 callq 1dad930
1dad920: 5d pop %rbp
...
...
...
0000000001db7c50 :
1db7c50: 55 push %rbp
1db7c51: 48 89 e5 mov %rsp,%rbp
1db7c54: 48 89 7d f8 mov %rdi,-0x8(%rbp)
1db7c58: 64 48 8b 04 25 00 00 mov %fs:0x0,%rax
1db7c5f: 00 00
1db7c61: 48 8d 80 f8 ff ff ff lea -0x8(%rax),%rax
1db7c68: 48 8b 00 mov (%rax),%rax
1db7c6b: 5d pop %rbp
1db7c6c: c3 retq
```
Looking at these functions, the offset relative to `%fs` register which is used to access `_PyThreadState_Current` variable in userspace can be extracted for later use in the eBPF program.
## Reading `*PyThreadState` from TSS
Before CPython 3.12 libpthread thread specific storage was used to store current `*PyThreadState`. There is a global variable (`autoTLSkey` or `PyRuntime->autoTSSkey`) which stores a key to TSS for the value of current thread state. BPF code implements logic of TSS variable lookup similar to libpthread to obtain the data.
The address of TSS key can be retrieved by parsing `PyGILState_Ensure` function.
Dissassembling `pthread_getspecific` allows to retrieve the following data from the libpthread binary:
```
struct TPthreadKeyDataInfo {
ui64 Size = 0;
ui64 ValueOffset = 0;
ui64 SeqOffset = 0;
};
struct TAccessTSSInfo {
TPthreadKeyDataInfo PthreadKeyData;
ui64 FirstSpecificBlockOffset = 0;
ui64 SpecificArrayOffset = 0;
ui64 StructPthreadPointerOffset = 0;
ui64 KeySecondLevelSize = 0; // PTHREAD_KEY_2NDLEVEL_SIZE
ui64 KeyFirstLevelSize = 0; // PTHREAD_KEY_1STLEVEL_SIZE
ui64 KeysMax = 0; // PTHREAD_KEYS_MAX
};
```
The pthread TSS is organized as a 2-level array of `struct pthread_key_data`. First array of `PTHREAD_KEY_1STLEVEL_SIZE` size contains pointers to the chunks of `struct pthread_key_data` of `PTHREAD_KEY_2NDLEVEL_SIZE` size. First chunk of 2nd level is preallocated in `struct pthread`. The maximum total number of thread key-value pairs `PTHREAD_KEYS_MAX`.
The key is an index in 2-level array. `i / PTHREAD_KEY_2NDLEVEL_SIZE` is index for top-level array, `i % PTHREAD_KEY_2NDLEVEL_SIZE` is index for the bottom-level array.
The pointer to the top-level array can be obtained through `struct pthread` by reading `current_task->thread.fsbase`.
## Thread state list retrieval from global variables (`_PyRuntime` or `interp_head`)
In CPython 3.7 global variable `_PyRuntime` was introduced. This structure holds most of the runtime data including the list of `PyInterpreterState`. Given the `PyInterpreterState` object we can access `PyThreadState`'s associated with it by reading `tstate_head` field. This field stores a pointer to the head of interpreter's thread states list. The address of `_PyRuntime` can be found in the ELF symbols sections.
Before the `_PyRuntime` there was another global variable containing interpreter list - `interp_head` which is available even in CPython 2.4. The `interp_head` address can be retrieved by disassembling `PyInterpreterState_Head` function which is a simpler getter for the variable.
The picture below shows the layout of interpreter state and thread state lists which is almost the same for both `_PyRuntime` and `interp_head` cases.
## Restoring the mapping `(pid, thread_id)` -> `*PyThreadState`
We store a mapping `(pid, thread_id)` -> `*PyThreadState` which allows `*PyThreadState` retrieval on failed BPF reads or other scenarios (e.g. TLS variable with current thread state stores NULL pointer because of released GIL).
In CPython 3.11 `native_thread_id` field was introduced on `PyThreadState` structure. For Linux this field stores TID in some PID namespace. Thereby we can store `(pid, native_thread_id)` pair as a key to the `*PyThreadState`. Upon lookup we can use current task's PID and TID in innermost PID namespace.
Before CPython 3.11 `thread_id` field on `PyThreadState` structure contained the `thread_id` returned by pthread library. We only support glibc libpthread of version 2.4+ here. Actually in this setting this `thread_id` is just a pointer to `struct pthread` of the thread. Upon lookup we can use current task's PID and `current_task->thread.fsbase` because it stores a pointer to TCB which is actually a header of `struct pthread`, so this is just a pointer to the current thread structure.
## Combination of approaches
By combining these approaches, we can improve the accuracy of the stack collection. `_PyThreadState_Current` is `NULL` if the current OS thread is not holding a GIL. In this case, the mapping `(pid, thread_id)` -> `*PyThreadState` can be used to find the correct `*PyThreadState`. Also, occasionally we need to trigger the `PyThreadState` linked list traversal to fill the map.
Collecting the stack of threads which are not holding a GIL is crucial for a more accurate picture of what the program is doing. The OS thread may be blocked on I/O operations or executing compression/decompression off-GIL.