Product SiteDocumentation Site

Threaded Coroutine Library

Making the most of your cores, in user space

Philipp Reisner

LINBIT Engineering

Edited by

Florian Haas

LINBIT Documentation

Legal Notice

Legal notice

The text of and illustrations in this document are licensed under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/.
Linux is a registered trademark of Linus Torvalds. Other names may be trademarks or registered trademarks of their respective owners.
Abstract
The threaded coroutine library (libtcr) is a framework for writing network server applications. Its purpose is to enable developers to write servers making use of multiple CPUs/cores to handle requests of a single (or multiple) connection(s) in a pipelined fashion. The library's core is a parallel main event loop to dispatch events on file descriptors based on the epoll(4), eventfd(2) and timerfd(2) system calls. For additional ease of use, lightweight user space thread switching is also a part of this framework. It requires a 2.6.25 or later Linux kernel. Libtcr is available under the terms of LGPL 3.

1. Introduction
2. Traditional Architectures
2.1. Forking servers
2.2. Threaded servers
2.3. select/poll based state machines
3. Improving throughput by pipelining
3.1. Threads for pipeline stages
3.2. Threads per CPU
4. The TCR library
4.1. Blocking IO functions
4.2. Using libtcr for a pipelined server
4.3. Using libtcr for computational problems
4.4. Abusing libtcr for cooperative task switching
4.5. Other event sources with libtcr
4.6. Debugging with libtcr
5. What's In It For Me?
6. Function Reference
tc_run — libtcr initialization
tc_thread_new — libtcr thread functions
tc_thread_pool_new — libtcr thread bundle functions
tc_wait_fd — libtcr file descriptor functions
tc_mutex_lock — libtcr mutex functions
tc_signal_subscribe — libtcr signaling functions
tc_waitq_wait_event — libtc wait queue functions
tc_sleep — sleep functions
tc_parallel_for — libtcr convenience macros for parallel computation

1. Introduction

Work on this library began with the goal to write a TCP server application utilizing all CPU cores available on the system in a useful manner. In fact, this library is a byproduct of the second rewrite of that application.
In Section 2, “Traditional Architectures”, we present the "classic" way of writing server applications. Section 3, “Improving throughput by pipelining” enumerates the possible ways of adding pipelining to the architecture. That section also goes into the downsides of various design options and outlines the architecture of libtcr at its end.
Finally, Section 4, “The TCR library” describes how to write applications in the presented framework, and Section 6, “Function Reference” contains the man pages packaged with this library.