High-performance thread pool optimized for high-frequency task submission. More...

#include <thread_pool.hpp>

Classes
struct	Statistics

Public Types
using	Task = std::function<void()>
using	QueuedTask = detail::move_callable<void()>

Public Member Functions
	HighPerformancePool (size_t num_threads=std::thread::hardware_concurrency(), size_t deque_capacity=WorkStealingDeque< QueuedTask >::DEFAULT_CAPACITY, bool register_workers=false)
	HighPerformancePool (HighPerformancePool const &)=delete
auto	operator= (HighPerformancePool const &) -> HighPerformancePool &=delete
	~HighPerformancePool ()
void	shutdown (ShutdownPolicy policy=ShutdownPolicy::drain)
	Shut the pool down.
auto	shutdown_for (std::chrono::milliseconds timeout) -> bool
	Attempt a timed drain: finish as many tasks as possible within `timeout`, then force-stop remaining workers.
template<typename F, typename... Args>
auto	try_submit (F &&f, Args &&... args) -> expected< std::future< std::invoke_result_t< F, Args... > >, std::error_code >
	Submit a task without throwing on shutdown.
template<typename F, typename... Args>
auto	submit (F &&f, Args &&... args) -> std::future< std::invoke_result_t< F, Args... > >
	Submit a task, throwing on shutdown.
template<typename F, typename... Args>
void	post (F &&f, Args &&... args)
	Fire-and-forget task submission (throwing variant).
template<typename F, typename... Args>
auto	try_post (F &&f, Args &&... args) -> expected< void, std::error_code >
	Fire-and-forget task submission (non-throwing variant).
template<typename Iterator>
auto	try_submit_batch (Iterator begin, Iterator end) -> expected< std::vector< std::future< void > >, std::error_code >
	Submit a range of `void()` callables in one go (non-throwing).
template<typename Iterator>
auto	submit_batch (Iterator begin, Iterator end) -> std::vector< std::future< void > >
	Submit a range of `void()` callables in one go (throwing).
template<typename Iterator, typename F>
void	parallel_for_each (Iterator begin, Iterator end, F &&func)
	Apply `func` to every element in [begin, end) in parallel.
Observers
auto	size () const noexcept -> size_t
	Number of worker threads in this pool.
auto	pending_tasks () const -> size_t
	Approximate count of tasks waiting in all queues.
auto	get_statistics () const -> Statistics
	Collect approximate performance counters.
Thread configuration
auto	configure_threads (std::string const &name_prefix, SchedulingPolicy policy=SchedulingPolicy::OTHER, ThreadPriority priority=ThreadPriority::normal()) -> expected< void, std::error_code >
	Name, schedule and prioritize all worker threads.
auto	set_affinity (ThreadAffinity const &affinity) -> expected< void, std::error_code >
	Pin all workers to the same CPU set.
auto	distribute_across_cpus () -> expected< void, std::error_code >
	Pin each worker to a distinct CPU core (round-robin).
Synchronisation
void	wait_for_tasks ()
	Block until all pending and active tasks have completed.
Tracing hooks
void	set_on_task_start (TaskStartCallback cb)
	Register a callback invoked just before each task executes.
template<typename Callback, std::enable_if_t<!std::is_same_v< detail::remove_cvref_t< Callback >, TaskStartCallback >, int > = 0>
void	set_on_task_start (Callback &&cb)
void	set_on_task_end (TaskEndCallback cb)
	Register a callback invoked just after each task completes.
template<typename Callback, std::enable_if_t<!std::is_same_v< detail::remove_cvref_t< Callback >, TaskEndCallback >, int > = 0>
void	set_on_task_end (Callback &&cb)

Detailed Description

High-performance thread pool optimized for high-frequency task submission.

Uses a work-stealing architecture: each worker thread owns a private WorkStealingDeque, and idle workers attempt to steal tasks from other workers' queues. A shared overflow queue absorbs bursts when all per-thread queues are full.

Optimizations for 1k+ tasks with 10k+ tasks/second throughput:

Work-stealing architecture with proper synchronization
Per-thread queues with efficient load balancing
Batch processing support for maximum throughput
Optimized wake-up mechanisms
Cache-friendly data structures with proper alignment
Performance monitoring and statistics

How task execution works: When you call submit(), the callable is wrapped in a std::packaged_task and placed into one of the per-worker queues (round-robin selection). A condition_variable then wakes one sleeping worker. The worker picks up the task from its own queue. If its own queue is empty, the worker tries to steal tasks from up to 4 other workers' queues (random selection). If no per-worker queue has work, the worker checks the shared overflow queue. If nothing is found at all, the worker sleeps for up to 100 microseconds before retrying.

Execution guarantees

Every successfully submitted task (submit() returned without throwing) is guaranteed to eventually execute, as long as the pool is not destroyed while shutdown() is draining.
submit() throws std::runtime_error if the pool is already shutting down. In that case the task is NOT enqueued and will NOT execute.
Tasks are executed in approximately FIFO order per queue, but the work-stealing mechanism means that the global execution order across all threads is non-deterministic. There is no ordering guarantee between two tasks submitted from different threads, or even from the same thread if they land in different worker queues.
The returned std::future becomes ready once the task has completed. You can call future.get() to block until the result is available, or future.wait() to just wait without retrieving the result.
If a task throws an exception, the exception is stored in the future. Calling future.get() will rethrow it. The worker thread itself continues to run and process further tasks.
shutdown() sets the stop flag and wakes all workers. Workers finish their current task and then drain all remaining queued tasks before exiting. The destructor calls shutdown() implicitly.

Thread safety: submit() and submit_batch() may be called from any thread concurrently. shutdown() is internally guarded and is safe to call more than once.

Exception handling: Exceptions thrown by tasks are caught inside the worker loop. They do not propagate to the caller directly, but are stored in the std::future returned by submit(). Call future.get() to observe or rethrow the exception. The worker thread is not affected and continues processing.

Statistics accuracy: Counters such as completed_tasks_, stolen_tasks_, and total_task_time_ are updated with std::memory_order_relaxed, so the values returned by get_statistics() are approximate and may lag behind the true counts by a small margin.

Blocking: wait_for_tasks() blocks the calling thread until every queued and currently active task has finished.

Lifetime: The destructor calls shutdown() and joins all worker threads. It is safe to let the pool go out of scope while tasks are still running; they will be drained first. Note that this means the destructor can block for a long time if tasks are slow.

Copyability / movability: Not copyable, not movable.

Note: Has overhead for small task counts (< 100 tasks) due to work-stealing complexity. Best for high-throughput scenarios like image processing, batch operations, etc.

Definition at line 562 of file thread_pool.hpp.

Member Typedef Documentation

◆ QueuedTask

using threadschedule::HighPerformancePool::QueuedTask = detail::move_callable<void()>

Definition at line 566 of file thread_pool.hpp.

◆ Task

using threadschedule::HighPerformancePool::Task = std::function<void()>

Definition at line 565 of file thread_pool.hpp.

Constructor & Destructor Documentation

◆ HighPerformancePool() [1/2]

threadschedule::HighPerformancePool::HighPerformancePool	(	size_t	num_threads = std::thread::hardware_concurrency(),
		size_t	deque_capacity = WorkStealingDeque<QueuedTask>::DEFAULT_CAPACITY,
		bool	register_workers = false )

inlineexplicit

Definition at line 579 of file thread_pool.hpp.

References threadschedule::WorkStealingDeque< T >::DEFAULT_CAPACITY.

Referenced by HighPerformancePool(), and operator=().

◆ HighPerformancePool() [2/2]

threadschedule::HighPerformancePool::HighPerformancePool ( HighPerformancePool const & )

delete

References HighPerformancePool().

◆ ~HighPerformancePool()

threadschedule::HighPerformancePool::~HighPerformancePool ( )

inline

Definition at line 602 of file thread_pool.hpp.

References threadschedule::drain, and shutdown().

Member Function Documentation

◆ configure_threads()

auto threadschedule::HighPerformancePool::configure_threads	(	std::string const &	name_prefix,
		SchedulingPolicy	policy = SchedulingPolicy::OTHER,
		ThreadPriority	priority = ThreadPriority::normal() ) -> expected<void, std::error_code>

inline

Name, schedule and prioritize all worker threads.

Each worker is named name_prefix + "_0", "_1", etc.

Returns: expected<void, std::error_code> - error if the OS rejected any configuration call.

Definition at line 1026 of file thread_pool.hpp.

References threadschedule::detail::configure_worker_threads(), threadschedule::ThreadPriority::normal(), and threadschedule::OTHER.

◆ distribute_across_cpus()

auto threadschedule::HighPerformancePool::distribute_across_cpus ( ) -> expected<void, std::error_code>

inline

Pin each worker to a distinct CPU core (round-robin).

Definition at line 1039 of file thread_pool.hpp.

References threadschedule::detail::distribute_workers_across_cpus().

◆ get_statistics()

auto threadschedule::HighPerformancePool::get_statistics ( ) const -> Statistics

inline

Collect approximate performance counters.

Definition at line 979 of file thread_pool.hpp.

References threadschedule::HighPerformancePool::Statistics::active_threads, threadschedule::HighPerformancePool::Statistics::avg_task_time, threadschedule::HighPerformancePool::Statistics::completed_tasks, pending_tasks(), threadschedule::HighPerformancePool::Statistics::pending_tasks, threadschedule::HighPerformancePool::Statistics::stolen_tasks, threadschedule::HighPerformancePool::Statistics::tasks_per_second, and threadschedule::HighPerformancePool::Statistics::total_threads.

◆ operator=()

auto threadschedule::HighPerformancePool::operator= ( HighPerformancePool const & ) -> HighPerformancePool &=delete

delete

References HighPerformancePool().

◆ parallel_for_each()

template<typename Iterator, typename F>

void threadschedule::HighPerformancePool::parallel_for_each	(	Iterator	begin,
		Iterator	end,
		F &&	func )

inline

Apply func to every element in [begin, end) in parallel.

The range is split into chunks and submitted as tasks. Blocks until all elements have been processed.

Definition at line 928 of file thread_pool.hpp.

References threadschedule::detail::parallel_for_each_chunked().

◆ pending_tasks()

auto threadschedule::HighPerformancePool::pending_tasks ( ) const -> size_t

inlinenodiscard

Approximate count of tasks waiting in all queues.

Definition at line 965 of file thread_pool.hpp.

Referenced by get_statistics().

◆ post()

template<typename F, typename... Args>

void threadschedule::HighPerformancePool::post	(	F &&	f,
		Args &&...	args )

inline

Fire-and-forget task submission (throwing variant).

Enqueues a callable without creating a std::packaged_task or std::future, giving roughly 3x higher throughput than submit() for tasks whose return value is not needed.

Exceptions

std::runtime_error If the pool is shutting down.

See also: try_post() for the non-throwing variant.

Definition at line 766 of file thread_pool.hpp.

References try_post().

◆ set_affinity()

auto threadschedule::HighPerformancePool::set_affinity ( ThreadAffinity const & affinity ) -> expected<void, std::error_code>

inline

Pin all workers to the same CPU set.

Definition at line 1033 of file thread_pool.hpp.

References threadschedule::detail::set_worker_affinity().

◆ set_on_task_end() [1/2]

template<typename Callback, std::enable_if_t<!std::is_same_v< detail::remove_cvref_t< Callback >, TaskEndCallback >, int > = 0>

void threadschedule::HighPerformancePool::set_on_task_end ( Callback && cb )

inline

Definition at line 1095 of file thread_pool.hpp.

References threadschedule::detail::make_copyable_callable().

◆ set_on_task_end() [2/2]

void threadschedule::HighPerformancePool::set_on_task_end ( TaskEndCallback cb )

inline

Register a callback invoked just after each task completes.

Parameters

cb	Receives the end time, the worker's `std::thread::id`, and the wall-clock duration of the task.

Definition at line 1087 of file thread_pool.hpp.

◆ set_on_task_start() [1/2]

template<typename Callback, std::enable_if_t<!std::is_same_v< detail::remove_cvref_t< Callback >, TaskStartCallback >, int > = 0>

void threadschedule::HighPerformancePool::set_on_task_start ( Callback && cb )

inline

Definition at line 1073 of file thread_pool.hpp.

References threadschedule::detail::make_copyable_callable().

◆ set_on_task_start() [2/2]

void threadschedule::HighPerformancePool::set_on_task_start ( TaskStartCallback cb )

inline

Register a callback invoked just before each task executes.

Parameters

cb	Receives the start time and the worker's `std::thread::id`.

Definition at line 1065 of file thread_pool.hpp.

◆ shutdown()

void threadschedule::HighPerformancePool::shutdown ( ShutdownPolicy policy = ShutdownPolicy::drain )

inline

Shut the pool down.

Parameters

policy drain (default) finishes all queued tasks; drop_pending discards queued tasks.

Definition at line 613 of file thread_pool.hpp.

References threadschedule::drain, and threadschedule::drop_pending.

Referenced by shutdown_for(), and ~HighPerformancePool().

◆ shutdown_for()

auto threadschedule::HighPerformancePool::shutdown_for ( std::chrono::milliseconds timeout ) -> bool

inline

Attempt a timed drain: finish as many tasks as possible within timeout, then force-stop remaining workers.

Returns: true if all tasks completed within the deadline, false if the timeout expired first.

Definition at line 656 of file thread_pool.hpp.

References threadschedule::drain, and shutdown().

◆ size()

auto threadschedule::HighPerformancePool::size ( ) const -> size_t

inlinenodiscardnoexcept

Number of worker threads in this pool.

Definition at line 959 of file thread_pool.hpp.

◆ submit()

template<typename F, typename... Args>

auto threadschedule::HighPerformancePool::submit	(	F &&	f,
		Args &&...	args ) -> std::future<std::invoke_result_t<F, Args...>>

inline

Submit a task, throwing on shutdown.

Equivalent to try_submit but throws std::runtime_error instead of returning an error code when the pool is shutting down.

Exceptions

std::runtime_error If the pool is shutting down.

Returns: std::future<R> that becomes ready when the task completes.

Definition at line 747 of file thread_pool.hpp.

References try_submit().

◆ submit_batch()

template<typename Iterator>

auto threadschedule::HighPerformancePool::submit_batch	(	Iterator	begin,
		Iterator	end ) -> std::vector<std::future<void>>

inline

Submit a range of void() callables in one go (throwing).

Exceptions

std::runtime_error If the pool is shutting down.

See also: try_submit_batch() for the non-throwing variant.

Definition at line 913 of file thread_pool.hpp.

References try_submit_batch().

◆ try_post()

template<typename F, typename... Args>

auto threadschedule::HighPerformancePool::try_post	(	F &&	f,
		Args &&...	args ) -> expected<void, std::error_code>

inline

Fire-and-forget task submission (non-throwing variant).

Returns: expected<void, std::error_code> – std::errc::operation_canceled on shutdown.

Definition at line 780 of file thread_pool.hpp.

References threadschedule::detail::bind_args(), and threadschedule::detail::make_move_callable().

Referenced by post().

◆ try_submit()

template<typename F, typename... Args>

auto threadschedule::HighPerformancePool::try_submit	(	F &&	f,
		Args &&...	args ) -> expected<std::future<std::invoke_result_t<F, Args...>>, std::error_code>

inline

Submit a task without throwing on shutdown.

Wraps the callable in a std::packaged_task and enqueues it. Returns an expected containing the std::future on success, or std::errc::operation_canceled if the pool is shutting down.

Template Parameters

F	Callable type.
Args	Argument types forwarded to `F`.

Parameters

f	Callable to execute.
args	Arguments forwarded to `f`.

Returns: expected<std::future<R>, std::error_code> where R = std::invoke_result_t<F, Args...>.

See also: submit() for the throwing variant.

Definition at line 691 of file thread_pool.hpp.

References threadschedule::detail::bind_args().

Referenced by submit().

◆ try_submit_batch()

template<typename Iterator>

auto threadschedule::HighPerformancePool::try_submit_batch	(	Iterator	begin,
		Iterator	end ) -> expected<std::vector<std::future<void>>, std::error_code>

inline

Submit a range of void() callables in one go (non-throwing).

Acquires the lock once per batch, distributing tasks across worker queues in round-robin fashion. Significantly more efficient than calling submit() in a loop for large batches.

Template Parameters

Iterator Forward iterator whose value_type is callable as void().

Returns: expected containing a vector of futures, or std::errc::operation_canceled on shutdown.

Definition at line 866 of file thread_pool.hpp.

Referenced by submit_batch().

◆ wait_for_tasks()

void threadschedule::HighPerformancePool::wait_for_tasks ( )

inline

Block until all pending and active tasks have completed.

Definition at line 1050 of file thread_pool.hpp.

The documentation for this class was generated from the following file:

thread_pool.hpp

Classes

Public Types

Public Member Functions

Detailed Description

Member Typedef Documentation

◆ QueuedTask

◆ Task

Constructor & Destructor Documentation

◆ HighPerformancePool() [1/2]

◆ HighPerformancePool() [2/2]

◆ ~HighPerformancePool()

Member Function Documentation

◆ configure_threads()

◆ distribute_across_cpus()

◆ get_statistics()

◆ operator=()

◆ parallel_for_each()

◆ pending_tasks()

◆ post()

◆ set_affinity()

◆ set_on_task_end() [1/2]

◆ set_on_task_end() [2/2]

◆ set_on_task_start() [1/2]

◆ set_on_task_start() [2/2]

◆ shutdown()

◆ shutdown_for()

◆ size()

◆ submit()

◆ submit_batch()

◆ try_post()

◆ try_submit()

◆ try_submit_batch()

◆ wait_for_tasks()