Documentation on KioMQ

Getting started

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Rust 2024 edition (the crate is built with edition = "2024").
A Tokio multi-thread runtime. Workers spawn tasks and, for sync processors, blocking threads.
Redis only if you use the redis-store backend.

Installation

cargo add kiomq

Or add it by hand:

[dependencies]
kiomq = "0.2.1"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

Cargo features

Feature	Default	What it enables
`redis-store`	✅	`RedisStore`, `SharedRedis`, `Config` — durable, distributed queues
`tracing`	—	Emits `tracing` spans and events from the internals

InMemoryStore is always available, with no feature flag and no external services.

Quick start

Mon, 01 Jan 0001 00:00:00 +0000

Async worker

The processor is a closure that receives an Arc<Store> and a Job, and returns Result<R, KioError>.

use std::sync::Arc;
use kiomq::{InMemoryStore, Job, KioError, Queue, Worker, WorkerOpts};

#[tokio::main]
async fn main() -> kiomq::KioResult<()> {
 let store: InMemoryStore<u64, u64, ()> = InMemoryStore::new(None, "demo");
 let queue = Queue::new(store, None).await?;

 let processor = |_store: Arc<_>, job: Job<u64, u64, ()>| async move {
 Ok::<u64, KioError>(job.data.unwrap_or_default() * 2)
 };

 let worker = Worker::new_async(&queue, processor, Some(WorkerOpts::default()))?;
 worker.run()?;

 queue
 .bulk_add_only((0..10u64).map(|i| (format!("job-{i}"), None, i)))
 .await?;

 let updating_metrics = queue.current_metrics.clone();
 // wait for all jobs to complete
 while !updating_metrics.all_jobs_completed() {
 tokio::task::yield_now().await;
 }

 worker.close();
 Ok(())
}

What each step does:

Core concepts

Mon, 01 Jan 0001 00:00:00 +0000

Job states

Every job is in exactly one state at a time. JobState also doubles as the event key you subscribe to — see Events .

State	Meaning
`Wait`	Ready to be picked up by a worker. The default.
`Prioritized`	In the priority sorted-set, waiting to be moved to active.
`Delayed`	Scheduled to run at a future timestamp.
`Active`	Currently being processed by a worker.
`Processing`	A worker has started executing the processor function.
`Stalled`	The worker holding the lock disappeared; pending recovery.
`Completed`	The processor returned `Ok`.
`Failed`	The processor returned `Err`, panicked, or stalled too many times.
`Paused`	The queue is paused; the job waits in the paused list.
`Resumed`	The queue resumed; the job is transitioning back to `Wait`.
`Progress`	A progress-update event — not a persistent state.
`Obliterated`	The queue was obliterated and the job was deleted.

The happy path is Wait → Active → Completed. A prioritised job starts in Prioritized; a delayed or cron job starts in Delayed.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Options are layered: QueueOpts sets the defaults for the whole queue, JobOptions overrides them per job, and WorkerOpts tunes the process that consumes them.

Queue options

QueueOpts is passed to Queue::new and applies to every job that does not override it.

use kiomq::{BackOffJobOptions, BackOffOptions, KeepJobs, QueueEventMode, QueueOpts,
 RemoveOnCompletionOrFailure};

let queue_opts = QueueOpts {
 attempts: 2,
 default_backoff: Some(BackOffJobOptions::Opts(BackOffOptions {
 type_: Some("exponential".into()),
 delay: Some(200),
 })),
 remove_on_fail: Some(RemoveOnCompletionOrFailure::Opts(KeepJobs {
 age: Some(3600), // keep for 1 hour
 count: None,
 })),
 event_mode: Some(QueueEventMode::PubSub),
 ..Default::default()
};

let queue = Queue::new(store, Some(queue_opts)).await?;

Field	Type	Default	Notes
`attempts`	`u64`	`1`	Default attempt limit for jobs that don’t set their own
`default_backoff`	`Option<BackOffJobOptions>`	`None`	Retry delay strategy for the whole queue
`remove_on_complete`	`Option<RemoveOnCompletionOrFailure>`	`None`	Retention for completed jobs; `None` keeps them
`remove_on_fail`	`Option<RemoveOnCompletionOrFailure>`	`None`	Retention for permanently failed jobs
`event_mode`	`Option<QueueEventMode>`	`Stream`	Stream (replayable) or pub/sub (broadcast-only) delivery
`repeat`	`Option<Repeat>`	`None`	Default repeat policy for every job on the queue

Job options

JobOptions is the third argument to add_job, and the second element of each bulk_add tuple. Anything left at its default falls back to the queue.

Scheduling & priorities

Mon, 01 Jan 0001 00:00:00 +0000

Priorities

priority is a score where lower runs first, and 0 means “no priority” — the job takes the ordinary Wait path.

use kiomq::JobOptions;

let urgent = JobOptions { priority: 1, ..Default::default() };
let ordinary = JobOptions { priority: 50, ..Default::default() };

queue.add_job("password-reset", payload, Some(urgent)).await?;
queue.add_job("weekly-digest", payload, Some(ordinary)).await?;

Prioritised jobs live in a sorted set and move to Active ahead of the plain wait list.

Errors & retries

Mon, 01 Jan 0001 00:00:00 +0000

Signalling failure

A processor reports failure by returning Err. The worker catches it, records the reason, and either schedules a retry or moves the job to Failed depending on the attempt limit.

use std::sync::Arc;
use kiomq::{Job, KioError, Store};

async fn charge(store: Arc<impl Store<Invoice, Receipt, ()>>, job: Job<Invoice, Receipt, ()>)
 -> Result<Receipt, KioError>
{
 let invoice = job.data.ok_or_else(|| {
 std::io::Error::new(std::io::ErrorKind::InvalidData, "missing payload")
 })?;

 // Any error that converts into KioError works — including io::Error.
 let receipt = billing::charge(&invoice).await?;
 Ok(receipt)
}

KioError has From conversions for io::Error, ParseIntError, SystemTimeError, JSON and Redis errors, so ? usually just works. For anything else, box it into GeneralError:

Events & observability

Mon, 01 Jan 0001 00:00:00 +0000

Subscribing

Queue::on listens to a single state; Queue::on_all_events listens to everything. Both return a Uuid you can use to detach the listener later.

use kiomq::{EventParameters, JobState};

// One state.
let listener_id = queue.on(JobState::Completed, |evt| async move {
 if let EventParameters::Completed { job_id, job_metrics, .. } = evt {
 tracing::info!(job_id, ?job_metrics, "completed");
 }
});

// Everything. The type parameters are <R, P> — the processor's return type
// and the progress type.
let all_id = queue.on_all_events(|evt: EventParameters<u64, u8>| async move {
 tracing::debug!(?evt, "queue event");
});

// Detach when you no longer care.
queue.remove_event_listener(listener_id);
queue.remove_event_listener(all_id);

Callbacks are async: they return a future, which the queue drives. Keep them short — a slow listener delays event delivery, so hand heavy work off to a spawned task.

Progress updates

Mon, 01 Jan 0001 00:00:00 +0000

Reporting progress

The third generic parameter of Job<D, R, P> is the progress type. It can be anything serialisable: a percentage, a struct, an enum of named phases.

use std::sync::Arc;
use kiomq::{Job, KioError, Store};

async fn processor<S: Store<u64, u64, u8>>(
 store: Arc<S>,
 mut job: Job<u64, u64, u8>,
) -> Result<u64, KioError> {
 // update_progress persists to the store and emits a progress event.
 job.update_progress(50u8, store.as_ref()).await?; // 50% done
 Ok(job.data.unwrap_or_default() * 2)
}

Two things to note: the job must be mut, and the store handed to your processor is exactly what update_progress needs — pass it as store.as_ref().

Process & worker metrics

Mon, 01 Jan 0001 00:00:00 +0000

KioMQ collects three kinds of metrics in the background. Queue counters are in-memory and free to read; process and worker metrics are published to the store with a TTL and refreshed by the queue’s timer subsystem.

Queue counters

queue.current_metrics is a cheap, always-live handle. The counters are refreshed from the store whenever get_metrics() is called, so between calls they may be slightly stale — prefer the helper methods over reading raw fields.

Backends

Mon, 01 Jan 0001 00:00:00 +0000

Every store implements the same Store<D, R, P> trait, so switching backends is a change to two lines of setup and nothing else.

Store	Feature flag	Durable	Multi-process	Best for
`InMemoryStore`	always available	No	No	Tests, dev loops, short-lived in-process work
`RedisStore`	`redis-store` (default)	Yes	Yes	Production fleets spread across machines

In-memory

Ideal for tests, development, and short-lived tasks. No external dependencies, no serialisation round-trip.

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

Releases follow semantic versioning . Entries are grouped by type — features, bug fixes, chores — and each links to the commit it came from. Versions are published to crates.io , and the matching API docs are on docs.rs .

v0.2.1

Latest 5 July 2026 release notes ↗

Bug Fixes

when using sysinfo, refresh all processes by @spencerjibz af127a9 (process_tree_tracker)
update existing metrics entrics instead of adding new ones if they’are expired by @spencerjibz 5b6716e (inmemory-store)
metrics::is_idle shouldn’t include last_id check by @spencerjibz 817d36b (queue_metrics)
fix logic bug in pause_or_resume_workers by @spencerjibz 8102a90
considered timed out runs during the test for processing delayed jobs by @spencerjibz 8a5171b (worker_tests)

Chores

remove unnecessary block_in_place call in the RedisStore.update_job_progress_sync method. by @spencerjibz d46e2da (redis-store)
add sysinfo and heapster as dependancy for collecting Node/Process telemtry by @spencerjibz 6a45bfe (deps)
enable the parking_lot features for tokio by @spencerjibz 44aa07a (deps)
public expose snapshots of process monitoring essentials(metrics & consts) by @spencerjibz 6b36301
expose process-metrics store and fetch methods via the Queue instance by @spencerjibz 2cacd74
run all timers in one task globally by @spencerjibz 5afacdb (timers)
use flume as the channel instead of tokio::mspc by @spencerjibz d660cab (deps)
reduce Worker_state_ttl to 100 seconds by @spencerjibz da7c0df
add cpu-thread-count to ProcessMetricsCollector struct and memory & cpu usage to ProcessMetrics struct by @spencerjibz 1f9e0a7 (process-metrics)
running Re-register timer for even idle workers by @spencerjibz 7d850be (timers)
add tokio-stream as a dependency to create useful stream on tokio utils by @spencerjibz 5dc6320 (deps)
remove flume in favour of tokio-broadcast by @spencerjibz 2aa3bef (deps)
Reduce channel buffers and simplify streams by @spencerjibz 7c9ed08
remove dashmap and add parking_lot by @spencerjibz f206257 (deps)
add compact_str as dependancy to reduce memory usage by @spencerjibz f09fc11 (deps)
add hostname crate and use it to get the machine’s name instead by @spencerjibz 53f0f0b (deps)
fix compile errors when tracing is enabled caused by compact_str refactor by @spencerjibz e7babde (tracing)
clean up errors by @spencerjibz c173c3f
fix error in redis-queue-tests, use to_string instead of to_compact_string by @spencerjibz 2c71a52 (tests)
add a timer existence check for queues by @spencerjibz 05357f0 (timers)
put individual queue_timer tasks to sleep for n millis where there is no work by @spencerjibz 89a6208 (timers)
remove triggerpromotion timers by @spencerjibz 3f1ea45 (timers)
add a worker to global registry on creation but register it’s timers later. by @spencerjibz 490d376 (timers)
add more metadata about workers to the process_metrics by @spencerjibz e3dd09e (process_metrics)
remove lock checks in queue::move_to_finish_or_failed. by @spencerjibz 7c71b0d (queue)

Documentation

add detailed documentation for the process-metrics module by @spencerjibz be703b8
add ProcessMetrics section to README and fix rustdoc warnings by @spencerjibz 3f9e1e9

Features

Add node/process metrics using the heapster and systinfo crates by @spencerjibz 7b50897
add publish process metrics to worker-timers and a dedicated health check for each worker by @spencerjibz 4a7ec8d (Timers)
add process metrics collection and fetching to RedisStore and InMemoryStore stores by @spencerjibz 772217a (stores)
add a lazy better timed_map implementation without locking on purged_expired method by @spencerjibz 9795f02

Refactoring

Extract worker metrics into metrics module by @spencerjibz 1c053f7 (metrics)
rename the node_metrics module && NodeMetrics struct to process_metrics & ProcessMetrics respectively by @spencerjibz 1e0153f
process metrics collector and add serde support by @spencerjibz 3f1f0b8
move timers(stall_check,extend_lock& collect_metrics) to the queue instead of workers by @spencerjibz bda760b
move all timer-logic to a singleton processor by @spencerjibz 2522114
move fetching scheduled delayed jobs to a dedicated timer by @spencerjibz e471aa3
move worker pausing logic from the timer_reciever-task to each worker’s main loop. by @spencerjibz a98d90c (worker)
Queue::add_worker now also registers existing workers by @spencerjibz 1820372
Switch queues to SkipMap instead of a ttl-timed map by @spencerjibz 9490dbf (ProcessMetricsCollector)
use a tokio::watch channel for process-metrics updates instead of the same flume channel as timers by @spencerjibz 5efc7a6
remove usages of dashmap in favour of parking_lot by @spencerjibz e31bf37
use std::process::id() instead of the sysinfo’s method by @spencerjibz 2ba550b
use different strategies for cpu metrics for linux and other platforms by @spencerjibz 3364fe9 (process_metrics)
use tokio::utils::DelayQueue instead of futures-intrusive && futures-delay-queue by @spencerjibz 3715a22 (delay_queue_map)
remove module delay_queue_map.rs to concurrent_timed_map.rs by @spencerjibz 4f3ca57
merge the event_stream_task and the timer_task_listener task into one background task. by @spencerjibz 32d8c7c (stores)
use tokio::join instead of a tokio::select to run both the stream_listener_task and timer_task. by @spencerjibz 9d75cf2 (queue_timer)
periodically update queue metrics inside the worker_metrics collection timers instead of within stores(redis-store for instance). by @spencerjibz 8af8891
Flattened down the worker’s reactor loop to a single cleanly controlled loop. by @spencerjibz 9e54443 (worker_reactor_loop)
replace numeric as casts with num_traits::AsPrimitive. by @spencerjibz 653c942

Release

0.2.0 (#80) by @github-actions[bot] (#80 ) (release)

Contributors

@github-actions[bot]
@spencerjibz

v0.2.0

6 May 2026 release notes ↗

Bug Fixes

yield in busy-wait loops and use saturating_sub by @spencerjibz d88463b (worker_test_helper)

Chores

remove atomig as dependency by @spencerjibz b212508
make dotenv an optional dependency by @spencerjibz (#75 )
remove trailing comma by @spencerjibz 182eacf (clippy)
remove static counters (id-counter && pc-counter) by @spencerjibz 8b525b7 (redis-store)
update tests and examples to use a shared redis connection by @spencerjibz 052af3f
update README.md to reflect updates to the RedisStore and update_progress method by @spencerjibz 3ae61da (docs)
update redis & deadpool-redis to v1.2.0 and v0.230 respectively; update everything to reflect this by @spencerjibz c6237d7 (deps)
bump rustls-webpki from 0.103.10 to 0.103.12 by @dependabot[bot] (#76 ) (deps)

Features

redis connection sharing between different store instance by @spencerjibz e827431 (RedisStore)
add a better implementation of skipmap based deque. by @spencerjibz 419f62e (InMemoryStore)

Other

Merge branch ‘redis-improvements’ of https://github.com/KioHQ/kiomq into redis-improvements by @spencerjibz 1f6c699
Merge branch ‘main’ into redis-improvements by @spencerjibz a7858ba
Merge pull request #78 from KioHQ/redis-improvements by @spencerjibz (#78 )

Refactoring

replace of all usages for atomig::Atomic with crossbeam::atomic::AtomicCell by @spencerjibz 2709ba6
use crossbeam::atomic::AtomicCell instead of manual Atomics(AtomicUsize, AtomicBool & AtomicU64) by @spencerjibz 71bb7af
remove mutex usage to guard per task metrics-histogram by @spencerjibz 25580cf (worker)
replace boolean flags with TimerState by @spencerjibz a6a826f (timers)
convert Store trait methods to async by @spencerjibz 2d87eab
use redis-transactions for atomic job insertion and remove counters per store by @spencerjibz 5ba4a3d (redis)
use redis-transactions for atomic job insertion and remove counters per store by @spencerjibz 8ae1b7e (redis)

Release

0.1.3 by @spencerjibz (#74 ) (release)

Tests

fix doc-test errors included by shared-redis-connection feature by @spencerjibz 219a61f (doc-test)

Contributors

@dependabot[bot]
@spencerjibz

v0.1.3

5 April 2026 release notes ↗

Bug Fixes

fix typo in cargo.toml and fmt the file by @spencerjibz 42499eb (cargo)
add derive debug and display to fix rebase issues by @spencerjibz 9f253ab
logic bug in end-to-end benchmarks by @spencerjibz (#73 )

Chores

add hdrhistogram for poll_duration_metrics collection by @spencerjibz 9ec9193 (deps)
implement deref & deref_mut traits for the HistogramWrapper by @spencerjibz 04b4484 (WorkerMetrics)
make poll_duration_history public in TaskInfo by @spencerjibz 36c45bc (WorkerMetrics)
export TaskInfo publicly by @spencerjibz 1cd7176
use async-mutex only for guarding Histogram per task by @spencerjibz 828fbc3
adhere to clippy::depantic by @spencerjibz (#66 ) (clippy)
update release name format in release GitHub Actions workflow by @spencerjibz 4b455b4 (ci)
remove buildjet cache usage from action-workflows by @spencerjibz 54b9898 (actions)
delay-queue-wip by @spencerjibz 82e57bc
add futures-intrusive && futures-delay-queue by @spencerjibz 576844b (deps)
apply clippy lint recommendations by @spencerjibz b94c0a9 (lints)
reinstate job promotion timer and move delayed job pick up logic to the worker main_loop instead of timers by @spencerjibz 2b67228
chore: log job state transitions in the queue(when tracing is enabled) & by @spencerjibz (#67 )
bump rustls-webpki from 0.103.4 to 0.103.10 by @dependabot[bot] (#68 ) (deps)
bump tar from 0.4.44 to 0.4.45 by @dependabot[bot] (#72 ) (deps)
bump slab from 0.4.10 to 0.4.12 by @dependabot[bot] (#71 ) (deps)
bump bytes from 1.10.1 to 1.11.1 by @dependabot[bot] (#69 ) (deps)
remove xutex as dependency; use tokio’s mutex for async cases by @spencerjibz 6940d76 (dep)

Documentation

add refactor as a type of change in pull request template by @spencerjibz b76b5a5

Features

add a poll_duration histogram for get collect statistcs by @spencerjibz 312b12a (WorkerMetrics)
add redis-version checking and the field to the redis-store by @spencerjibz 82406a7 (RedisStore)

Other

chore(deps) add dashmap as a new dependency by @spencerjibz ce97639

Performance

use early returns when fetch jobs from empty collections by @spencerjibz 0b2a563 (InMemoryStore)
store worker_metrics in redis as a hash instead of individual values by @spencerjibz (#70 ) (RedisStore)

Refactoring

add the last_updated_time and ttl fields to worker_metrics by @spencerjibz 935f680 (WorkerMetrics)
add a clean lockfree timer design based off channels by @spencerjibz 20f9759 (DelayedQueue)
require Sync trait for Data type and Store type (D&S) by @spencerjibz 43d3e2d
move job promotion from this DelayQueue to an independent tasks with a dedicated track by @spencerjibz 383b7c5 (DelayTimer)
use a channel based delayed-queue from futures-delayed-queue crate && remove usage of block_in_place in this module too by @spencerjibz 7b76f81 (TimedMap)
use the same channel queue instead of tokio::util::DelayQueue by @spencerjibz 7e088c3 (DelayQueueTimer)
simplify delay_queue_timer and include job promotion in it too by @spencerjibz 5c0f0e5
use dashmap::HashMap in timedmap & delayed_queue-map by @spencerjibz 0685cdf
use dashmap::Hashmap instead crossbeam::skipmap to store runnings in the worker by @spencerjibz 3f308a2

Release

0.1.2 by @spencerjibz (#65 ) (release)

Contributors

@dependabot[bot]
@spencerjibz

v0.1.2

14 March 2026 release notes ↗

Bug Fixes

update metrics after pausing queue and fix inmemorystore pause behaviour by @spencerjibz 7a12cea (queue)

CI

use –offline instead of –locked for tests in ci by @spencerjibz 54247c5
update git-cliff config to generate well-grouped change and release notes by @spencerjibz 9388bc1 (release-pipline)

Chores

revert usage of –offline flag in ci by @spencerjibz 120baba
regenerate changelog with updated git-cliff configuration by @spencerjibz 7a076ab (changelog)
remove parking_lot in favour of xutex (only one is needed) by @spencerjibz 5045bb6 (deps)
make simd-json as optional dependancy by @spencerjibz 25f5b59 (deps)
fix a typo in readme, && ignore some code-snippets in docs tests by @spencerjibz df0c42a (README)
add more useful keywords for the crates by @spencerjibz 0163531 (crates.io)
fix missing commas in cargo.toml by @spencerjibz 94c3d64 (cargo.toml)
limited the crate keywords to five(publish limit) by @spencerjibz ca26156 (crates.io)

Refactoring

restart all timers instead of resuming with stored state by @spencerjibz 2ab4cd0 (delayed_queue_timer)

Release

0.1.1 by @spencerjibz (#63 ) (release)

Contributors

@spencerjibz

v0.1.1

7 March 2026 release notes ↗

Bug Fixes

lint warning about links by @spencerjibz 916016b (rustdoc)

CI

exclude unnecessary folders and files from cargo publish by @spencerjibz 9c88c32 (Crates.io)
workflow should references to the old library version in the readme by @spencerjibz (#62 ) (cd-workflow)

Chores

clippy fix (unused) by @spencerjibz 68f67e5

Documentation

add ci-build, crates.io, docs.rs & licence badges to readme by @spencerjibz 55a4caa (readme)
update code snippets to include waiting for job completion by @spencerjibz 770f85c (readme)
add example code-snippet of redis and rocksdb usage by @spencerjibz 0d45ff9 (readme)
use readme in lib.rs, correct links in docs by @spencerjibz 25ba74b
minor tweak in rocksdb usage example by @spencerjibz 3ce01a7 (readme)

Release

0.1.0 by @spencerjibz (#61 ) (release)

Contributors

@spencerjibz

v0.1.0

7 March 2026 release notes ↗

Bug Fixes

fix support for job prioritization by @spencerjibz 8e514e2
fix job retrial after failure by @spencerjibz (#11 )
fix retrials & remove crossbeam-skiplist by @spencerjibz 800fb39
failing missed deadline jobs by @spencerjibz 8595655
fix serializing redis-stream-event by @spencerjibz 71c0ccd
skipmap.len is just an len appromixation, use atomic counter for this instead by @spencerjibz (#14 )
fix flaky test by @spencerjibz (#15 )
correctly serialize queue-events by @spencerjibz f09e065
promoting delayed jobs by @spencerjibz 0d36fa8
fix complete rocksdb-store::add_bulk methods by @spencerjibz 01b2f3d
counter bugs in in-memory and rocksdb store by @spencerjibz 85c20ed
fix overflow and simple clean up by @spencerjibz 234f7f5
fix out of range errors by @spencerjibz (#17 )
fix typo by @spencerjibz b0852f4
bug fix by @spencerjibz 5f08980
correct naming of metrics by @spencerjibz 9b1e9c7
bug fixes by @spencerjibz 2fd910f
set return value & add timeout to listening to events by @spencerjibz e71a2d3
timedmap tests by @spencerjibz 152e2f5
bug fix: locks.len_exnpire is async by @spencerjibz c71823e
inmemory-store, only update stored-metrics from publish_event by @spencerjibz 6ff80a4
updating prioritized and failed job counts in queue-metrics by @spencerjibz cdd2726
pass the right attempt for failed jobs by @spencerjibz e672423
redis-store returns pause_count instead of failed_job cout by @spencerjibz 0f027d5
typo in fetch_worker_metrics in redis-store by @spencerjibz 9d17207
cargo fmt by @spencerjibz 3ab26ed
fetching_metrics from redis-store by @spencerjibz 0173d57
fix flawky tests by @spencerjibz 9e84dee
fix tests by @spencerjibz (#49 )
tracing::Instrument::instrument is ambigious to TaskMonitor.instrument by @spencerjibz 6e6f342 (method-ambiguity)
stop relying on timestamps to generate release.md by @spencerjibz 3b57340 (ci)

CI

add ci tests by @spencerjibz 04c4110
Merge pull request #2 from spencerjibz/add-ci by @spencerjibz (#2 )
re-export both backtrace tracing macros by @spencerjibz 3d797ca
pass down removal options from queue if not specified for a job by @spencerjibz be9473f
efficiently pausing workers with global activity count, let workers finished all their work before pausing by @spencerjibz (#12 )
batch pipeline commands by @spencerjibz fac9673
format spacing in readme by @spencerjibz 79cc9ac
fix and improve ci with caching & using nextest by @spencerjibz c696b5c
don’t use release flag in ci by @spencerjibz 740cdda
use keydb and improved workflow by @spencerjibz 62e06c4
use redis instead of keydb in ci by @spencerjibz 4bfbd33
fix ci by @spencerjibz d671d70
add the tracing crate by @spencerjibz 26b15a0
add tracing and tracing-subscriber to examples by @spencerjibz 67196ab
clean and add tracing to examples by @spencerjibz aec1a40
gate tracing under a feature flag and make dep optional by @spencerjibz (#34 )
Apply code review feedback: remove explicit type in loop, drop unused return value assignment by @Copilot 223ecab
Migrate workflows to Blacksmith by @blacksmith-sh[bot] 18f8dc2
run store tests in concurreny in ci by @spencerjibz 1386a80
add a release-workflow by @spencerjibz aab3e02
populated CD workflow file by @spencerjibz (#51 )
add create generation to workflow by @spencerjibz 7b85613 (cd-workflow)
Fix remaining clippy errors and incorporate release-pipeline lint relaxation by @Copilot 652c1b6
fix cd pipeline by @spencerjibz 9da2cb8
add rocks build time system dependencies for publishing job by @spencerjibz 73c036f (cd-pipeline)
chose better category slugs for publishing crate by @spencerjibz a7a8478 (cd)

Chores

cargo fmt by @spencerjibz a7eb821
add changelong with git-cliff by @spencerjibz 165824e
add author, description, links and licence metadata by @spencerjibz 09dfba9 (cargo.toml)
add best practice library clint protections by @spencerjibz f28968c
unused/dead_code elimination by @spencerjibz dac2919
add missing debug implementations by @spencerjibz 6015372
only expose few methods publicity & tracing vital calls by @spencerjibz 2b79b7c (queue)
add more tracing spans by @spencerjibz fa75a53 (tracing)
manual clippy –fix by @spencerjibz 31456a6 (clippy)
more docs in worker_opts by @spencerjibz d561826
relax some lint constricts by @spencerjibz 0967db7
fix lint warnings in tests by @spencerjibz 24dd862
add clippy:nursery recommendation & fmt by @spencerjibz (#52 ) (clippy)
use the right branch for releases by @spencerjibz f4f4c63 (cd)
use real sermver version of crates by @spencerjibz a6e8f58 (cargo)
use real sermver version of crates by @spencerjibz 5c96437 (cargo)

Documentation

add readme by @spencerjibz 7458a6a
update readme by @spencerjibz 4ea9557
Update README.md by @spencerjibz 69c92b5
Update README.md by @spencerjibz 2c1749a
update readme to reflect current developement progress by @spencerjibz d19d9cc
Update README.md for clarity and formatting by @spencerjibz 9753782
update Readme by @spencerjibz f583669
add extras to readme by @spencerjibz (#45 )
Add logo and project overview to README by @spencerjibz 6e40043
Add comprehensive Rustdoc documentation across the kiomq codebase by @Copilot a62c4b7
Add Rustdoc for RedisStore struct, fields, and public methods; doc fetch_redis_pass and get_queue_metrics by @Copilot b9c4640
Fix updating_metrics bug in README examples and add panics/framed section by @Copilot decbf13
add doc comment and example to the framed re-export in lib.rs by @Copilot 7c1eb18
condense README to match lib.rs conciseness (44% shorter) by @Copilot 4fbc363
restore Progress updates section and improve Panics & errors wording by @Copilot 71c7dc7
add compact table of contents to README by @Copilot (#50 )
Fix clippy warnings: visibility, casts, docs, match arms, async, and more by @Copilot db811ff
typo by @spencerjibz d221802

Features

add basic required deps by @spencerjibz 3405f0a
add simple queue impl with a job queue too by @spencerjibz 7ee4ec8
add job progress updating by @spencerjibz f64948e
add base worker by @spencerjibz db9b270
add timer implementation by @spencerjibz 34188fa
add worker options by @spencerjibz 7519a91
add backtrace_capture utils by @spencerjibz 8626a5a
add more utils to wokrer by @spencerjibz 694ecf3
add timer queue functions by @spencerjibz cbed76e
add timers to worker by @spencerjibz 9121195
add support for storing stacktraces and errors separately by @spencerjibz 6677dd6
add events to queue & worker by @spencerjibz (#4 )
support queue-obliteration by @spencerjibz c158955
add backoff utils by @spencerjibz dabd8f1
add joinError to KioError type by @spencerjibz e09d35b
add working video_transcoding-example by @spencerjibz 9dd62c4
add job options with delay by @spencerjibz d90d537
add priority to job_data and priority helpers by @spencerjibz 922e358
enable scheduling without a seperate timer by @spencerjibz 8aa3e2c
add the right job status for delayed jobs by @spencerjibz (#9 )
add delay field to event_stream by @spencerjibz b73346e
add and move events into their own module by @spencerjibz 75d93f1
add stream-listener-task by @spencerjibz 1474263
add queue-stream-event by @spencerjibz 6ab4b9a
add job-removal options by @spencerjibz b2d6bb2
add tokio-console by @spencerjibz 65e8fb1
add queue options to example by @spencerjibz 839faac
add support for bulk_add by @spencerjibz 4840b76
add task-notifiers & waiting-job-count to metrics by @spencerjibz edcf098
add methods to bulk add jobs with returning them by @spencerjibz a1b9d67
add proesssing to JobState and clean up events by @spencerjibz 855f64c
add repeat opts with cron support and test them in example by @spencerjibz 2504142
add support for cron as inital delay by @spencerjibz c20932c
add obliterate as an event, rm prioritized collection on obliteration by @spencerjibz 7e20e2b
add sample queue tests by @spencerjibz 8aab52f
add worker state as an enum by @spencerjibz dc542d3
add worker tests by @spencerjibz 6ebecf2
add client to queue, support both favours of processors (sync and async) by @spencerjibz 9039e2b
add nextest-config file by @spencerjibz 28b11cc
add async-trait by @spencerjibz 981a0ce
add store to examples by @spencerjibz c1e6308
add rocksdb as a store with a feature flag by @spencerjibz 74f9a92
add basic example with redis-store by @spencerjibz c202069
add an in-memory-store and its example by @spencerjibz 0ed568d
add timed-map to use in the in-memory-store by @spencerjibz 26b8169
add timed locks to both stores by @spencerjibz 101be3d
implement store.remove for in-memory-store && fix typo in JobState enum" by @spencerjibz 1725ff9
add some utility functions by @spencerjibz da06fb8
add more fields to job-metrics by @spencerjibz 8bf8942
add support for purging expired entries for store by @spencerjibz 0f0f12b
add job metrics to complated event by @spencerjibz 64b6b6d
enable disabling expirations on customTimedMap by @spencerjibz 0a84fde
add delay from options to job_metrics by @spencerjibz (#30 )
add resoure span to the worker by @spencerjibz b152f98
add logging to main_task and bug fix: stop main_loop on cancellation by @spencerjibz c3d8613
add logging to timers by @spencerjibz dcf5070
add logs and remodel loop and timer to promote delayed jobs too by @spencerjibz d2b096f
add a resource-span to queue by @spencerjibz e9b0cae
issue-templates by @spencerjibz b4c02df
add criterion benchmarks using inmemory-store by @spencerjibz 75c3427
add an action for crition benchmarks by @spencerjibz 8bdaca1
add missing meta-data for action by @spencerjibz 0c911e1
add permissions for action to post results by @spencerjibz (#39 )
add tests for panics in processors by @spencerjibz 893fc33
add tokio-metrics as a dependency by @spencerjibz 639e34a
add worker_metrics collection to worker and setup periodic heartbeat/collection timer by @spencerjibz 8c760dc
add metrics collection to in-memory-store by @spencerjibz a11d63c
add metrics collection for redis-store by @spencerjibz e0c6a4e
add timestamps to track metrics updates by @spencerjibz cab75f0
add asset folder with logo by @spencerjibz 5ec1bf9
add a helper macro for worker_tests by @spencerjibz 2a6158d
add isolated tests for each store (inmemory or redis-store) by @spencerjibz 2aeedc9
add helper macros for queue-tests for diffrent stores by @spencerjibz e970dc6

Other

Initial commit by @spencerjibz 0da1a6c
rust library template 32dfa0a
base job impl by @spencerjibz c1646b9
remove custom field-serializers by @spencerjibz aae3824
use dashmap and use owned values by @spencerjibz fad0c4d
use some generics for queue methods by @spencerjibz 43a7b47
wip by @spencerjibz 82c12f1
reorganise modules and supplement worker by @spencerjibz c414985
clippy fix by @spencerjibz 110e3b9
Merge pull request #1 from spencerjibz/base-impl by @spencerjibz (#1 )
ignore panic_test in backtrace_utils by @spencerjibz ad436a9
make timer simple by @spencerjibz 6e4830a
minor fixes by @spencerjibz d1d36de
complete running worker with panic catches by @spencerjibz 4c13f3f
use an serializable backtrace by @spencerjibz 26830bf
remove unecessary deps and use std version of backtrace by @spencerjibz 3bba177
create similar stacktraces by @spencerjibz 61f9702
clippy fix by @spencerjibz 19267d7
Merge pull request #3 from spencerjibz/working-runner by @spencerjibz (#3 )
readme edit by @spencerjibz 1d6a267
make job_state serializable from redis by @spencerjibz 77d94a6
remove arc-swap by @spencerjibz 570f33e
expose the worker’s cancellation token by @spencerjibz 6474c44
access the current job count from queue by @spencerjibz d95fabf
serialize datetimes as microsecond timestamps by @spencerjibz de2ccfe
ignore video files by @spencerjibz d79837d
improve backoff and add a method to fetch metrics to a job by @spencerjibz 149862b
create a sample video file if none exists by @spencerjibz (#5 )
Next stage (#6) by @spencerjibz (#6 )
clean up after running transcoding by @spencerjibz cc9af51
download and install ffmpeg if its not available (#8) by @spencerjibz (#8 )
simple code refactor by @spencerjibz 5a73a8f
add support for skipping first_tick by @spencerjibz dfd8f29
job-schedulering-timer template by @spencerjibz 6379443
set concurrency to number of logical cpus by @spencerjibz c3601b8
set number of cpus as default concurrency option by @spencerjibz bd26ab4
better delay support by @spencerjibz 1fa6f20
completed impl by @spencerjibz 732da55
only remove jobs if found by @spencerjibz a1ef2b3
make move_to_active_result debug printable by @spencerjibz 375f1e7
re-use timer for scheduling delayed jobs, remove futures_concurrency primitive for running task by @spencerjibz 345de47
use crossbeam queue, yield in busy loops to prevent race conditions by @spencerjibz c3aa115
emit progress to stream when it’s updated by @spencerjibz a9c9c52
emit events directly from redis-streams by @spencerjibz (#10 )
rebase conflict fix by @spencerjibz de77c24
add live metrics and opts to the queue by @spencerjibz 85368ce
remove dashmap, completely lockfree impl by @spencerjibz 0a0159b
update example to support all job features: delaying, prioritization by @spencerjibz 1f2bac8
clean up by @spencerjibz 18feb2d
use lockfree version of typed-emitter by @spencerjibz 2396420
update uuid:dep by @spencerjibz 53322e2
boxed some futures to reduce their size by @spencerjibz b05890e
wip by @spencerjibz f5ba7b0
only return joinhandle if timer is running by @spencerjibz a7254fc
Revert “fix retrials & remove crossbeam-skiplist” by @spencerjibz d9a236e
prioritize delayed jobs by @spencerjibz c283935
fail jobs that miss delay reschedule by @spencerjibz 34982c9
reduce cpu usage during idle times by @spencerjibz ba7b441
make timers pausable by @spencerjibz 569b423
pause both timer when queue and worker are idle by @spencerjibz 8a6e9b9
improve examples by @spencerjibz 7856f22
reduce string usage by @spencerjibz 979c1e0
Support either pubsub or redis_stream as queue-event-stream by @spencerjibz 68c16dc
wip by @spencerjibz ab5da6a
refactor and cleanup by @spencerjibz a1104ef
support for both modes by @spencerjibz 3945796
use simple types in job and queue by @spencerjibz a7733f1
wip by @spencerjibz 6754e9f
use simd_json by @spencerjibz 6938133
cheap parsing when using redis-stream-events by @spencerjibz 4144ac2
restore examples by @spencerjibz e4c1577
make bulk_add faster by @spencerjibz 5c8ff95
clippy fix by @spencerjibz 06521b4
repeat-opts-working progress by @spencerjibz 38f3a4f
simple fixes by @spencerjibz 1592c5a
increment ids correctly by @spencerjibz 6d44ee7
update examples by @spencerjibz 00129bf
code refactor by @spencerjibz (#13 )
update the job_id after addition by @spencerjibz f27f96e
more queue tests by @spencerjibz 339f4d8
:emitter_on is not an async function: correct this by @spencerjibz 538a281
remove tokio_shared_rt and use a multithread runtime for worker tests by @spencerjibz 060620a
fetch number of cpus by @spencerjibz 3ed68b1
support both sync and async callbacks by @spencerjibz a26d72f
update examples and tests by @spencerjibz 44d9e75
reorganise code structure by @spencerjibz e72038a
make job-update progress sync by @spencerjibz 54ab3a4
update examples by @spencerjibz 0efb189
restore connection-example by @spencerjibz 0728dcc
Format README.md with proper headings and lists by @spencerjibz b83ec7e
make all errors public and the worker’id by @spencerjibz 8ef613f
return an error if a worker is already running and a called to run is made by @spencerjibz 23dc2cd
make CollectionSuffix enum and worker_state field pub by @spencerjibz 5c07d40
create a store trait and a redis store by @spencerjibz 438f627
clean up and use dynamic store logic in queue-impl by @spencerjibz a131a23
refactor to use dynamic stores by @spencerjibz ddae040
Integrate the Store trait into queue, worker and utils function by @spencerjibz 60ccf6e
update tests with store by @spencerjibz 8d2cd80
clippy fix by @spencerjibz 47f6a30
make collectionSuffix serialized as an u64 tag by @spencerjibz 86efd8b
move event-stream-handle creating from store implementations by @spencerjibz 2762b50
complete Store trait impl for redis-store by @spencerjibz eee2409
dont run rocksdb-example in tests by @spencerjibz 29ffc99
yield in task picker by @spencerjibz e32a34f
use parking_lot primitives instead by @spencerjibz (#16 )
simple improvement to in-memory-store by @spencerjibz 41b3654
refactor store.set_fields and move the move_to_state method out of the store by @spencerjibz 2b1467a
clean and reduce store methods by @spencerjibz ecb50b4
expose important methods from store by @spencerjibz 51f8715
clean up events && split functions in utils by @spencerjibz 61aebeb
remove stream-handle, use emiiter directly to publish events for rocksdb and in-memory-store by @spencerjibz 506da8b
clippy fix@ by @spencerjibz 61d5a38
use a checked insert method by @spencerjibz fc80401
use a delay-queue for timers instead of spawning task (previous timer) by @spencerjibz (#18 )
use tasktracker, add graceful shutdown by @spencerjibz (#19 )
make redis-store optional by @spencerjibz 88c960e
make redis-store a default feature by @spencerjibz a6cfee3
merge similar examples into a few with feature flags by @spencerjibz (#21 )
use new timedmap by @spencerjibz 183a2f9
only purge expired when ther e is some expiring soon by @spencerjibz 9f76297
use atomiccell instead of arc-swap and use tokio-mutex for timed-map by @spencerjibz 183919d
Refactor set_key to use Key directly by @spencerjibz f14b97c
clippy fix && reduce critical section for race condtion (when both mutexes are held by @spencerjibz 785879b
purge both locks and jobs concurrenctly by @spencerjibz 923ecd3
use length in purge_expired: inMemory by @spencerjibz f13468e
use a better and faster mutex by @spencerjibz (#22 )
remove job from events, update examples and tests to reflct this by @spencerjibz 0e09ab3
minor bug fixes(panics) too by @spencerjibz d34e460
update tests by @spencerjibz (#23 )
clippy fix in examples by @spencerjibz 6872fb5
minor changes by @spencerjibz (#24 )
only use async version of xutex::Mutex by @spencerjibz ad0c370
use semaphore to limit concurrency by @spencerjibz (#25 )
update queue-metrics in rocksdb-store by @spencerjibz (#33 )
clean up main loop by @spencerjibz 7a98d6f
wip by @spencerjibz 6d761d9
clippy fix by @spencerjibz 2e420b5
reduce verbosity of logging and refactor smt by @spencerjibz 85e4b72
more logging by @spencerjibz aab3733
uncomment tokio-console directives by @spencerjibz f60217f
store joinhandle of running tasks instead of their ids only by @spencerjibz 8545cd5
better debug formating for TimerType::PromoteJob by @spencerjibz 36f672a
PR template by @spencerjibz (#36 )
main_loop fix: only proceed processing if a permit is available by @spencerjibz (#37 )
make Timer public by @spencerjibz a08e0a9
make a worker_opts a copy type by @spencerjibz 98a2c35
clippy fix by @spencerjibz d2bbefe
clean & better usage of atomic ordering by @spencerjibz 989189c
guard data in timed_maps with a parking_lot primitives by @spencerjibz 83d0178
use parking_lot mutex instead rwlock for in-memory-store by @spencerjibz (#38 )
maintain Acquire-Release ordering by @spencerjibz 68a8518
exclude library tests from benchmarks by @spencerjibz 77b695e
use the latest stable for benchmarks by @spencerjibz ebe2dbc
wip by @spencerjibz 7552a70
exclude library tests in benchmarks && fix caching by @spencerjibz faed406
catch panics from sync_processors by @spencerjibz a200802
expose worker-metrics methods from store by @spencerjibz 31e60e1
collect metrics from timers by @spencerjibz 4449e36
minor metrics updates by @spencerjibz 9c209a9
a clippy fix by @spencerjibz (#41 )
make all fields in WorkerMetrics pub by @spencerjibz be3a0a1
make worker_metrics clonable and comparable by @spencerjibz b4653a6
export our timedmap implementation to the library users by @spencerjibz 2d843cd
make the insert_expirable method sync by @spencerjibz b76a039
don’t hold on to mutex for long by @spencerjibz a0c8b10
strictly use async-mutex event in sync code by @spencerjibz 55654f6
collective metrics for the user’s processor function instead of the worker task by @spencerjibz 5dc9271
Fix task metrics collection: instrument spawned future, fix TTL bug, fallback task_id, add test by @Copilot 910ca34
comments clean up by @spencerjibz 80d6f37
clippy clean-up by @spencerjibz (#43 )
library-rename by @spencerjibz 6b573d6
library name correction in readme by @spencerjibz 585e1b2
replace crossbeam-sub-crates(queue& utils) by @spencerjibz 48af5ca
replace usage of atomic_ref_cell by @spencerjibz eabf62b
clippy fixes by @spencerjibz (#47 )
fix clippy warnings by @spencerjibz (#46 )
Complete documentation coverage: all public items documented, no missing-docs gaps by @Copilot 55274f2
Center lib.rs header with div align=center and downshift all heading levels by one by @Copilot a8731ce
Add logo image to lib.rs centered header block by @Copilot 11c134d
Link RedisStore in top description alongside InMemoryStore by @Copilot bad5147
Move Tokio runtime requirements section before Installation in lib.rs docs by @Copilot 1ec88cd
fix updating_metrics bug and add panics section to README by @Copilot 11063e6

Contributors

@Copilot
@blacksmith-sh[bot]
@spencerjibz

Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Running the suite

Benchmarks use Criterion :

cargo bench

Criterion writes an HTML report with plots and regression analysis:

target/criterion/report/index.html

What is measured

The queue_bench harness covers the hot paths:

Benchmark	What it exercises
`queue_bulk_add`	Enqueue throughput via `bulk_add` at several batch sizes
`queue_end_to_end_throughput`	Enqueue → reserve → process → settle, with a worker running
`queue_single_job_latency`	Time for one job to travel from `add_job` to `Completed`

Run one group at a time by name:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Running the suite

Use cargo nextest :

cargo nextest run

Warning

Use cargo nextest run, not cargo test. The integration tests share process-global state (the metrics collector), so they must each run in their own process — which nextest does by default. Under cargo test (a single process) they interfere with one another and fail. CI runs nextest; local development should too.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

KioMQ is MIT licensed and developed in the open. Issues, discussion, and pull requests are all welcome.

Getting set up

git clone https://github.com/KioHQ/kiomq
cd kiomq
git config core.hooksPath .githooks # enable the repo's git hooks
cargo install cargo-nextest # the test runner CI uses

The pre-push hook runs the same checks as CI, so failures surface before you open a pull request:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --features tracing,redis-store -- -D warnings
cargo nextest run --locked --features tracing,redis-store
cargo test --locked --doc --features tracing,redis-store # nextest skips doctests
cargo doc --locked --workspace --no-deps --features tracing,redis-store

Note that clippy runs with -D warnings — a lint is a build failure. Run it before you push and save yourself a round trip.