========================================================================
                          steve, the jobserver
========================================================================

steve is a GNU make-compatible system-wide jobserver implemented using
FUSE.  It enables running multiple build jobs simultaneously while
sharing a single job pool, and therefore avoiding superlinear job count
multiplication.  Note that while GNU make uses a named pipe, steve uses
a character device to track acquisition of job tokens and reclaim them
if processes are killed without returning them.


Features
--------

- Per-process token accounting.  Steve notes how many tokens each
  process acquired, and reclaims them if the process exits without
  returning them.

- Poor man's round-robin token balancing.  While steve cannot perfectly
  balance jobs between multiple tasks, it does attempt to split them
  between multiple processes running simultaneously and prevent any
  single program from acquiring all tokens immediately.

- Support for jobserver-level load-based scheduling with configurable
  minimum of jobs to sustain when load is exceeded, and per-process job
  limiting.

- Support for adjusting parameters at runtime via ioctls (CLI provided
  by stevie).


Installation
------------

Steve uses the Meson_ build system.  Typically, it can be built
and installed using the following commands::

    $ meson setup . build
    $ meson install -C build

For more details, please see the Meson documentation.

Steve needs access to ``/dev/cuse`` which is normally restricted
to the superuser.  However, as a best security practice it is
recommended to run steve using a dedicated user account.  This can be
achieved either by:

1. using ``--user {username}`` option when starting steve, to make it
   drop privileges after opening ``/dev/cuse``, or

2. giving the user running steve permissions to open ``/dev/cuse``.

The default device file created by CUSE (``/dev/steve``) is accessible
only to root.  The recommended way of changing permissions to these
files is through udev rules.  That said, world access may be
undesirable, as malicious client could easily consume all the tokens
and block other processes from obtaining any.

The distribution provides a number of example files that can be modified
and installed:

``data/steve.service``
  A systemd service file.  It assumes that steve is installed as
  ``/usr/bin/steve`` and runs as ``steve:steve`` user which can access
  ``/dev/cuse``.

``data/steve.initd`` and ``data/steve.initd``
  A init.d script and conf.d file for OpenRC, with the same assumptions.

``data/steve.udev``
  Udev rules file (needs to be renamed to ``*.rules``) that gives
  the ``jobserver`` group access to ``/dev/steve``.

``data/sandbox.conf``
  Gentoo sandbox configuration file that gives sandboxed processes
  access to ``/dev/steve``.

.. _Meson: https://mesonbuild.com/


Usage
-----
When started without any arguments, steve claims ``/dev/steve``
and starts serving job tokens::

    $ steve
    steve running on /dev/steve for 12 jobs
      at least 1 jobs will be always available

By default, steve uses the logical CPU count.  The ``-j`` option can
be used to override the job count::

    $ steve -j6
    steve running on /dev/steve for 6 jobs
      at least 1 jobs will be always available

For more options, see ``steve --help``.

To use steve from a build system, one needs to set ``MAKEFLAGS``
appropriately.  To use it for all builds in Portage, these flags can
be set in ``make.conf``::

    MAKEFLAGS="--jobserver-auth=fifo:/dev/steve"

Note that to use steve with Ninja, ``-j`` must not be passed.  Given
that ``MAKEOPTS`` is used to provide defaults for other build systems,
the best approach is to override ``NINJAOPTS``, e.g.::

    MAKEOPTS="-j12 -l13"
    NINJAOPTS="-l13"

To inspect the current status, ``SIGUSR1`` can be sent to steve::

    $ killall -USR1 steve

This will cause the process to print the current token count, as well
as the list of processes holding them::

    steve: currently 0 tokens available out of 12
    PID 221613 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 3 tokens:
      job 0x00 running for 0.0084036s
      job 0x01 running for 0.0536742s
      job 0x02 running for 0.163137s
    PID 202894 (ninja -v -l16 distribution) holds 7 tokens:
      job 0x04 running for 3.0512s
      job 0x00 running for 3.05184s
      job 0x09 running for 8.47799s
      job 0x08 running for 18.8555s
      job 0x01 running for 20.4352s
      job 0x03 running for 20.7428s
      job 0x02 running for 23.3102s
    PID 194264 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 0 tokens:
    PID 194247 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 0 tokens:
    PID 194237 (make LDFLAGS=-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu LIBPATH=/usr/lib/gcc/x86_64-pc-linux-gnu/15 STAGE1_CFLAGS=-m64 -march) holds 0 tokens:
    PID 188922 (/usr/bin/python3.14 /usr/lib/python-exec/python3.14/emerge -vB --nodep llvm-core/clang sys-devel/gcc --jobs) holds 2 tokens:
      job 0x01 running for 108.73s
      job 0x00 running for 108.735s


stevie
------
stevie is the client for steve.  It currently provides two functions:

1. Running the specified command with a job token acquired
   and appropriate ``MAKEFLAGS``, e.g.::

       stevie make foo

2. Controlling steve remotely, e.g.::

       stevie --get-jobs --set-jobs 6

For the full list of getters and setters, see ``stevie --help``.


Use in containers
-----------------
steve can be used across container boundaries, provided that
the container system permits sharing character devices.
For systemd-nspawn, the device can be bound to the container using
the following configuration::

    [Files]
    Bind=/dev/steve
    BindReadOnly=/etc/sandbox.d/90steve

Additionally, when running the container as a service, it is necessary
to permit the service to access it, e.g. via::

    $ systemctl edit systemd-nspawn@${container_name}.service

Then adding::

    [Service]
    DeviceAllow=/dev/steve rw


Docker invocation
-----------------
To run a container and make the device available inside the container,
add the device mapping when invoking Docker. For example::

    $ docker run ... --device /dev/steve:/dev/steve:rw <image> ...


Compatibility notes
-------------------
Steve implements the `POSIX jobserver protocol`_ based on named pipes.
Furthermore, due to use of CUSE it is limited to Linux.  Support for
other platforms is not planned at the moment.

Due to technical limitations of the Linux kernel, steve cannot use
an actual named pipe; it uses a character device instead.  This may
cause issues with strict jobserver clients that verify that
``--jobserver-auth`` specifies an actual named pipe.  However, upstreams
can generally be convinced to allow character devices as well.

Per-process accounting implies that steve requires that the same process
acquired and returns job tokens.  If a process "passes" a job token to
a different process and exits, steve reclaims the token immediately.
Attempting to return a token not acquired by the process returning it
results in a ``ENOSPC`` error being returned to the client.

To work around bugs in existing software, steve permits a single job
token to be "returned" by a new client prior to acquiring any clients.
However, this behavior is not guaranteed to be preserved in the future,
and new clients must not rely on it.

.. _POSIX jobserver protocol:
   https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html
