RFC: OpenRC initscript

[automorphism88]

I'm using Gentoo, and since bees comes with a systemd service but I'm using OpenRC, I decided to write my own initscript. It is based somewhat on the beesd script (which I couldn't just use directly since it wouldn't work with start-stop-daemon), and designed to work with a conf.d file that's similar to the one that's installed in /etc/bees, but with some additional options added to control logging and {io,}nice level. It also avoids bashisms so that it can be run with dash as /bin/sh.

Is this something you would be interested in merging upstream? My /etc/init.d file can be found here: https://github.com/automorphism88/gentoo-overlay/blob/master/sys-fs/bees/files/bees.initd and the corresponding template /etc/conf.d file can be found here: https://github.com/automorphism88/gentoo-overlay/blob/master/sys-fs/bees/files/bees.confd. Note that while this initscript works for me, it should be considered experimental. I haven't tried to get this merged into the official Gentoo ebuild yet since I figured it would be even better to upstream the script directly into bees.

A few things I noticed about the beesd script when I was using it as a reference were:

1. Is there some reason to chmod 700 the database file instead of chmod 600? Executable permission doesn't seem to be necessary.
2. stat -c %s FILE is a simpler, faster, and more portable way of checking file size than the method used in the beesd script (which, I believe, relies on a GNU extension in sed to interpret \t as tab). Incidentally, stat -c %i can also be used to check whether a directory is a btrfs subvolume, since a btrfs subvolume will have an inode number of 256.
3. Why not mount the root subvolume with noatime? Unlike btrfs-specific mount options like compression, this one can be set per-subvolume, so it will default to relatime even if the rest of the filesystem is mounted with noatime. Probably wouldn't make much difference in this case, I'm just in the habit of using noatime everywhere unless there's a specific need to do otherwise.

[Zygo]

+1 to all of the above. I don't really write the distro integration scripts, I just pull them. Usually when I try to edit them, I break them.

1. executable isn't necessary, 0600 is fine.

2. stat -c %i $PATH can't figure out the subvolume ID, but btrfs ins rootid $PATH can. stat can also report the fstype for the path. bees needs a directory with inode 256, root/subvol id 5, and fstype btrfs.

3. Sounds like an excellent idea. bees will examine every file and touch some of them. The files that are modified are doomed to have their inodes updated anyway, but the ones that are merely read shouldn't be hit with unexpected inode updates.

[automorphism88]

2. `stat -c %i $PATH` can't figure out the subvolume ID, but `btrfs ins rootid $PATH` can.  `stat` can also report the fstype for the path.  bees needs a directory with inode 256, root/subvol id 5, and fstype btrfs.

I'm referring to this line in beesd where btrfs sub show is tried to test whether $BEESHOME is a subvolume or ordinary directory.

3. Sounds like an excellent idea.  bees will examine every file and touch some of them.  The files that are modified are doomed to have their inodes updated anyway, but the ones that are merely read shouldn't be hit with unexpected inode updates.

I'm referring to the invocation of mount in beesd here which implicitly uses relatime.

[Zygo]

You might also want to look at #104 which aims to make the script configuration a lot more stateless.

[automorphism88]

I don't really write the distro integration scripts, I just pull them. Usually when I try to edit them, I break them.

I wouldn't consider this a distro integration script any more than the systemd service file is. OpenRC is used on other distros as well as Gentoo, and I don't think anything in my script is Gentoo-specific.

The issue of UUIDs in config files is at least partially solved by OpenRC multiplexing, since you can have /etc/conf.d/bees contain global options which are shared across machines, and /etc/conf.d/bees.foo containing a specific UUID. Then if you symlink /etc/init.d/bees to /etc/init.d/bees.foo, the bees.foo service will first read the /etc/init.d/bees config file, and then override it with any specific options set in /etc/init.d/bees.foo.

I could make this into a PR, but I'm not sure how to integrate it into the build system in terms of things like install paths and enabling/disabling installation. The /etc/init.d and /etc/conf.d paths are not required, for instance Arch uses /etc/openrc/init.d and /etc/openrc/conf.d, but OpenRC does assume that init.d and conf.d folders are in the same directory.

You might also want to look at #104 which aims to make the script configuration a lot more stateless.

That seems like a change which could be done independently. My script doesn't use /etc/bees or the beesd wrapper script. I could look at also adding support for specifying filesystems by path instead of UUID, though, the way that script does, which appears to be the only advantage it has over the present script in the context of an OpenRC service.

One thing that might be useful is if it were possible to just run beesd with start-stop-daemon instead of having to duplicate that code so that bees can be run directly. I think all that might be required for that would be for the beesd script to run bees with exec so that it has the same PID instead of forking. However, I assumed beesd doesn't do that so that it can use a trap to clean up on exit. That's something that I implemented separately (also making it keep track of whether or not the subvolume was already mounted at start, so that it doesn't unmount it on stopping unless it mounted it on starting).

[kakra]

I'm the maintainer of the Gentoo ebuild. @Zygo if you don't mind I come up with a suggestion, maybe also fixing the other problems mentioned here.

[automorphism88]

I'm the maintainer of the Gentoo ebuild. @Zygo if you don't mind I come up with a suggestion, maybe also fixing the other problems mentioned here.

I'm the one who recently opened a bunch of bugs on the gentoo bugzilla about bees - thanks for fixing them! I considered submitting this there as well, but figured it'd be better to do it upstream.

By the way, @kakra, if it's not too off-topic, I noticed your fix of the version UNKNOWN issue set the version to "v${PV}" so that the help output displays "version v0.6.1" which seems redundant (version version 0.6.1?). Why not just make it "${PV}"? See as an example the output of git --version.

[kakra]

That's to match upstream. Bees itself uses the Git tag which is prefixed by "v".

[nagelp]

I'm the maintainer of the Gentoo ebuild. @Zygo if you don't mind I come up with a suggestion, maybe also fixing the other problems mentioned here.

@kakra, any news on this? I just installed sys-fs/bees on Gentoo, but there is no init.d script. @automorphism88's links at the top of this issue don't work anymore.

[MCPO-Spartan-117]

I'm guessing this init script got lost then?

[kakra]

I'm going to update the ebuild soon, let me move this up in my todo list then.

@MCPO-Spartan-117 Thanks for pinging...

[zachron]

anything further with this?

[Forza-tng]

Hi @kakra ,

Just a ping to see if we can get the OpenRC scripts included in bees. This would help Gentoo and Alpine based distros a lot :)

https://bugs.gentoo.org/890961

OpenRC init scripts

conf.d/bees

########
# bees configuration file
#
# Documentation is available at:
# https://zygo.github.io/bees/config.html
#
# Important gotchas and kernel bugs
# https://zygo.github.io/bees/btrfs-kernel.md
########
# Multiple instances of bees
#
# It is possible to run multiple instances of
# bees by naming the service file to
# bees.<name>.
#
#  bees.foo
# * cp /etc/conf.d/bees /etc/conf.d/bees.foo
# * ln -s /etc/init.d/bees /etc/init.d/bees.foo
# * update 'fsuuid' and 'hashsize' in /etc/conf.d/bees.foo
# * rc-service bees.foo start
#
#  bees.bar
# * cp /etc/conf.d/bees /etc/conf.d/bees.bar
# * ln -s /etc/init.d/bees /etc/init.d/bees.bar
# * update 'fsuuid' and 'hashsize' in /etc/conf.d/bees.bar
# * rc-service bees.bar start
########

# Extra bees command line arguments. Make sure
# you read the documentation before changing these.
# https://zygo.github.io/bees/options.html
bees_args="--scan-mode 4 --strip-paths --thread-count 8"

# Bees works on a filesystem as a whole. The
# filesystem's unique UUID is used as
# identifier so the correct filesystem is mounted.
# Use `btrfs filesystem show` to display known
# btrfs filesystems and set the correct uuid.
fsuuid="c08bb98b-3b98-4dbb-a7c0-5540c2af781b"

# The size of bees hash table in bytes. A large table
# increases deduplication rate, but uses more memory.
# You are encouraged to choose a size that fits your
# needs, and not rely on defaults.
# See the sizing chart below for recommended sizes.
hashsize="$((512*1024*1024))" # 512MiB

# Set bees loglevel.
# 1 is lowest, 8 is most verbose.
# WARNING: a high loglevel increases logfile size greatly.
loglevel="6"

# Location of the bees status file
#statusfile="/run/bees/${fsuuid}.status"

# Custom logfile location
#logfile="/var/log/bees/${RC_SVCNAME}.log"

# CPU scheduler controls how Linux schedules CPU
# resources betweed processes.
# "other" for default scheduler
# "batch" for Batch scheduler
# "idle"  for Idle scheduler 
#
# It is recommended to use batch or idle combined
# with SSD_NICELEVEL=19
scheduler="batch"

# ionice values are  0 none, 1 real-time,
# 2 best effort and 3 idle.
#SSD_IONICELEVEL=3

# nice can be -20 to 19, where 19 is lowest priority.
SSD_NICELEVEL=19

# Out Of Memory score adjustment. A high value increases
# the likelihood that the process will be terminated
# during OOM condition. Allowed values -1000 to 1000.
SSD_OOM_SCORE_ADJ=500


########
# Hash table sizing
#
# The relative size of the hash table compared
# to the data set determines the achievable
# deduplication extent size.
# 
# NOTE: The whole hash table will be loaded into RAM.
# 
# A = unique data size
# B = hash table size
# C = average dedupe extent size
#   A  |   B   |   C
# ---------------------
#  1TB |   4GB |    4K
#  1TB |   1GB |   16K
#  1TB | 256MB |   64K
#  1TB | 128MB |  128K <- recommended
#  1TB |  16MB | 1024K
# 64TB |   1GB | 1024K
# The hashsize must be a multiple of 128KiB.
#
# A good starting point is 128KiB average extent size,
# which would mean the following hash sizes for different
# sized data sets. This is just an estimate as many factors
# affect what achievable extent size is possible.
#
# Data
# size   | hashsize
# --------------------------------------
# 256GiB | $((32*1024*1024))"   # 32MiB
# 512GiB | $((64*1024*1024))"   # 64MiB
# 1TiB   | $((128*1024*1024))"  # 128MiB
# 2TiB   | $((256*1024*1024))"  # 256MiB
# 4TiB   | $((512*1024*1024))"  # 512MiB
# 8TiB   | $((1024*1024*1024))" # 1GiB
# 16TiB  | $((2048*1024*1024))" # 2GiB
########

rc_cgroup_settings="
cpu.idle 1
io.weight 1
io.bfq.weight 1
memory.min 600M
"

init.d/bees

#!/sbin/openrc-run

## bees variables
rundir="/run/bees"
logdir="/var/log/bees"
mnt="$rundir/mnt/${fsuuid}"
device="/dev/disk/by-uuid/${fsuuid}"
: "${beeshome:="${mnt}/.beeshome"}"
hashfile="${beeshome}/beeshash.dat"

: "${logfile:="${logdir}/${RC_SVCNAME}.log"}"
: "${loglevel:=5}"
: "${bees_args:="-m 3"}"
: "${scheduler:="idle"}"
: "${SSD_IONICELEVEL:=3}"
: "${SSD_NICELEVEL:=19}"
export BEESSTATUS="${statusfile:-${rundir}/${fsuuid}.status}"

LD_PRELOAD="/usr/lib64/libjemalloc.so"

## OpenRC
name="${RC_SVCNAME}"
description="Best-Effort Extent-Same, a btrfs dedupe agent"
command="/usr/libexec/bees"
command_args="-v ${loglevel} ${bees_args} \"${mnt}\" "
command_background="true"
command_progress="true"
extra_started_commands="suspend resume"
description_suspend="Suspends bees without unloading it."
description_resume="Resumes normal operation."

start_stop_daemon_args="--scheduler ${scheduler} --stdout ${logfile} --stderr ${logfile}"
pidfile="/run/${RC_SVCNAME}.pid"
retry="TERM/${GRACEFUL_TIMEOUT:-120}/KILL/1"

depend() {
	use lvm device-mapper localmount
}

checkconfig() {
	if [ -z $hashfile ]; then
		eerror "hashsize not set!"
		return 1
	fi
	if [ ! -b "${device}" ]; then
		eerror "${device} not found!"
		return 1
	fi
	if [ ! "$(blkid -s TYPE -o value "${device}")" == "btrfs" ]; then
		eerror "${device} is not a btrfs filesystem"
		return 1
	fi;
	if [ ! -z "${logfile}" ]; then
		checkpath -f -m 0600 -o root:root "${logfile}"
	fi
}

start_pre() {
	ebegin "Config check"
	checkconfig || return 1

	if  findmnt "${mnt}" >/dev/null ; then
		eerror "${mnt} is already mounted. Aborting."
		return 1
	fi
	checkpath -d -m 0700 -o root:root "${logdir}"
	checkpath -d -m 0700 -o root:root "${rundir}"
	checkpath -d -m 0700 -o root:root "${rundir}/mnt"
	checkpath -d -m 0700 -o root:root "${mnt}"

	ebegin "Preparing ${fsuuid} mountpoint"
	mount -osubvolid=5 "${device}" "${mnt}" || return 1

 	if [ ! -d "${beeshome}" ]; then
		btrfs subvolume create "${beeshome}" || return 1
		checkpath -d -m 0700 -o root:root "${beeshome}"
	else
		# btrfs subvolume show "${beeshome}" > /dev/null || return 1
		checkpath -d -m 0700 -o root:root "${beeshome}"
	fi

	checkpath -f -m 0600 -o root:root "${hashfile}"
	old_hashsize="$(du -b "${hashfile}" | cut -f1)"
	if [ "${old_hashsize}" == "0" ] ; then
		truncate -s "${hashsize}" "${hashfile}"
	elif [ "${hashsize}" != "${old_hashsize}" ] ; then
		eerror "Hash table size mismatch! Expected ${hashsize}b but found ${old_hashsize}b"
		return 1
	fi
	pushd "${mnt}" >/dev/null
	eend $?
}
stop_pre() {
	# make sure bees is not suspended before stopping.
	start-stop-daemon --signal SIGCONT --pidfile "${pidfile}"
	return $?
}

stop_post() {
	ebegin "Unmounting ${fsuuid}"
	umount "${mnt}"
	eend $?
}

suspend() {
	# Suspending bees is useful to avoid a 'btrfs send' bug
	# that exists with some kernels.
	# See https://zygo.github.io/bees/btrfs-kernel.md
	ebegin "Suspending ${RC_SVCNAME}"
	start-stop-daemon --signal SIGSTOP --pidfile "${pidfile}"
	eend $?
}

resume() {
	ebegin "Resuming ${RC_SVCNAME}"
	start-stop-daemon --signal SIGCONT --pidfile "${pidfile}"
	eend $?
}

[Zygo]

Have a look at topics/openrc, starting from 3f4fa17.

Beware the scripts have not been tested.

External configuration is long gone in my dev branch--the bees binary runs without a shell wrapper, and has its own INI-based config file now--but the config schema keeps changing as filtering, unreachable block removal, full-extent planning, and defrag get implemented, and none of them need quite the same config knobs I thought they did when I planned these features. My dev branch is now an unpublishable mess, so for the moment I've extracted the script-related changes onto a branch.

There were a few issues with the openrc script, particularly about BEESHOME handling and mount namespace setup. Integrating that revealed that beesd had similar issues. I've put fixes for both on the same branch due to the potential for behavior changes. See previous note about no testing.

I'm not too fond of having yet another island of bees documentation to maintain in openRC conf comments. Between comments embedded in the built-in config and the full user-level documentation of the config schema, we will have plenty of config docs for users to read. Everywhere else should point to /usr/share/doc/bees or similar. I've accepted the extra comments for now because bees will read its own config in the future, and only use the legacy scripts to interpret legacy user configurations.

Changes:

• I've lined up the config variable names for DB_SIZE between the two config schemas.
• systemd sets up private mount namespaces again
• fallback to unshare now uses shared propagation mode, not private, so when a user umounts something, bees doesn't keep it mounted in its namespace.
• BEESHOME is handled consistently and according to the real rules: no subvol required, mkdir to create on non-btrfs, relative paths remain relative so the security model works.
• Both systemd and openrc now call through beesd for consistent behavior wrt mounting and BEESHOME.
• beesd extended to support an explicitly named config file to make invocation from openrc possible. (They could have both called a common shell wrapper, but I'm trying to reduce the number of shell wrappers all the way to zero.)
• openat2 enforcement of relative BEESHOME security policy.
• Between 3f4fa17 and master there's a ton of other small bug fixes.

[kakra]

@Forza-tng If you could test it with Gentoo (I do not use openrc systems anymore), I can work on including it in the ebuild.