We do a lot of upgrades in the summer. This year we’re migrating from the Torque scheduler (which is no longer open-source) to the Slurm scheduler (which is). It’s a good learning experience in addition to being a systems improvement.
First I installed it on a cluster, successful. That took a while: It turns out schedulers have complicated installation processes with specific dependency chains. To save time in the future, I decided that I would attempt to automate the installation.
This has gone better than you might initially guess.
I threw my command history into a script, spun up a VM, and began iterating. After a bit of work, I’ve made the installation script work consistently with almost no direct user input.
Then I tried running it on another machine and ran headfirst into SELinux.
The problem
The installation itself went fine, but the OS displayed this message every time I tried to enable the Slurm control daemon:
[root@host system]# systemctl enable slurmctld.service Failed to enable unit: Unit file slurmctld.service does not exist.
I double- and triple-checked that my file was in a directory that systemctl expected. After that I checked /var/log/messages and saw a bunch of errors like this …
type=AVC msg=audit(1589561958.124:5788): avc: denied { read } for pid=1 comm="systemd" name="slurmctld.service" dev="dm-0" ino=34756852 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0
… and this:
type=USER_END msg=audit(1589562370.317:5841): pid=3893 uid=0 auid=1000 ses=29 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct="slurm" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'UID="root" AUID="[omitted]"
Then I ran ls -Z on the service file’s directory to check its SELinux context:
-rw-r--r--. 1 root root unconfined_u:object_r:admin_home_t:s0 367 May 15 13:01 slurmctld.service [...] -rw-r--r--. 1 root root system_u:object_r:systemd_unit_file_t:s0 337 May 11 2019 smartd.service
Notice that the smartd file has a different context (system_u...) than does the slurmctld file (unconfined_u...). My inference was that the slurmctld file’s context was a (not-trusted) default, and that the solution was to make its context consistent with the context of the working systemctl unit files.
The solution
Here’s how to give the service file a new context in SELinux:
chcon system_u:object_r:systemd_unit_file_t:s0 slurmctld.service
To see the appropriate security context, check ls -Z. Trust that more than my command, because your context may not match mine.
Concluding remarks
I am early-career and have done very little work with SELinux, so this is not a specialty of mine right now. As such, this may or may not be the best solution. But, mindful of some security advice, I think it is preferable to disabling SELinux altogether.