Ganglia

Word to the wise: do not enable the multiplecpu multicpu module. It doesn't get disabled even if you append ".disabled" to the file name. Now, I have 265 CPU metrics.

Limiting logins under SSSD

Under SSSD, you can pretty easily limit logins to specific users or groups. The syntax is different from that of /etc/security/access.conf, and is actually easier. Red Hat has some documentation (may require login). There is also a man page for sssd.conf(5).

Under the your domain, add some lines to configure "simple" access control:

[domain/default]  

access_provider = simple 

simple_allow_users = topbanana 

simple_allow_groups = bunchofbananas,wheel

Using the NVIDIA Python plugin for Ganglia monitoring under Bright Cluster Manager

The github repo for Ganglia gmond Python plugins contains a plugin for monitoring NVIDIA GPUs. This presumes that the NVIDIA Deployment Kit, which contains the NVML (management library), is installed via the normal means into the usual places. If you are using Bright Cluster Manager, you would have used Bright's cuda60/tdk to do the installation. That means that the libnvidia-ml.so library is not in one of the standard library directories. To fix it, just modify the /etc/init.d/gmond init script. Near the top, modify the LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/cm/local/apps/cuda/libs/current/lib64

The modifications to Ganglia Web, however, are out of date. I will make another post once I figure out how to do modify Ganglia Web to display the NVIDIA metrics.

UPDATE: Well, turns out there seems to be no need to modify the Ganglia Web installation. Under the host view, there is a tab for "gpu metrics" which shows 22 available metrics.

root cron jobs and /etc/security/access.conf

On RHEL6, if your root cron jobs do not run, check your /var/log/secure file for lines that look like:

crontab: pam_access(crond:account): access denied for user `root' from `cron'

You may also see the following message when, as root, you type "crontab -e":

Permission deniedYou (root) are not allowed to access to (crontab) because of pam configuration.

If there are any like that, check /etc/security/access.conf -- you need to allow root access via cron and crond by adding the following line:

+ : root : cron crond 

Keepass, Google Drive, and multi-OS access

UPDATE 2014-06-11: The kdbx file seems to work fine on Dropbox. Luckily, I had a backup there.

If you store your Keepass DB file on Google Drive plus InSync in order to access it via Linux, Windows, and Android, you might want to stop doing so. My Keepass file seems to be irretrievably corrupted.

More on SSSD - getting finger(1) and command completion of ~ to work

I noticed after getting SSSD up and running that the finger(1) command no longer worked, and neither did command completion of ~username. In the first case, finger(1) never found any users, no matter if I used the exact username. In the second case, if I typed at the command line cd ~d<tab>, it would not expand to a list of possibilities. However, cd ~david worked just fine.

Turns out, there needs to be one setting in /etc/sssd/sssd.conf:

[domain/default]
    ...
    enumerate = True

That allows a local precache to be created so that finger(1) can iterate over user info to find a matching record.

Then, restart the sssd service.

Useful links:

• Red Hat Bug 709178 - "groups user" and "finger gecos" fails
• Princeton Institute for Advanced Studies Linux wiki - Princeton LDAP Authentication

SSSD setup in a Bright Cluster

UPDATE 2015-05-28: Also, on the master node, the User Portal web application needs a setting in /etc/openldap/ldap.conf:

    TLS_REQCERT      never


At my current job, we use Bright Cluster Manager and Univa Grid Engine on RHEL 6.5. We were seeing issues where submitted jobs ended up in an "Error" state, especially if many jobs were submitted in a short period, either an array or a shell script loop running qsub iteratively. The error reason was:

    can't get password entry for user "juser". Either the user does not exist or NIS error!

However, logging into the assigned compute node and running "id" or even some C code to do user lookups passed.

By default, our installation used nslcd for LDAP lookups. Univa suggested switching to SSSD (System Security Services Daemon) as Red Hat had phased out nslcd. The Fedora site has a good overview.

The switch to using SSSD turned out to be fairly easy, with some hidden hiccups. Running authconfig-tui and keeping the existing settings, and then hitting "OK" immediately turned off nslcd and started up sssd, instead. All the attendant changes were made, too: chkconfig settings, /etc/nsswitch.conf. However, I found that users could not change passwords on the login nodes. They could login, but the passwd command failed with "system offline".

Turns out, SSSD requires an encrypted connection to the LDAP server for password changes. This is a security requirement so that the new password is not sent in the clear from the client node to the LDAP server. (See this forum post by sgallagh.) This means an SSL certificate needs to be created. Self-signed will work if the following line is added to /etc/sssd/sssd.conf:

     [domain/default]
     ...
     ldap_tls_reqcert = never

To create the self-signed cert:
     root # cd /etc/pki/tls/certs
     certs # make slapd.pem
     certs # chown ldap:ldap slapd.pem

Then, edit /cm/local/apps/openldap/etc/slapd.conf to add the following lines:
    TLSCACertificateFile /etc/pki/tls/certs/ca-bundle.crt
    TLSCertificateFile /etc/pki/tls/certs/slapd.pem
    TLSCertificateKeyFile /etc/pki/tls/certs/slapd.pem

Also, make sure there is a section - my config did not have access to shadowLastChange:

    access to attrs=loginShell,shadowLastChange

        by group.exact="cn=rogroup,dc=cm,dc=cluster" read

        by self write

        by * read

Then, restart the ldap service.

UPDATE Adding some links to official Red Hat documentation: https://access.redhat.com/solutions/42746