Jupyter (FKA iPython) project gets $6m funding

The Helmsley Charitable Trust, the Alfred P. Sloan Foundation, and the Gordon and Betty Moore Foundation just announced a $6m grant to UC Berkeley and Cal Poly to fund Project Jupyter. Jupyter evolved from the iPython project, abstracting the language-agnostic parts. It also serves as an interactive shell for Python 3, Julia, R, Haskell, and Ruby. I think the most notable thing it provides is a web-based GUI “notebook” similar to what has been available in Maple and Mathematica for a while. (Maybe Matlab, too: I have not used Matlab much.)

Correction: Jupyter serves as an interactive shell for a lot more than what I listed. Here is the full list.


Leap second added

The leap second has been added, and my systems seem to not have barfed. In particular, I was wary of what the NFS and Lustre file servers and clients would do.

Jun 30 20:00:00 myserver ntpd[5973]: 061b 0b leap_event


Ganglia module (kludge) to monitor temperature via IPMI

Since I don't have environmental monitoring in my server room, I used ipmitool to read my cluster nodes' on-board sensors to sort of get at the cold aisle ambient temperature. One should be able to see a list of available sensor readings with "ipmitool sdr" or "ipmitool sensor", the latter giving other associated parameters for the metrics, like alarm thresholds.

Since access to /dev/ipmi0 is restricted to root, my kludge was to create a root cron job which runs every N minutes, writing the appropriate value to a log file:

ipmitool -c sdr get "Inlet Temp" | cut -f2 -d, > $LOGFILE

Then, the Ganglia python module reads that file. I followed the example.py module which is distributed with Ganglia, and the associated example.pyconf.

The code is at my github repo.


Logjam: another day, another https vulnerability

A vulnerability has just been discovered in https, specifically in the Diffie-Hellman key exchange. This arose from the old export restrictions set by the US, so that its law enforcement and security agencies could break encryption used by foreign entities. Ars Technica, as usual, has a good write-up.

The researchers who discovered the flaw have a dedicated website which gives pointers on what to do if you run a web server, or just a browser. They have a server scanner, or you can use the one at Qualys SSL Labs.

Ivan Ristić has some more detail on increasing the strength of DH on Apache. Unfortunately, it may not be supported by the version of Apache you happen to have running.

Ubuntu swap partitions across upgrades

I recently upgraded from 14.10 Utopic to 15.04 Vivid, and only just realized that the swap partition was not mounted. Had to reformat it, and update /etc/fstab since the UUID had changed, too.


pylab confusions

There are three pylabs that one may encounter in using Python. Two have been around for a while, and the third just showed up less than a month ago.

The “real” pylab is the procedural interface to matplotlib, i.e. a MATLAB-like command line interface. It imports matplotlib.pyplot and numpy into a single namespace. You can use it from ipython’s prompt by calling the magic function “%pylab”. It is no longer recommended by the matplotlib people. The recommended way is to import with abbreviated namespace names, and use the qualified functions. For example:

import matplotlib.pyplot as pltimport numpy as np
x = np.linspace(0, 2, 100)
plt.plot(x, x, label='linear')plt.plot(x, x**2, label='quadratic')plt.plot(x, x**3, label='cubic')
plt.xlabel('x label')plt.ylabel('y label')
plt.title("Simple Plot")
Then there is the idea/proposal by Keir Mierle to improve on the pylab idea of a single package one might use to utilize Python for interactive analysis. This is written up in the SciPy wiki, but does not seem to have been updated since 2012.

And finally, if you are like me, and have not been thinking too hard, and typed “pip install pylab” you get this new package from PyPI, first added on 2015-04-23. It does nothing but pull in several other Python packages, i.e. it serves as a metapackage. You can see the source is basically a dummy, with all the action in the requirements defined in setup.py.


Ganglia procstat.py fix to handle process names containing underscores

UPDATE: Accepted. Current version here.

This has bugged me for a while: Ganglia's Python module procstat.py which monitors process CPU and memory usage did not show any data for Grid Engine's qmaster, which has a process name of "sge_qmaster". Turns out, this is because it tries to parse out the process name by assuming it does not have underscores in it. This snippet is from the get_stat(name) function in procstat.py:

if name.startswith('procstat_'):
    fir = name.find('_')
    sec = name.find('_', fir + 1)
    proc = name[fir + 1:sec]
    label = name[sec + 1:]
I just submitted a pull request to change this to something which handles process names with some number of underscores. The snippet to replace the above:

if name.startswith('procstat_'):
    nsp = name.split('_')
    proc = '_'.join(nsp[1:-1])
    label = nsp[-1