Scripting Bright Cluster Manager 9.0 with Python

It has been more than 6 years since the previous post about using the Python API to script Bright Cluster Manager (CM). Time for an update.

I have to do the same as before: change the “category” on a whole bunch of nodes.

NB the Developer Manual has some typos, where it makes it look like you can specify categories as strings of their names, e.g. cluster.get_by_type('Node')

30k-core cluster built on Amazon EC2 cloud

Ars Technica reports on a company called Cycle Computing that has built a 30k-core cluster on Amazon EC2 cloud for a client. It costs $1279 per hour to run, including fees to Amazon and Cycle Computing.

The cluster ran for about 7 hours, doing a computation for a pharmaceutical company. It had 3809 compute instances, each with 7GB RAM making 30472 cores, 26.7 TB RAM, and 2PB of disk space.

In the past, Amazon has built a virtual cluster that got onto the TOP500 list.

Pretty amazing.

How to be a lazy Linux cluster admin

Great little article for sysadmins managing Linux clusters.

Cluster means different things to different people. In the context of this article, cluster is best defined as scale-out -- scale-out clusters generally have a lot of the same type of components like Web farms, render farms, and high performance computing (HPC) systems. Administrators will tell you that with scale-out clusters any change, no matter how small, must be repeated up to hundreds of thousands of times; the laziest of admins have mastered techniques of scale-out management so that regardless of the number of nodes, the effort is the same. In this article, the authors peer into the minds of the laziest Linux® admins on Earth and divulge their secrets.

For me, the most fun part of the job is in Tip 3 -- Automate Everything. Coding is fun, especially if the result is not having to type so much. Similarly, coding up APIs to access your monitoring data gives you programmatic access to your data which will simplify analysis.