LIGO SCientific Collaboration

System Administrator
Humanity is on the edge of a new frontier in astronomy: gravitational wave astronomy. Gravitational waves allow us to observe further back into the universe's history than ever before. They allow us to observe the collisions of galaxies and learn about the big bang. On September 14, 2015, LIGO made history with the first direct observation of gravitational waves by an earth instrument.

These instruments, called interferometers, use lasers and mirrors to measure interference patterns modulated by gravitational waves. They produce a LOT of data.

All that data gets organized into channels, frames, and streams, then processed in parallel using custom software called LDAS.

Before the cloud, scientists used Beowulf clusters as supercomputers to analyze all that data.

To extend the cluster, we wrote a BOINC screen saver called Einstein@Home, pictured above.
My Contributions
Beowulf clusters are made from commodity off-the-shelf, standard desktop hardware with a few tweaks. In Medusa's case, the whole cluster was networked via one giant 300+ port meshed Ethernet switch. 

I was hired as part of a team to administer the hardware and write custom software- the precursor to modern DevOps tooling. Today's observability and configuration tools that we take for granted hadn't been invented yet, so we wrote many low-level tools ourselves. These are projects I contributed to: 

smartmontools: We spent a lot of time tweaking and testing new hard drives not released to the public yet. 
Our lab was the main contributor to the open source project.
I wrote configuration management tools that could be considered precursors to systems like Chef or Puppet. 

We pushed the linux kernel to the limit. I tuned everything from hard drive sector alignment to the number of concurrent processes and the number of open files allowed by the kernel. I hacked on file systems, and ethernet frame sizes to optimize for our specific data.

I helped solve low level problems with the JFS filesystem by contributing the the linux kernel driver code in collaboration with its lead author. 

I made lifelong friends, learned about Linux internals, and fell in love with distributed computing at scale.