Python

In March 2025, I had an idea for improving quality control in PyLossless. I wanted to bring back being able to review an EEG recording to select independent components, choose a time window, and immediately compare the raw signal against the result of removing those components. It had been done in the previous version in MATLAB, but I was less familiar with the inner working of PyLossless at the time. I was also a complete novice when it came to things like Qt 5, PyQt, and the mne-qt-browser. ...

A few years ago, I used Cookiecutter to show how to get from import blah to pip install blah. This sounds simple, but it isn’t. A Python script can live almost anywhere, but a formal package needs the right structure to be built into a wheel. It needs dependency metadata, versions, releases, a licence, and some way to test that it still works. Cookiecutter gave us a standard template. Instead of remembering every file and setting and creating them manually, we answered a few questions and started with a working package. I gave a talk about that workflow in 2022. ...

Starting an EEG study means making dozens of small decisions. Where will the original recordings live? How will they become a BIDS dataset? Which software environment will the study use? Where does preprocessing end and analysis begin? A student has to answer these questions before they’ve had enough experience to know what the consequences will be. Their supervisor then has to review those answers, remember what everyone decided, and figure out whether the project can still be reproduced two years later. ...

In April 2025, I wanted to put PyLossless on the DRAC clusters as an official module. I had at least half a dozen datasets to clean, and I was still managing many projects with separate virtual environments. Reinstalling the same preprocessing package for every dataset made no sense. It turned out that PyLossless was already available as a module. When I asked the internal team to update it, they gave me a list of changes they wanted first, especially if I planned to develop it myself. That request turned into active stewardship of the project and, eventually, co-ownership of its official repository. ...

I first presented Cython in 2020. Five years and several major releases later, I wanted to see whether that old advice still held up. This talk looked at what changed in Cython, whether it got faster, and whether it became any easier to use. I worked through a few demo problems and compared the results with modern Python and its performance libraries. Abstract Python is often praised for its speed of development but criticized for its execution speed. However, this has changed significantly in recent years due to major improvements in both Python itself and its most popular libraries. Libraries such as TensorFlow, OpenCV, NumPy, and Pandas all use a tool called Cython. Cython is an extension of Python that allows functions to be compiled into C or C++, mitigating Python’s performance limitations. ...

Before rewriting slow Python or moving it into Cython, it helps to find out what is actually slow. This Compute Ontario Colloquium was an introduction to Scalene. Scalene separates time spent in Python from time spent in native libraries, and it can also profile memory and GPU use. I ran it on the Alliance systems from a Jupyter notebook, then compared native Python, vectorized code, Cython, and just-in-time compilation. Abstract Python is a language developers choose to write in for convenience rather than speed. However, speed can be recovered by offloading calculations to libraries which leverage lower-level languages like NumPy, Cython, and more. Scalene is a high-performance CPU, GPU, and memory profiler which can illustrate where code should be passing calculations to other libraries for significant increases in speed. Scalene also includes support for Jupyter Notebooks, OpenAI suggestions for vectorizing code, as well as a significantly lower overhead and higher accuracy than other profilers. This talk will introduce the concepts required for understanding why external libraries are faster than native Python, interactions with approaches such as Cython and just-in-time compilers, as well as a live demonstration of Scalene on the Alliance systems inside of a Jupyter Notebook. Familiarity with Python, virtual environments, and Jupyter notebooks will be assumed. ...

In 2022, I gave a talk on creating and distributing Python packages. I used Cookiecutter to show how a template could turn an empty directory into a package with a predictable structure. The goal was to make packaging less mysterious. A new Python package needs more than a folder of source code. It also needs metadata, tests, documentation, a licence, and enough structure that someone else can install it and contribute to it. Cookiecutter gave us a concrete example to inspect instead of building every file by hand. ...

In the early days of working for SHARCNET, my colleague and I decided to standardize how cluster metrics were computed across our internal data frames. As mentioned in a previous post, part of the solution was pandas. The second part was figuring out how to deploy the package for others to contribute to, as well as install on their own specific HPC clusters. Some quick searching revealed that PyPI and pip were the way to go. ...

Eventually I got to the point in data analytics where keeping things in lists, or lists of lists, was no longer quite cutting it. My processing was slowly starting to grind to a halt, and things were getting way too abstract. I decided to call up a friend who had worked in the business longer than me, and they suggested “pandas.” I was vaguely familiar with it because users and clients had used it in the past. A “DataFrame” did sound like it would take care of a lot of my problems after reading the documentation casually… ...

Back when I first got hired at SHARCNET, I used a lot of Python. I mean a lot. This meant I quickly became the lightning rod for all Python-related questions and commentary. During a fun Friday chat, a colleague remarked that Python was, on average, 40x slower than C++. I defended my current language of choice, saying it was better than that, surely. To make a long story short, I was wrong. It really is about 40x slower depending on the problem. Determined to prove myself capable, and my language of choice a bit more defensible, I decided to look into ways to make Python faster. ...