Why parallel computing?
From my very unbiased position, I think working through parallel computing provides you with extremely useful skills and covers such a wide variety of concepts. I specifically built this lab for the Advanced Computer Science students because it was suggested by the professor, and there seemed to be a lot of possibilities in where to take the project. This, however, came back to bite me as I struggled to focus on a single portion of the lab while balancing working on other projects. From flashing operating systems on Raspberry Pis to setting up keys between nodes, and actually writing code, this project was a much greater undertaking than I first envisioned.
Why parallel computing matters
I think Wikipedia puts it best when it says, “Large problems can often be divided into smaller ones, which can then be solved at the same time.”
The idea of splitting up work into smaller chunks is not a new concept, though, and your computer already does this. Many applications already take advantage of multi-core processors, but even applications that don’t use multi-core processing still take advantage of SIMD and multi-ALU systems.
If you look at the processor speeds of CPUs over the past 10 years, you might think that nothing has changed, but under the surface, Intel, AMD, and other tech giants are really improving the number of actions a processor can perform in a single cycle. This can be seen most clearly by the fact that there exist multiple arithmetic logic units (ALUs) on a single core. This essentially means that each core is performing multiple mathematical operations at the same time, allowing the core to speed up its work by a factor equal to the number of ALUs on its chip. The one issue is that not all work can be done simultaneously. For example, a conditional statement must be evaluated before either of its branches because the computer does not know which one to choose. Computer architects are so clever, in fact, that they have found workarounds for even that through branch prediction (this is a tangent; if you are interested in learning more, look here: https://en.wikipedia.org/wiki/Branch_predictor).
All of this is to say that parallel computing is already widely used and speeds up the work computers do significantly.
The general structure of the project
The nodes
This project included five total nodes: the head node and four compute nodes. The head node ran modern desktop Ubuntu and managed the network, storage, and organization of the compute nodes. The head node was an old desktop PC with basic specs. The compute nodes were Raspberry Pi 4s and were designed only to talk to the head node and perform calculations.
The network
The network was hosted by a DHCP server on the head node. This DHCP server had to allow for IP forwarding to the public network to enable updates to the compute nodes. Students were encouraged to create their own Ethernet cables and use them to connect the compute nodes and head node to an unmanaged switch. I thought this was a very important part of the lab because most computer scientists I meet today are just experienced in their small corner of writing Python or C++ in VS Code, and doing something with physical hardware really encourages students to understand what is happening under the hood. Additionally, I required that students set up the DHCP server from scratch (mostly) so they could increase their technical knowledge outside of pure code writing.
OpenMP, OpenMPI, and SLURM
While building a project completely from scratch would have been extremely cool, it is not at all feasible to replicate the work these packages do in just a month as high school students. For this reason, I suggested that students use OpenMP, OpenMPI, and SLURM to coordinate and perform parallel operations. OpenMP and OpenMPI are packages that allow you to parallelize code across multiple computers and cores on those computers, whereas SLURM controls the allocation of resources. While the project could have been done without SLURM, I thought it was important that the parallel computer they set up was as close as possible to one they might find at a research lab or university, and the ability to add programs to a queue is almost always necessary on a supercomputer. Similar to the reasoning for the networking hardware, forcing students to learn about packages they may not have otherwise encountered makes them better computer scientists overall. Specifically, it encourages students to read documentation and work through incompatibilities between systems (Ubuntu and Raspberry Pi OS do not support the same version of SLURM).
The actual code
This part was not really the focus of the project, but in order to create some form of final result, it was necessary. I will not take credit for any part of this section of the lab, as this was all done by the instructor of the class. Generally, students had to write code to analyze LIGO data and DNA ancestry data. In the future, I think it could be really cool for the lab to be extended to give time for students to work through more projects on a distributed computer and learn how to deal with the inherent difficulties of distributed systems.
A fortunate surprise
While working on this lab, students were invited to a lunchtime lecture on how a researcher at UT uses distributed computing in her daily life to analyze certain aspects of coral and marine features. It was gratifying to see that professional institutions use similar frameworks (SLURM), which really reinforces the idea that these students are learning concepts they might actually use in their own research in the future.
A simple example of distributed computing
Imagine you need to process a big list of numbers and compute the square of each value. In a serial program, you would loop through the list one by one.
values = [1, 2, 3, 4, 5, 6, 7, 8]
result = []
for value in values:
result.append(value * value)
print(result)This works, but if the list becomes very large, the runtime grows linearly with the number of elements.
A parallel version might split the list across multiple workers, each one computing squares for its own chunk of the data. The workers then combine their results into one final output.
from multiprocessing import Pool
values = [1, 2, 3, 4, 5, 6, 7, 8]
def square(x):
return x * x
with Pool(processes=4) as pool:
result = pool.map(square, values)
print(result)This is a simplified demonstration, but it captures the core idea: break work into chunks, run the chunks concurrently, then merge the results.
Data issues
One issue that arises in any parallel computing project is how memory is shared. When you have multiple devices that all need access to a single source of data or memory, you start to run into I/O bottlenecks where that single storage location slows down execution. This can sometimes mean there are benefits to having multiple copies of the same data, but this raises even more issues with keeping data in sync and avoiding overwriting the wrong sections of memory. For simplicity in this project, I encouraged a centralized form of storage using an NFS share hosted on the head node.
Final thoughts
While this was by no means the best-built project, and the final resulting computer was no faster than the computers the students were writing their code on, we all learned so much more this way. In the future, I would love to improve the documentation and have a more structured set of instructions for students to follow.
If you have any questions about this lab, feel free to reach out to me. This project is still in use, so I can’t divulge any of its code here, but I am more than willing to discuss any part of the project in more depth with anyone who is interested.