%% Generated by Sphinx.
\def\sphinxdocclass{report}
\documentclass[a5paper,10pt,english]{sphinxmanual}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsible pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}

\PassOptionsToPackage{booktabs}{sphinx}
\PassOptionsToPackage{colorrows}{sphinx}

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}



\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}



\usepackage[Bjarne]{fncychap}
\usepackage{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}

\usepackage{nbsphinx}

% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}


\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{Parsl Guts}
\date{Aug 21, 2024}
\release{1.3.0\sphinxhyphen{}dev}
\author{Ben Clifford}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{Release}
\makeindex
\begin{document}

\ifdefined\shorthandoff
  \ifnum\catcode`\=\string=\active\shorthandoff{=}\fi
  \ifnum\catcode`\"=\active\shorthandoff{"}\fi
\fi

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}


\sphinxstepscope


\chapter{Introduction}
\label{\detokenize{introduction:introduction}}\label{\detokenize{introduction::doc}}
\sphinxAtStartPar
Hello.

\sphinxAtStartPar
These are notes for a 3 hour course (6 x 25 minute sessions) for experienced Parsl users who want to level\sphinxhyphen{}up their ability to use Parsl or to hack on the Parsl codebase by learning more about how Parsl works inside. This is a contrast to the user guide, which focuses on what Parsl looks like from the outside.

\sphinxAtStartPar
This text is not intended to be a comprehensive guide to all parts of the Parsl codebase: there is plenty more to learn about.

\sphinxAtStartPar
I’ll try to include links to relevant external resources: source code, other Parsl documentation, community talks, github issues, papers and other research work.

\sphinxstepscope


\chapter{A sample task execution path}
\label{\detokenize{taskpath:a-sample-task-execution-path}}\label{\detokenize{taskpath::doc}}
\sphinxAtStartPar
the codepath of a task from invoking an app to running on an HTEX worker, and back again

\sphinxAtStartPar
this assumes a basic hpc\sphinxhyphen{}like environment. so lets have a sample configuration involving htex and the slurm provider and not much else.

\sphinxAtStartPar
TODO: sample configuration, as defaulting as possible:

\sphinxAtStartPar
here’s an app that adds two numbers

\sphinxAtStartPar
TODO: app definition

\sphinxAtStartPar
and now we can initialise a Parsl context manager, invoke the app and wait for its result.

\sphinxAtStartPar
TODO: with/invoke code blocks

\sphinxAtStartPar
Deliberately nothing fancy there: this is getting\sphinxhyphen{}started levels of Parsl use.

\sphinxAtStartPar
Now lets pick apart what happens. I’m going to ignore: the startup/shutdown process (what happens with parsl.load() and what happens at the end of the with block), and I’m going to defer batch system interactions to another section (TODO: hyperlink blocks)


\section{a decorated app}
\label{\detokenize{taskpath:a-decorated-app}}
\sphinxAtStartPar
first lets look at what we defined when we defined our app. Normally \sphinxhref{https://docs.python.org/3/reference/compound\_stmts.html\#def}{\sphinxcode{\sphinxupquote{Function definitions}}} defines a function (or a method) in Python. With the python\_app decorator, instead that defines a PythonApp object. That’s something we can invoke, like a function, but it’s going to do something parsl specific.

\sphinxAtStartPar
look at definition of python app and see the \sphinxcode{\sphinxupquote{\_\_call\_\_}} definition. when you invoke an app myapp(5,6), then the relevant \sphinxcode{\sphinxupquote{PythonApp.\_\_call\_\_}} method is invoked. the decorator stashed away the actual user defined body of code in an attribute, and it can pass that into \sphinxcode{\sphinxupquote{\_\_call}}


\section{the data flow kernel}
\label{\detokenize{taskpath:the-data-flow-kernel}}
\sphinxAtStartPar
we can have a look at that method and see that to “invoke an app”, we call a method on the DataFlowKernel (DFK), the core object for a workflow (historically following the \sphinxhref{https://en.wikipedia.org/wiki/God\_object}{God\sphinxhyphen{}object antipattern}).

\sphinxAtStartPar
inside the DFK:
\begin{itemize}
\item {} 
\sphinxAtStartPar
create a task record and an AppFuture, and return that AppFuture to the user

\end{itemize}

\sphinxAtStartPar
Then asynchronously:
\begin{itemize}
\item {} 
\sphinxAtStartPar
perform “elaborations” \sphinxhyphen{} see elaborations chapter

\item {} 
\sphinxAtStartPar
send the task to an Executor (TODO:hyperlink class docstring). in this case we aren’t specifying multiple executors, so the task will go to the default single executor which is an instance of the High Throughput Executor (TODO: hyperlink class docstring) \sphinxhyphen{} which generates an executor level future

\item {} 
\sphinxAtStartPar
wait for completion of execution (success or failure) signlled via the executor level future

\item {} 
\sphinxAtStartPar
a bit more post\sphinxhyphen{}execution elaboration

\item {} 
\sphinxAtStartPar
set the AppFuture result

\end{itemize}

\sphinxAtStartPar
so now lets dig into the high throughput executor. the dataflow kernel hands over control to whichever executor the user configured (the other options are commonly the thread pool executor (link) and work queue (link) although there are a few others included). but for this example we’re going to concentrate on the high throughput executor. If you’re a globus compute fan, this is the layer at which the globus compute endpoint attaches to the guts of parsl \sphinxhyphen{} so everything before this isn’t relevant for globus compute, but this bit about the high throughput executor is.

\sphinxAtStartPar
The data flow kernel will have performed some initialization on the high throughput executor when it started up, in addition to the user\sphinxhyphen{}specified configuration at construction time \sphinxhyphen{} (TODO: perhaps this is in enough of one place to link to in the DFK code?). for now, I’m going to assume that all the parts of the high throughput executor have started up correctly.

\sphinxAtStartPar
htex consists of a small part that runs in the user workflow process (TODO: do I need to define that as a process name earlier on in this chapter? it’s somethat that should be defined and perhaps there should be a glossary or index for this document for terms like that?) and several other processes.

\sphinxAtStartPar
The first process in the interchange (TODO: link to source code). This runs on the same host as the user workflow process and offloads task and result routing.

\sphinxAtStartPar
Beyond that, on each worker node on our HPC system, a copy of the process worker pool will be running. These worker pools connect back to the interchange using two network connections (ZMQ over TCP) \sphinxhyphen{} so on the interchange process you’ll need 2 fds per node \sphinxhyphen{} this is a common limitation to “number of nodes” scalability of Parsl. (see \sphinxhref{https://github.com/Parsl/parsl/issues/3022}{issue \#3022} for a proposal to use one network connection per worker pool)

\sphinxAtStartPar
so inside htex.submit:
we’re going to:
\begin{itemize}
\item {} 
\sphinxAtStartPar
serialize the details of the function invocation into a sequence of bytes. this is non\sphinxhyphen{}trivial even though everyone likes to believe it is magic and simple. In a later chapter I’ll talk about this in much more depth (TODO: link pickle)

\item {} 
\sphinxAtStartPar
send that byte sequence to the interchange over ZMQ

\item {} 
\sphinxAtStartPar
do a bit of book keeping

\item {} 
\sphinxAtStartPar
create and return an executor future back to the invoking DFK \sphinxhyphen{} this is how we’re going to signal to the DFK that the task is completed (with a result or failure) so it is part of the propagation route of results all the way back to the user.

\end{itemize}


\section{the interchange}
\label{\detokenize{taskpath:the-interchange}}
\sphinxAtStartPar
The interchange matches up tasks with available workers: it has a queue of tasks, and it has a queue of process worker pool managers which are ready for work. so whenever a new task arrives from the user workflow process, or when a manager is ready for work, a match is made. there won’t always be available work or available workers so there are queues in the interchange.

\sphinxAtStartPar
the matching process so far has been fairly arbitrary but we have been doing some research on better ways to match workers and tasks. (TODO: what link here? if more stuff merged into Parsl, then the PR can be linkable. otherwise later on maybe a SuperComputing 2024 publication \sphinxhyphen{} but still unknown)

\sphinxAtStartPar
so now, the interchange sends the task over one of those two zmq\sphinxhyphen{}over\sphinxhyphen{}TCP connections I talked about earlier… and we’re now on the worker node where we’re going to run the task.


\section{the process worker pool}
\label{\detokenize{taskpath:the-process-worker-pool}}
\sphinxAtStartPar
Generally, a copy of the process worker pool runs on each worker node. (other configurations are possible) and consists of a few closely linked processes:

\sphinxAtStartPar
the manager process which interfaces to the interchange (this is why you’ll see a jumble of references to managers or worker pools in the code: the manager is the externally facing interface to the worker pool)

\sphinxAtStartPar
worker processes \sphinxhyphen{} each worker process is a worker. there are a bunch of configuration parameters and algorithms to decide how many workers to run \sphinxhyphen{} this happens near the start of the process worker pool process in the manager code. (TODO: link to worker pool code that calculates number of workers)

\sphinxAtStartPar
the task arrives at the manager, and the manager dispatches it to a free worker. it is possible there isnt’ a free worker, becuase of the preloading feature for high throughput (TODO link to docstring) \sphinxhyphen{} and the task will have to wait in another queue here \sphinxhyphen{} but that is a rarely used feature.

\sphinxAtStartPar
the worker then deserialises the byte package that was originally serialized all the way back in the user submit process: we’ve got python objects for the function to run, the positional arguments and the keyword arguments.

\sphinxAtStartPar
so at this point, we invoke the function with those arguments (link to the \sphinxcode{\sphinxupquote{f(*args, **kwargs)}} line)

\sphinxAtStartPar
and the user code runs!

\sphinxAtStartPar
it’s probably going to end in two ways: a result or an exception
(actually there is a common third way, which is that it kills the unix\sphinxhyphen{}level worker process for example by using far too much memory or by a library segfault \sphinxhyphen{} or by the batch job containing the worker pool reaching the end of its run time \sphinxhyphen{} that is handled, but we’re ignoring that here)

\sphinxAtStartPar
now we’ve got the task outcome \sphinxhyphen{} either a Python object that is the result, or a Python object that is the exception. We pickle that object and send it back to the manager, then to the interchange (over the \sphinxstyleemphasis{other} ZMQ\sphinxhyphen{}over\sphinxhyphen{}TCP socket) and then to the high throughput excecutor submit\sphinxhyphen{}side in the user workflow process.

\sphinxAtStartPar
Back on the submit side, there’s a high throughput executor process running listening on that socket. It gets the result package and sets the result into the executor future (TODO code reference). That is the mechanism by which the DFK sees that the executor has finished its work, and so that’s where the final bit of “task elaboration” (TODO: link to elaboration chapter) happens \sphinxhyphen{} the big elaboration here would be retries on failure, which is basically do that whole HTEX submission again and get a new executor future for the next try. (but other less common elaborations would be storing checkpointing info for this task, and file staging)

\sphinxAtStartPar
When that elaboration is finished (and didn’t do a retry), we can set that same result value into the AppFuture which all that long time ago was given to the user. And so now future.result() returns that results (or raises that exception), back in the user workflow, and we’re done.

\sphinxAtStartPar
TODO: label the various TaskRecord state transitions (there are only a few relevant here) throughout this doc \sphinxhyphen{} it will play nicely with the monitoring DB chapter later, to they are reflected not only in the log but also in the monitoring database.

\sphinxstepscope


\chapter{Blocks}
\label{\detokenize{blocks:blocks}}\label{\detokenize{blocks::doc}}
\sphinxAtStartPar
In the task overview, I assumed that process worker pools magically existed on worker nodes. In this section, I’ll talk a little bit more about how that actually happens, using Parsl’s provider abstraction.

\sphinxAtStartPar
The theme of this section is: get process worker pools running on some nodes that we want to do the work.

\sphinxAtStartPar
We don’t need to describe the work (much), because once the workers are running they’ll get their own work from the interchange, as I talked about in the previous section.

\sphinxAtStartPar
LRM providers, batch jobs, workers, scaling strategies, batch job environments (esp worker\_init)

\sphinxstepscope


\chapter{Serializing tasks with Pickle and dill}
\label{\detokenize{pickle:serializing-tasks-with-pickle-and-dill}}\label{\detokenize{pickle::doc}}
\sphinxAtStartPar
TODO: an emphasis on the common parsl problems: (un)installed packages, functions and exceptions

\sphinxAtStartPar
intro should refer to not regarding this as magic, despite most people desperately hoping it is magic and then not trying to understand whats happening. this is needs a bit of programming language thinking, way more than routing “tasks as quasi\sphinxhyphen{}commandlines”

\sphinxAtStartPar
I’ll use the term pickling and serializing fairly interchangeably: serialization is the general word for turning something like an object (or graph of objects) into a stream of bytes. Pickling is a more specific form, using Python’s built in {\hyperref[\detokenize{pickle::doc}]{\sphinxcrossref{\DUrole{doc,doc,doc}{Serializing tasks with Pickle and dill}}}} library (TODO: hyperlink pickle).

\sphinxAtStartPar
As I mentioned in an earlier section, (TODO: backlink hyperlink?) when htex wants to send a function invocation to a worker, it serializes the function and its arguments into a byte sequence, and routes that to a worker, where that byte sequence is turned back into objects that are in some sense equivalent to the original objects. Task results follow a similar path, in reverse.

\sphinxAtStartPar
That serialization is actually mostly pluggable, but basically everyone uses some variant of pickle (most often the dill library) because that’s the default and there isn’t much reason to change.

\sphinxAtStartPar
For most things that look like simple data structures, pickling is pretty simple. For example, almost anything that you can imagine some obvious representation in JSON, plain pickle won’t have a problem.

\sphinxAtStartPar
There are a few areas where it helps to have some deeper understanding of whats going on, so that you don’t run into “mystery pickling errors because the magic is broken.”

\sphinxAtStartPar
TODO: review my pickle talk, figure out what is relevant or not. maybe don’t need to talk about pickle VM opcodes, just the remote\sphinxhyphen{}execution facility at a higher level? and the import facility at a higher level? no need to talk about recursive objects \sphinxhyphen{} that’s not a user facing problem (unless you’re trying to build your own pickle scheme)


\section{More info}
\label{\detokenize{pickle:more-info}}
\sphinxAtStartPar
I’ve talked about Pickle in more depth and outside of the Parsl context at PyCon Lithuania (TODO: link slides and video)

\sphinxAtStartPar
Proxystore \sphinxhyphen{} reference its use in Parsl, and reference a citation for just proxystore. TODO

\sphinxstepscope


\chapter{Elaborating tasks}
\label{\detokenize{elaborating:elaborating-tasks}}\label{\detokenize{elaborating::doc}}
\sphinxAtStartPar
stuff that the DFK does to a task that isn’t “just run this task”

\sphinxAtStartPar
dependencies, retries, checkpointing, file staging, join\_app joining, monitoring resource wrapper

\sphinxstepscope


\chapter{understanding the monitoring database}
\label{\detokenize{monitoring:understanding-the-monitoring-database}}\label{\detokenize{monitoring::doc}}
\sphinxAtStartPar
this should focus on making use of data in the monitoring database, not on how monitoring is architected, exceptions

\sphinxstepscope


\chapter{modularity and plugins}
\label{\detokenize{plugins:modularity-and-plugins}}\label{\detokenize{plugins::doc}}
\sphinxAtStartPar
which bits you can swap for other plugins: how and why

\sphinxAtStartPar
why includes sustainability work on different quality of code/maintenance

\sphinxAtStartPar
if there’s a decision point that looks like a multi\sphinxhyphen{}way if statement \sphinxhyphen{} having a bunch of choices is a suggestion that choices you might not have implemented might also exist, and someone might want to put those in. various plugin points then look like “expandable if” statements. a good contrast is the launcher plugin interface, vs the hard\sphinxhyphen{}coded MPI plugin interface (cross reference issue to fix that)

\sphinxAtStartPar
it’s also a place to plug in “policies” \sphinxhyphen{} that is user\sphinxhyphen{}specified decisions (such as how to retry, using retry handlers) that take into account the ability of our users to write Python code as policy specifications.

\sphinxAtStartPar
Parsl exists as a library within the python ecosystem. Python \sphinxstyleemphasis{exists}.

\sphinxAtStartPar
Doing that sort of stuff is what I’d expect as part of moving from being a tutorial\sphinxhyphen{}level user to a power user.

\sphinxstepscope


\chapter{Colophon}
\label{\detokenize{colophon:colophon}}\label{\detokenize{colophon::doc}}
\sphinxAtStartPar
Written in rst

\sphinxAtStartPar
Rendered with sphinx

\sphinxAtStartPar
Edited with \sphinxcode{\sphinxupquote{vi}} and vscode

\sphinxAtStartPar
This text was prepared against Parsl version 2024.08.19



\renewcommand{\indexname}{Index}
\printindex
\end{document}