How to Clean Up

When you work with external resources such as a database or temporary files, you often need to run some cleanup actions after you've done the work.

Python provides two options - the context manager and the try/finally block. Both are valid options, but the context manager is often lauded as being more Pythonic.

Despite this, try/finally block is still widely used. As we will discover, try/finally is simple to use and well-suited in some cases, but it does come with pitfalls, as the messenger tragically discovered.

Let us try and contact the messenger.

Part 1: The Assured Action

A variant of the try/finally pattern exists in most languages1 and functions pretty much in the way you might imagine. Consider this trivial example:

def walk_path():
    try:
        print("Taking a step")
    finally:
        print("Leaving a footprint")

'Taking a step'
'Leaving a footprint'

The interpreter enters the first block and executes the statement. The interpreter then proceeds to the finally block. The final block always executes. It does not matter if the first block succeeds or fails.

If a failure is raised during execution as shown below, the error disrupts the normal flow and the interpreter stops executing the try block immediately. The interpreter then jumps directly to the finally block.

def walk_path():
    try:
        raise Exception("A fallen tree")
    finally:
        print("Leaving a footprint")

If you choose to handle the failure using an except block as shown below, the error is caught and handled by the except block before proceeding to the finally block. An error could also occur inside the except or else block. The interpreter would still execute the finally block before raising the new error.

def walk_path():
    try:
        raise Exception("A fallen tree")
    except Exception:
        print("Climbing over the trunk")
    else:
        print("Kicking the trunk away using superhuman strength")
    finally:
        print("Leaving a footprint")

Part 2: The Trapped Messenger

This brings us to the behavior of returning values. A function can return a value from within the try block. When the interpreter encounters this return statement, It prepares to send this value back to the caller. But first, it must honor the finally block. So it pauses the return process and executes the finally block first before returning the prepared value from the try block.

def walk_path():
    try:
        return "Reaching the destination"
    finally:
        print("Leaving a footprint")

The finally block can also contain its own return statement, as shown below. When this happens, the return in the finally block wins and the return value from the try is effectively ignored.

def walk_path():
    try:
        return "Reaching the destination"
    finally:
        return "Returning home"

This behavior applies to exceptions as well. We can place a loop around our structure to observe break and continue statements.

def scout_path():
    for step in range(3):
        try:
            raise Exception("A hidden trap")
        finally:
            break
    return "The scout survives"
'The scout survives'

A break statement inside a finally block will swallow any unhandled exception from the try block. A continue statement will do the exact same thing. The exception disappears completely.

Part 3: The Trapped Voice

Lets examine a more complex example with nested try/finally statements. Do return statements break out of parent try/finally blocks?

def send_message():
    try:
        print("Everything is fine")
        return 0
    finally:
        try:
            try:
                print("Everything is still fine")
            finally:
                for x in range(2):
                    print(f"Scouting area {x}")
                return 1
        finally:
            for x in range(2):
                print(f"Covering tracks in area {x}")
                return 2
print(f"Return value: {send_message()}")

Everything is fine
Everything is still fine
Scouting area 0
Scouting area 1
Covering tracks in area 0
Covering tracks in area 1
Return value: 2

No, try/finally statements can be nested, and return statements from child blocks do not prevent parent finally blocks from running. However, the value from the last return statement trumps the rest and is still the one that is returned by the function.

Part 4: The Plot Thickens

As you can imagine, any language which allows you to write dead code that produces unintended outcomes is problematic. The Python language developers recognized this danger, and proposed that return/break/continue statements should be disallowed in finally blocks in PEP6012.

However, it was voted down for the following reason:

Reading the references in the PEP it seems to me that most languages implement this kind of construct but have style guides and/or linters that reject it. I would support a proposal to add this to PEP 8 (if it isn’t already there).

I note that the toy examples are somewhat misleading – the functionality that may be useful is a conditional return (or break etc.) inside a finally block.

-Guido 2019 PEP601

Guido’s reasoning was that there may be valid scenarios where the user requires full control of exception handling in the finally block, and may wish to override the raising of exceptions. Preventing this behavior would effectively hamstring advanced users.

However, in 2024 the community tried again with PEP7653. This time they were armed with evidence. They analyzed the top 8000 PyPi packages and found that:

Most of the usages (of return in finally) are incorrect, and introduce unintended exception-swallowing bugs. - PEP765

This was enough for the proposal to get over the line, and from Python 3.14 onwards, using return, break or continue in a finally clause emits a SyntaxWarning.

Part 5: Why use try/finally at all?

A context manager is the pythonic choice most of the time when you’re working with resources that already expose acquire/release semantics that can easily slot into __enter__ and __exit__. It’s the natural choice for files, locks, database connections, temporary state changes, etc. It’s declarative and minimizes surface area for errors.

However, they don’t always make sense:

• Context manager code lives in a different location, and so introduces an extra layer of abstraction. Sometimes your code is so simple and localized that introducing an extra layer of abstraction would make the code less readable with little benefit.

• Context managers can only use variables passed in during initialization, whereas finally can reference variables mutated during execution in the try block.

• With try/finally, you can combine except and finally clauses to explicitly manage different failure modes for more granular control.

Closing the Circle

The novice believed the original message was secure. The master understood the final seal controls the truth.

The finally block always speaks last. You must ensure its final words do not obscure the truth of what came before.

When crafting software, we often mistake a side effect for a feature. We assume that because a behavior works once, it will always work in all situations. In the transition from Python 3.11 to 3.12 the importlib.resources underwent such a refinement. A shortcut that once worked was revealed to be an illusion.

Let us peer at the sky slowly.

Part 1: The Nature of the Traversable

In Koan 14 we explored how non-python files can be included within a python package. importlib.resources is a Python standard library which lets you bundle up configuration files, database schemas, binaries and any other non-source files you may need to run your code.

When you run importlib.resources.files(“my_package.test_data”), you might expect to get back a Path, but instead you get a Traversible object.

A Traversable is an abstract class which provides “a subset of pathlib.Path methods suitable for traversing directories and opening files.”.

importlib.resources.files returns a Traversable instead of a regular pathlib.Path because package data files can exist in various formats on disk:

1. Within a tar file (in the case of wheels)

2. In a plain directory (in the case of editable installs)

3. Spread across multiple packages (in the case of namespace packages, which we learnt about in Koan 7).

It is a promise that something behaves like a file system path even if it is trapped inside a zip file or a remote package.

Part 2: The Differences Emerge

In our offending example, importlib.resources.files returns a MultiplexedPath. And this usually gives us most of the features we need to work with a path:

But sometimes we have code trapped in a library or an upstream function which expects a Path and uses methods unsupported by a Traversable:

And that might tempt us into using a shortcut to convert the MultiplexedPath1 into a Path.

Part 3: The Accidental Shortcut

In Python <=3.11, MultiplexedPath.joinpath("") would return the first physical path it found. It felt like a convenient way to “extract” a real path.

By passing an empty string to joinpath you were asking the system to join “nothing” to the collection. The system responded by handing back the first item it held as a PosixPath. However, this turned out to be a quirk of the implementation, rather than a contract.

In Python 3.12 the implementation of joinpath was made more generic and robust. It began to strictly follow the logic of its ancestors. When you ask to join a path to a MultiplexedPath it now iterates through its internal requirements.

When you pass an empty string or a dot the iterator now reaches the end of its instructions and finds nothing to give. This tripped up a lot of developers, and users of libraries.

Part 4: Facing the Eclipse

If your intent is to turn a Traversable object into a real physical file path, you must use a tool designed for that purpose. We should not use joinpath to change the nature of an object. Instead, use as_file to convert a MultiplexedPath to a PosixPath safely:

The Emperor lost his kingdom because he had mistaken a pattern for a law. He claimed power over the night because he had never seen the night without the light. In the same way, a programmer who relies on the side-effects of a method mistakes a consistent coincidence for a formal promise.

In Python, a raw string literal is a string marked by the r prefix. It instructs the Python interpreter to treat the string “as it is written”, ignoring any escape sequences (such as \n for newline, or \t for tab).

Last week in Koan 17 we explored how raw string literals can be used to prevent the “plague of backslashes” needed to search for strings containing escape sequences.

While raw string literals are certainly handy, there is one pitfall to watch out for. If a raw string ends in an odd number of backslashes, the last backslash escapes the closing quote. The string never ends, and the parser wanders into the void.

Part 1: The Broken Promise of Rawness

In a raw string literal one might assume that the backslash loses all of its power. However, this is not true. The backslash is stripped of its ability to create special characters (escape sequences), but it is not stripped of its ability to escape the next character.

Consider what a parser might do for the following:

s = r"The path is \"

When the parser encounters \”, it follows a strict rule: A backslash before a quote means the quote is part of the data, and not the end of the container. Even though this is a raw string, the parser still lets you include quotes:

The backslash prevented the string from closing, but it also remained in the string.

Part 2: The Unbound String

This leads us to the weaver's bundle. If you want a raw string that ends with a single backslash, you create a problem for the parser. With the following string, your intent might be to create a string containing just a backslash, but instead:

1. The parser sees the opening ”

2. It enters raw mode

3. It sees \

4. It sees ” immediately after

5. Because of the backslash, it treats the “ as a literal character inside the string

6. It continues looking for the closing quote, encounters the end of the line and raises a SyntaxError

This is why you can’t create a raw string literal that ends in an odd number of backslashes. The sharp edges of the bamboo pierce the binding rope.

Part 3: Securing the Bundle

If you absolutely must end a string with a backslash, you cannot use a raw string literal in isolation. You must separate the content from the termination.

There are several ways to tie this knot.

1. Concatenate the raw string literal with a regular string with a single escaped backslash

2. Use an fstring, and either include the backslash as a variable, or escape it outside a variable

3. Same as the first method, but concatenate the string by using join on an array of strings containing the escaped backslash

The Master’s Reflection

The raw string is a tool of convenience, but it is still a servant to the parser. While it treats backslashes as literals for the sake of the string's value, it still respects backslashes for the sake of the string's syntax. A diligent developer separates the boundary from the content when these conflicts arise.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

Regex: Searching for the tiger

Python’s `re` module is a robust toolset for writing regular expressions (regex), but its behavior often deviates from other engines. Understanding the nuances of the interpreter and the Unicode standard is essential for writing predictable patterns.

Let us explore the treacherous terrain of the re module.

Part 1: Anchoring: match() vs. search()

A common misconception is that re.match() scans the entire string. In reality, it is implicitly anchored to the start of the string (effectively adding a \A or ^ anchor). If you want to find a pattern anywhere in the input, use re.search() instead.

You can think of re.match() like the watchman at the gate, fastidiously inspecting anything coming into the city. If the string you are searching for is not right at the start, he declares it absent.

re.search on the other hand is more like a scout. He wanders through the city looking for that which is hidden. This scans the entire string to find a match.

If you wish to find a pattern that may be hidden deep within the text, you must send the scout (re.search) instead of the guard (re.match).

Part 2: The Map and the Territory (Raw Strings)

Python strings process backslashes (\) as escape sequences before they reach the regex engine. An escape sequence1 is a sequence of characters following a backslash which the Python interpreter treats differently. There are several such strings:

To prevent this, and to match a literal backslash in regex one must create a “plague of backslashes”. For example, to match a literal backslash you would need four backslashes (\\\\)

As you can see this is prone to error as you can easily use the wrong number of backslashes. A master uses raw string literals instead2 (r""). This tells Python to ignore the escape sequences and pass the string exactly as written to the regex engine.

Part 3: The Illusion of the End

By default, the $ anchor matches the end of the string or the position immediately preceding a newline at the very end. This is fine most of the time, but sometimes you may want to verify that a string contains no trailing whitespace or newlines:

To strictly enforce the absolute end of the string and disregard any trailing newlines, use \Z. This only matches the absolute end of the input.

Part 4: The Thousand Faces of Numbers

In the modern era of Python 3, the definition of the "digit" escape sequence (\d) has expanded beyond the Arabic numerals 0-9.

Python 3’s re module is Unicode aware, which means the regex token \d does not just mean the ASCII digits 0–9 anymore. It matches any character that has the Unicode property of Numeric_Type=Decimal.

For example, the Tamil characters ௮ (U+0BEA) and ௰ (U+0BF0) are officially categorized as digits/numbers in the Unicode database.

When a developer writes a regex to extract a price, they might assume \d+ will only return characters that can be easily cast to an integer using standard libraries, or characters they can safely display in an ASCII-only receipt.

You can imagine the chaos that would be caused if a payment system accepts ௮௰ as a valid “number” but the downstream payment processor only understands 80. It might crash or, worse, treat the value as 0 or null, leading to logic errors.

If your logic depends on strictly parsing standard 0-9 integers, you must instruct the engine to ignore the wider world using the re.ASCII flag.

Part 6: The Broken Collection (findall)

The re.findall() method is a powerful tool for gathering all occurrences. However, its changes it’s return type based on the pesence of “capturing groups” (parentheses) in the regex.

If there are no capturing groups, it returns a list of strings. If there is a group, it returns only the contents of that matched group.

Because of variability in return type, it is not advisable to run re.findall() on a regex you do not know in advance (i.e. from a variable or user input).

If you want to ensure consistent output regardless of pattern structure, use re.finditer().

The Final Pattern

The regex engine is a powerful but tricky feature to master, but the following guidelines can help light your way:

1. Know when to choose a watchman (match) or a scout (search)

2. Use raw strings over escaped backslashes.

3. Know the true boundaries when searching the end of a line ($ vs \Z).

4. Handle Unicode when matching numbers

5. Prefer re.finditer() over re.findall() for ensure consistent output regardless of regex

Do not assume the pattern you see in your mind is the pattern you have written on the scroll.

Firstly, I’d like to apologize sincerely for letting this slip through the cracks. I try to be as diligent as possible when writing each post, but this one wasn’t my best work. I’ve put some steps in place so that this doesn’t happen again. Also, a big thanks to @vekyll who pointed some of the issues out.

Secondly, I’ve edited the original post to correct those mistakes, but I thought it would be worthwhile to revisit some fundamental concepts underpinning the topic here, and also expand on the corrections.

Python Statements and Expressions

The building blocks of Python are statements. A statement is a block of code which does something. If a statement can fit in one line, it is a Simple Statement, otherwise it’s a Compound Statement.

An expression is a special type of statement which evaluates to a single value:

An expression is an accumulation of expression elements like literals, names, attribute access, operators or function calls which all return a value. In contrast to many other languages, not all language constructs are expressions. There are also statements which cannot be used as expressions, such as while.

from Python Glossary

1. Assignments don’t return values

An assignment is a statement, but it is not an expression. This is because it doesn’t return a value. Some of the text from the original post implied that assignment returns a value. For e.g.

This assignment itself returns the value that was assigned

This is false, assignment never returns a value.

You also can’t use assignments inside expressions like you can in C (like if (x = y)). However, one exception was introduced in Python 3.8. Assignment expressions (also known as the walrus operator) like if (x := y): lets you assign values within an expression.

2. Chained Assignment evaluates RHS, then binds names left -> right

The original post stated that chained assignments are evaluated right to left, but this is misleading. For this example from the original post:

x = 10
x = y = z = x + 2

Python evaluates the right-hand-side x + 2 first, then binds the value to the names from left to right. i.e x, y, and then z last.

Python does not evaluate chained assignment by returning a value from each individual assignment and then reassigning it leftwards.

3. Comparison Chains Do Not Behave Like Assignment Chains

The original post ambiguously stated that chained operations like 1 < x < 10 and chained assignments operate under the same principle:

The same principle applies to chained comparison operators such as <, > and ==.

This is false of course, and the rest of the section does gets it right. Comparison chains are essentially syntactic sugar which evaluates to distinct comparisons and-ed together: (1 < x) and (x < 10).

Thanks again for continuing to read Python Koans. I hope you’ve learnt something new along the way. I’ve got lots of great content and a new learning platform planned for the new year. Stay tuned!

The Hidden Teaching: The Boundary of Execution

One of the most ubiquitous pieces of Python code is the expression if __name__ == “__main__”. You will often find this in standalone scripts or entrypoints to cli tools. But why is this needed and what are the consequences if this simple statement is left out? Some of these may surprise you.

Let us begin again.

Part 1: The Dual Nature of Python Files

When Python processes a file, whether by running it directly (python training_hall.py) or by importing it (import training_hall):

1. It reads the code from top to bottom of the file, executing all code that is not inside a function or class definition.

2. It inserts all objects (functions, classes and variables) into the current namespace (using the scoping rules we learnt in Koan 4).

In our introductory snippet, the functions fetch_staff and conduct_lesson are defined on import, becoming functions available in the global scope. The code inside them is not executed. This is correct behavior for a utility.

Observe what happens without the protective boundary:

if another file attempts to import this file, the terminal will print:

The master enters. The lesson begins.
A staff is taken from the rack.

The lesson begins prematurely. The boundary was not drawn to separate definition from invocation.

The special attribute __name__ is a string that holds the module’s identity.

• When the file is executed directly, Python’s master interpreter sets __name__ to the string “__main__”.

• When the file is imported, __name__ is set to the module’s file name (e.g., “training_hall”).

The guard acts as a lock:

if __name__ == “__main__”:
    conduct_lesson()

Now, if imported, __name__ is “training_hall”. The condition is False, and conduct_lesson() is not run.

Part 2: The Breakage of Relative Paths

Sometimes modules use relative imports to refer to its siblings:

When you run python mypackage/temple.py directly, you will be greeted with this error:

ImportError: attempted relative import with no known parent package

When run directly, temple.py is not part of a package, so it’s __name__ becomes __main__, not mypackage.temple. Python’s import system expects a module run as main to be at the package root. The relative import from . (the current directory) breaks as the package structure is lost.

The fix is to run the code as a module: python -m mypackage.temple; or switch to a relative import: import mypackage.temple.

Part 3: A Fine Pickle

The pickle module can be used to serialize and deserialize python objects to disk. To do this, it stores object references by their importable path, e.g. mypackage.temple.ClassName. Imagine you had two files - save_code.py which saved an object to disk in the main module, and another - load_code.py which tried to load the pickled object from disk:

This would throw an error:

AttributeError: Can’t get attribute ‘Model’ on <module ‘__main__’>

When you define a class or function in __main__, its path is “__main__.ClassName”, which gets saved in the pickled object. When the object is unpickled in load_code.py “__main__.ClassName” doesn’t exist in the module. So data pickled in the __main__ module cannot be re-imported later when unpickling elsewhere. The fix is to ensure that you always pickle in an importable module:

Part 3: The Child Process and the Re-Import

In certain operating systems, especially Windows, Python’s multiprocessing library avoids sharing memory state directly between a parent and its new child process. Instead, it re-imports the entire main module into the new child process to ensure a clean start. Consider the risk:

Without a guard, the child process, upon re-importing master.py, would execute the process creation line again, leading to an infinite creation of child processes.

Again, our simple guard prevents this:

The child process, when re-importing the module, sets the module’s name to its file name (e.g., ‘master’), not “__main__”, thus bypassing the process creation code and safely starting its task.

Part 4: Python Execution Types

There are other ways a module may be executed without becoming the true primary executable, each leading to a different __name__ value.

• When running a module with python -m my_package.my_module, the runpy1 utility is used. This forces __name__ to “__main__”, allowing execution of a module within a package structure.

• In environments like Jupyter or the basic Python REPL, an executed cell or a pasted block of code is considered the immediate execution context, and its __name__ is also temporarily “__main__”.

• In test environments (like pytest) or when a file is genuinely imported __name__ will not be “__main__”. Test files are executed as imported modules, not as main scripts, ensuring that their test suites do not inadvertently run the protected code.

Ending The Lesson

The Master’s teaching was simple: the staff stands ready on the rack, until the lesson is initiated by the master. The guard if __name__ == “__main__”: serves as the master’s permission, ensuring that the module’s core logic is run only when needed.

It separates code that is merely defined from code that is executed and answers a fundamental question Python asks of every file it touches: Are you a library I should use, or a program I should run?

The Hidden Teaching: The Depth Below the Surface

When the eye sees a character it perceives a glyph1, which is a single visual form. Computers represent the glyph using a number (a code point) within a lookup table (a codespace). To the eye the forms may be identical, but to the machine they are different.

To work with text is to know the difference between what is seen and what is unseen.

Part 1: The Illusion of Sameness

The novice’s confusion is a common one. Two strings can appear to be the same character yet possess different underlying structures. This is the issue of canonical equivalence2 in Unicode.

Consider the character é. It can be represented in two ways:

1. Normalization Form C (NFC): A single code point, U+00E9 (LATIN SMALL LETTER E WITH ACUTE). This is the composed form.

2. Normalization Form D (NFD): Two code points, U+0065 (LATIN SMALL LETTER E) followed by U+0301 (COMBINING ACUTE ACCENT). This is the decomposed form.

Python treats them as distinct sequences of code points, which are unequal in length and content:

Part 2: Achieving Canonical Truth with unicodedata

To compare characters based on their visual or canonical meaning, we must normalize them. Python’s unicodedata module provides the tool for this alignment.

The unicodedata.normalize() function converts strings to a standard form ensuring that sequences that are visually the same (canonically equivalent) are also code-point-wise the same.

By normalizing both strings to NFC we ensure that the decomposed form e\u0301 is converted into its single-point composed form é making the comparison meaningful and true to the eye’s perception.

The common wisdom for text processing is:

• Use NFC (Composition) for storage and transmission of unicode text.

• Use NFD (Decomposition) for text processing and complex comparisons.

Part 3: The Deception of String Length

As we saw previously, the built-in len() function counts the number of code points, not the number of visual characters or graphemes. This is important for tasks like truncating text display or cursor movement which operate on grapheme clusters3.

To correctly count the visual characters, one must implement a more sophisticated counting mechanism which understands the rules of grapheme clustering. For simple cases, normalization can offer a partial solution.

For example, if we normalize to NFC before counting, we often align the code point count closer to the grapheme count:

This is a helpful step but it is not a complete answer for all complex scripts and combining marks which may still result in a grapheme that is more than one code point, even in NFC.

Part 4: Seeing the Unseen

Beyond composition lies the matter of invisible characters. Control characters4 like the Zero Width Space U+200B or soft hyphens are valid code points but have no visual rendering. They are often copied from web sources and cause silent failures in parsing or comparison.

When inspecting strings the standard print() function often hides the true nature of the text. The repr() function however reveals the string’s raw code point sequence making the invisible visible.

The Zero Width Space \u200b is revealed by repr(). Before comparison or processing it is often necessary to aggressively strip or replace such characters.

Part 5: Homographic Attacks

The principle of canonical equivalence and unseen control characters have been pu to nefarious use in phishing attacks. Attackers register a domain name containing one or more unicode characters which are canonically equivalent to a well known character.

They then host an identical copy of the legitimate website on the newly registered domain, and steal user data or infect their computer with malware.

The following is an example of an homographic attack; the Latin letters “e” and “a” are replaced with the Cyrillic letters “е“ and “а“5

This necessity to look beneath the surface reveals a universal principle: Trust the representation not the appearance.

Revealing The Ink

When the Master revealed the second character with the candle’s heat, he showed the novice a fundamental truth. The text seen by the eye is merely a glyph, but the text seen by the machine is a sequence of code points.

To write robust code is to not be deceived by the visual form. One must apply the heat of normalization to reveal and align the true sequence of underlying code points. Only then can the comparison be honest and the measurement be true.

The Master asked the young cook “Where is the millet for the morning congee?”

The cook gestured to the open sack. “It is there Master where it has always been.”

The Master reached into the sack but pulled out only dust. “When the sack is tied and carried to the marketplace, the millet must travel with it. The sack itself is but a suggestion of grain.”

The cook lowered his eyes. “I forgot to check the sack.”

The Python Packaging Ecosystem

In Python projects, code is rarely enough on its own. Configuration files, database schemas, assets, and binaries often need to accompany it. These files are what allow code to function correctly in real environments.

When working directly from a source tree, all files are accessible through the filesystem. But when you build a package into a wheel or source distribution and install it elsewhere, only the files you explicitly include are carried forward. Any undeclared files vanish in the process.

The governing principle is the package manifest. If non-code files are required, you must declare them so the build system knows to include them. Without this, your project risks arriving at its destination empty-handed.

Part 1: The Initial Oversight

Suppose you begin with a project that reads a configuration file:

logic.py might try to access defaults.yaml:

This works only if you run the script from the project root. Change the working directory and it breaks. Reliance on the current directory is fragile.

Part 2: Editable Installs

During development, many use editable installs. Running:

pip install -e .

places a reference to your source tree in site-packages. Imports resolve against the live code, so changes take effect immediately without reinstalling.

This approach removes the need to run from the project root. But when you distribute the package, if defaults.yaml was never declared as package data, users installing it elsewhere will find that Python cannot locate the file.

Part 3: Using importlib.resources

The recommended way to access package data is importlib.resources (or its backport for older Python versions). It looks up resources relative to the package, independent of how or where it is installed.

This resolves the path issue. But it does not solve the packaging problem: if defaults.yaml is not bundled into the distribution, it still won’t be present in the installed package.

Part 4: Declaring Resources

To ensure resources travel with your package, you must declare them in your build configuration. By default, only Python source files are included.

Below are minimal pyproject.toml snippets showing how to include *.yaml files using setuptools and hatchling:

Setuptools

Hatchling

Both achieve the same effect: defaults.yaml is bundled into the wheel and available through importlib.resources.

Part 5: Namespace Packages

With implicit namespace packages (PEP 420), parts of a namespace can be spread across multiple distributions. Such packages omit an __init__.py in the top-level directory, enabling a flat layout as we learnt in Koan 7:

Here, mycompany is an implicit namespace shared by two subpackages. To ensure defaults.yaml travels with mycompany.analytics, declare it under that subpackage.

This guarantees that only mycompany.analytics carries defaults.yaml, while mycompany.ml remains unaffected.

Shipping The Millet

The package manifest is the measure of completeness. If you omit declarations, your distribution will carry only dust. When crafting Python packages, verify that every essential file (code or data) travels with it.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

In monasteries bowls are often used for meditation rituals. Some are empty, others hold offerings: coins, water, and small tokens of care. In Python, dataclasses behave much like these bowls.

Each dataclass instance is a bowl; the data it carries are the offerings inside. How Python evaluates equality and hashing depends on both the contents and the form of the bowl itself.

Part 1: What is a Dataclass?

Dataclasses, introduced in Python 3.7, are a convenience feature. They save you from writing boilerplate code when you need simple classes that hold data.

For example, without dataclasses you might write:

And with a dataclass it becomes:

Python automatically creates the __init__, __eq__, and __repr__ functions for you when you use the dataclass decorator.

Part 2: The Empty Bowl

Consider a bowl with no marked offerings:

Although values are assigned on the class, there are no annotated fields. When Python evaluates this bowl:

Only attributes declared with annotations are considered the recognized contents of the bowl, contributing to equality and hash calculations. Adding type annotations transforms the bowl:

Now the offerings are acknowledged, and hashing reflects their presence. Two bowls with identical annotated fields are equal, and their hashes correspond to their contents.

Part 2: Comparing Different Bowls

Next, imagine two bowls, each holding the same offerings:

The bowls may contain identical coins and water, yet they differ in form. Python compares both the contents and the type of the object:

Equality requires both the offerings and the vessel to match. Hashing, on the other hand, considers only the annotated fields.

Part 3: Fields vs Class Variables

The distinction between annotated fields and class variables is crucial. Only annotated fields are considered offerings within the bowl. Class variables exist on the outside: they do not contribute to equality or hashing.

Part 4: Mutable vs Frozen Dataclasses

You may have noticed that we are delcaring classes with frozen=True. Dataclasses can be created mutable or frozen:

• Mutable dataclasses allow their contents to change after creation. This is the default.

• Frozen dataclasses make the contents immutable, like sealing the offerings inside a bowl.

Frozenness also affects hashability. Only frozen dataclasses are hashable by default, because mutable bowls could change after being placed in a set or used as dict keys.

In Koan 8 we learnt that the hashability of a class is determined by the implementation of the __hash__ function. When a dataclass is declared frozen, Python automatically creates a __hash__ function for you.

Part 5: Default Values and Factories

Default values make bowls easier to create without specifying every offering:

For mutable fields like lists or dictionaries, a default factory is necessary. Using a default argument with a mutable object can lead to shared state between instances:

This is because default arguments are evaluated once at definition time, not execution time, as we learnt in Koan 3.

Instead, a default_factory creates a new object for each instance:

Now each bowl has its own independent list of offerings. Python calls the factory function at the time the instance is created, ensuring the contents of each bowl remain separate.

This distinction prevents subtle bugs and preserves the integrity of each bowl’s contents.

Offering the Bowl

A bowl without marked offerings is indistinguishable from another empty bowl of the same type. Two bowls may contain identical offerings, yet remain separate if their forms differ. Hashing depends only on the contents, while equality considers both the contents and the form.

Through careful attention to fields, mutability, defaults, and the auto-generated methods, dataclasses provide a simple, predictable structure for Python data objects.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

The Peril of Unquoted Arguments

We often have the need to run commands in shells1 using Python. Subprocess is a cross-platform2 python module which helps you do this.

Shells commands are separated by command delimiters (such as ‘;’ ‘&&’ ‘||’ and ‘\n’ in POSIX3). Argument delimiters on the other hand define how a single command’s arguments are split. On POSIX-compliant systems the IFS4 environment variable defines the characters used to split arguments. By default, IFS is set to split on spaces, newlines and tabs.

When a path or argument to a command contains a space, the shell does not see a single continuous entity. It sees two distinct things. A single path has become two arguments.

Let us begin with a simple example.

Part 1: The Space as a Separator

Consider the creation of a directory with a space in its name.

When we use the os module it understands the path as a single string because it does not involve the shell's interpretation. The directory is created as intended with a space in its name.

Now let us use the subprocess module with the shell=True flag. This flag instructs Python to pass our command to the shell for execution as a single string.

When this code runs, the shell sees the command as mkdir my new path. The shell interprets this as a command to create a directory named my a second directory named new and a third named path. The single path becomes three paths.

Part 2: The Command Injection

The danger is not only in misinterpretation but also in malicious injection.

Consider if the path came from an external source such as a user defined argument for example.

The shell sees mkdir my; rm -rf /. The semicolon is a command separator in the shell. The shell will first execute mkdir my and then it will execute rm -rf / which deletes the root directory.

The unquoted path has allowed the user to inject a new command. This is a profound and dangerous failure of boundary. A simple space or semicolon can shatter the integrity of the system.

Part 3: The Principle of Quoting

To prevent this we must use quoting. Quoting places a protective barrier around the string telling the shell to treat it as a single unit regardless of its contents.

The shell now sees mkdir "my; rm -rf /". The entire string is treated as a single argument for mkdir. No new directories are created. No commands are executed. The semicolon is rendered harmless a mere character within the string.

Part 4: The Path of Wisdom

The wise path is the one that avoids the shell entirely when not needed. Most linters (like ruff) will detect this for you.

Here we pass a list of arguments to subprocess.run. Python does not pass a single string to the shell. Instead it executes mkdir directly as a separate process and passes user_input as its first argument. The shell is never involved and the risk is eliminated.

This is the preferred way. It is clean and safe.

Part 5: A Real-World Command Injection (CVE-2024-9287)

Command Injection is not just a theoretical problem. Last year, a high severity vulnerability was found to affect all Python versions <= 3.13:

A vulnerability has been found in the CPython `venv` module and CLI where path names provided when creating a virtual environment were not quoted properly, allowing the creator to inject commands into virtual environment "activation" scripts (ie "source venv/bin/activate"). This means that attacker-controlled virtual environments are able to run commands when the virtual environment is activated. Virtual environments which are not created by an attacker or which aren't activated before being used (ie "./venv/bin/python") are not affected.

Previously, when a virtual environment was created, the activation scripts (activate, activate.bat, etc.) would use the environment name provided by the user to construct the venv path without enclosing it in quotation marks.

For example, a name like my test venv would be written into the script as /home/user/my test venv. An attacker could craft a virtual environment name like my-venv-with-space-and-command; malicious_command which would be interpreted by the shell as a path followed by an additional command to execute.

The fix5 (see summarized code below) was to ensure that all paths written into the venv activation scripts are properly quoted using the shlex module. The fix uses shlex.quote to ensure that any special characters or spaces in the path are escaped or enclosed in single quotes. This prevents the shell from misinterpreting the path as separate arguments or commands.

Forging the Blade

The Master Blacksmith’s lesson was simple. Just as a piece of metal may appear to be a single piece of steel, but splits into shards when struck by a hammer; unquoted command strings are split into different arguments or commands by the shell.

When constructing commands in Python:

• Avoid shell=True and use subprocess.run([…]) instead

• If you must use subprocess with shell=True, quote the command string

• When constructing shell commands outside of subprocess, use shlex to avoid command injection from untrusted input.

The blacksmith does not strike a piece of steel without thought. So too should the developer not pass a path to the shell without care.

Last week we discovered a new tool to inspect Python programs. This week, we shall use it to understand what is happening inside list comprehensions between Python 3.10 and Python 3.12.

A Gentle Reminder

The CPython virtual machine is a stack-based machine. This means it performs most of its operations by pushing and popping values onto a data structure called the stack.

dis.dis disassembles Python code into the individual bytecode instructions used by the interpreter to operate on the stack. For example:

• Instructions like LOAD_FAST or LOAD_CONST push values onto the stack.

• Instructions like BINARY_ADD or CALL_FUNCTION pop values from the stack to perform an operation and then push the result back.

This allows us to infer the state of the stack at each step.

Part 3: Python 3.10: The Old Way

Let’s start with a simple example: [x for x in range(5)]

When we observe the output, we can see two sections. The first refers to the outer code (or main program), but what about the second? This is actually a new hidden function that Python creates for the list comprehension. Let’s break down exactly what it does:

The Outer Code (Main Program)

This section shows what the main program does.

• 0 LOAD_CONST 0 (<code object <listcomp>...>): Python first loads the compiled code for the list comprehension itself. It's a separate, self-contained piece of code.

• 2 LOAD_CONST 1 ('<listcomp>'): Loads the name <listcomp> to identify the function.

• 4 MAKE_FUNCTION 0: This is the crucial step. This instruction creates a new “hidden” function object from the code object loaded in the first step.

• 6 LOAD_NAME 0 (range): Loads the built-in range function.

• 8 LOAD_CONST 2 (5): Loads the constant value 5.

• 10 CALL_FUNCTION 1: Calls the range function with the argument 5 to create a range object.

• 12 GET_ITER: Gets an iterator from the range(5) object.

• 14 CALL_FUNCTION 1: This calls the list comprehension function that was created earlier. The iterator from range(5) is passed to it as an argument.

• 16 RETURN_VALUE: The function returns None to indicate completion.

The Inner Code (The List Comprehension Function)

This is the actual "hidden" function that was created by Python to build the list. It's a separate, compiled block of code.

• 0 BUILD_LIST 0: Creates an empty list on the stack to hold the results of the comprehension.

• 2 LOAD_FAST 0 (.0): Loads the iterator that was passed into this function from the outer code. It's stored in a local variable named .0.

• >> 4 FOR_ITER 4 (to 14): This is the start of the loop. It iterates over the loaded iterator. If there are no more items, it jumps to instruction 14 to exit the loop.

• 6 STORE_FAST 1 (x): The FOR_ITER instruction gets the next item from the iterator and stores it in a local variable named x.

• 8 LOAD_FAST 1 (x): Loads the value of x back onto the stack.

• 10 LIST_APPEND 2: Appends the value on the top of the stack (which is x) to the list that was created in the first step. The 2 refers to the number of items it has to inspect on the stack.

• 12 JUMP_ABSOLUTE 2 (to 4): Jumps back to the top of the loop (FOR_ITER) to get the next item.

• >> 14 RETURN_VALUE: Once the loop is finished, this instruction returns the final list back to the outer code.

Now we can begin to understand why the code in the original Koan might have failed. We’re getting a NameError because the list comprehension is being run within a hidden function, and creating it’s own scope which doesn’t know about the river variable.

Part 4: Python 3.11: The New Way

But why don’t we get the NameError in Python 3.12?

Because the entire process happens within the main code block, without creating a hidden function. The process is:

• 2 PUSH_NULL: A NULL value is pushed to the stack. This is part of the new calling convention for functions in Python 3.11+.

• 4 LOAD_NAME 0 (range) and 6 LOAD_CONST 0 (5): The range function and the integer 3 are loaded onto the stack.

• 8 CALL 1: The range function is called with one argument, 5, returning a range(5) object.

• 16 GET_ITER: An iterator is created from the range(5) object.

• 18 LOAD_FAST_AND_CLEAR 0 (x): This is a key instruction for the new approach. It efficiently handles the "isolation" of the loop variable x. If a variable named x already exists in this scope, its value is saved to the stack, and the local variable x is set to NULL. This "pushes" the clashing local variable out of the way.

• 20 SWAP 2 and 22 BUILD_LIST 0: The BUILD_LIST instruction creates a new empty list. The SWAP instructions are used to organize the stack, ensuring the empty list and the iterator are in the correct positions for the loop to begin.

• 26 FOR_ITER 4 (to 38): The start of the loop. It gets the next item from the iterator and pushes it to the stack. If there are no more items, the loop ends, and execution jumps to line 38.

• 30 STORE_FAST 0 (x): The value from the iterator is stored into the local variable x. This is the loop variable.

• 32 LOAD_FAST 0 (x) and 34 LIST_APPEND 2: The value of x is loaded and then appended directly to the list.

• 36 JUMP_BACKWARD 6 (to 26): Jumps back to the top of the loop to get the next item.

• 38 END_FOR: The loop is finished.

• 40 SWAP 2 and 42 STORE_FAST 0 (x): This is the "pop" part of the stack manipulation. The SWAP and STORE_FAST instructions restore the original value of x that was saved in step 18.

• The remaining instructions handle printing the result and returning from the function.

This disassembly clearly shows that there is no separate code block for the list comprehension. The entire process, from creating the iterator to building the list and appending items, happens in one continuous flow of instructions.

The use of LOAD_FAST_AND_CLEAR and SWAP instructions handles the necessary variable isolation without the overhead of creating an entirely new function frame. This change, known as Inlined Comprehensions was introduced in Python 3.12 with PEP 709.

Part 5: The Two List Comprehensions

Now we can see exactly what happened in our original Koan. In Python 3.10:

• exec() does not have access to the local scope of the surrounding code.

• In the traceback, you can see a separate <listcomp> frame, which acts as a barrier.

• The exec("river*single_drop") statement tries to find the variable river in its own scope (which is the <listcomp> function's scope) and then the global scope. It cannot access river from the lambda's scope, leading to the NameError.

In Python 3.12 and later, where the list comprehension is inlined:

• The list comprehension runs in the same frame as the surrounding lambda function.

• exec() can now access the local variables of the surrounding scope, including the variable river.

• The code runs successfully, and since exec() returns None and the list comprehension produces a list containing None.

Part 6: Making it work on Python 3.10

There is in fact a way to stop the error from occurring in Python 3.10. the exec function accepts an optional dictionary of local and or global variables. So we can explicitly provide the variables missing in the list comprehension scope. And we can use the trick we learnt in Koan 5 to define and run the lambda in the one statement:

Drinking the water

Just as the water in one cup flowed through a sieve, and the other was sourced directly from the river; the difference between the two Python outputs is due to a change in how list comprehensions are implemented under the hood by CPython.

• In Python 3.10 list comprehension act as a separate, nested function.

• In Python 3.12 and later, the list comprehension is inlined.

While the two cups of water may appear identical, the methods used to obtain them are profoundly different.

List Comprehensions

List comprehensions are one of the most popular features in Python, and it allows us to write idiomatic (or Pythonic) code.

But, as the student discovered, although the end result may be the same in the above example, the process it took to arrive at the solution is very different.

In fact, in Python 3.12, the process used to implement list comprehension changed significantly.

Let us begin at the river.

Part 1: The Inner World of the For Loop

In Koan 4 we learnt that Python has 4 scopes (Local, Enclosing, Global and Builtin). However, there is also another “hidden” scope. Consider this loop:

The loop above reassigns the value of x in the enclosing scope. After the loop runs, x is no longer 10. Its value is now 2. The loop's variable x "leaked" into the surrounding code.

Now consider the same code implemented as a list comprehension:

This code prints 10. The x in the comprehension is a new x. It does not touch the outer x. The list comprehension creates its own hidden scope.

Part 2: Peering into the water

But how does Python ensure that list comprehension has a distinct scope? To observe what is happening, we must first gather some tools to help us peek inside the code.

When you run a Python script, the CPython interpreter performs two main steps:

1. Compilation: The source code (the .py file you wrote) is compiled into a stream of bytecode instructions. This bytecode is cached into a .pyc file (pyc stands for "Python Compiled").

2. Execution: The CPython virtual machine (a program written in C) reads and executes the bytecode instructions one by one. If the source file has changed, the bytecode is re-created, and if not, the cached .pyc file is run directly

Inspecting the bytecode can inform us about how the program is run by the interpreter. However, the bytecode is stored in an efficient binary format that is not human readable.

To help us, we can use two functions:

• compile() to compile a python source code string into a code object (also known as bytecode), with the following arguments:

  • filename: the filename from which the source code was read. If we are using the REPL, then it can be any string.

  • mode: use “exec” for multi-line statements, “eval” for single line statements, and “single” for single line interactive statements.

• dis.dis to disassemble the bytecode and print it in a readable format

The output of dis.dis typically consists of several columns:

• Line Number: This is the line number from the original Python source code. Note that a single line of Python code can compile into many bytecode instructions.

• Offset: This is the byte offset of the instruction within the bytecode stream. It's used by jump instructions to navigate the code.

• Instruction: This is the name of the bytecode instruction (e.g., LOAD_CONST, CALL_FUNCTION, RETURN_VALUE).

• Argument: This is a value or a reference used by the instruction. It might be an index into a table of constants, variables, or names.

• Argument Mnemonic: This provides a human-readable name for the argument, making it easier to understand what the instruction is doing (e.g., (range), (3), (None)).

We now have all the tools we need to peek inside the list comprehension, and to investigate why Python < 3.12 and Python >= 3.12, behave so differently.

Until next time, may your stillness reveal the unseen flow.

Continue to Part 2

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

The Hidden Teaching: The Flow of Assignment

Just as a master potter knows that some steps must precede others regardless of the order they’re written, a seasoned Python developer understands that a seemingly simple assignment is not a singular event. It’s a flow. The value does not “flow” from left to right. Python evaluates the right-hand side first and binds that result to each target from left to right.

Part 1: Assignment Chains

Consider a chain of assignments:

This may appear to assign x + 2 to x, y, and z simultaneously. But Python does not work that way. The expression x + 2 is evaluated once, producing the value 12, which is then sequentially bound to z, y, and x. Assignment in Python is a statement, not an expression, so it does not return a value during this process.

Part 2: Identity and Comparisons

Consider the following assignment chain:

list2 = list3 = list1 = [1,2,3]

On the surface, this appears simple. It creates a single list object [1, 2, 3] and binds list1, list3, and list2 to that same object.

This is a shared reference, not a copy.

Modifying one of these variables will affect all others. We can confirm this by checking the identity of the objects using the id() function.

Part 3: Comparison Chains

The same principle of sharing values does not apply to chained comparison operators such as <, > and ==.

1 < x < 10

This expression is not evaluated from right to left. Instead, the interpreter performs a series of distinct comparisons and joins them with an implicit and operator.

The expression 1 < x < 10 is not shorthand for a single operation. It is shorthand for two separate operations:

(1 < x) and (x < 10)

The interpreter is smart enough to perform the minimal number of checks. If 1 < x is False, the expression x < 10 is not evaluated at all because the entire and expression will be False. This is known as short-circuiting and you can use this to your advantage when writing logical expressions:

(high_probability_of_being_false) and (low_probability_of_being_false)

If you write the expression with the higher likelihood of being False first, then you avoid the second expression from being executed. This is something you might have to consider when writing performance sensitive code.

Part 4: Chained Comparisons with Side Effects

What about chained comparisons containing functions with side effects? While Python evaluates a < b < c as (a < b) and (b < c), what happens if b is the result of a function call? Consider the following example:

The output will be Getting B's value... followed by The truth holds. The function get_val_b is called once and its returned value 5 is used for both 1 < 5 and 5 < 10.

The comparison is evaluated as a logical and, but the function call that produces the shared value for both comparisons is only executed once.

Firing the pot

The master potter’s lesson was simple. The order in which instructions are given are not the order in which they must be executed. In Python, chained instructions are executed according to the following rules:

• Assignment chains evaluate the right-hand side once and bind that result sequentially to each target.

• Comparison chains are logical combinations of individual comparisons.

When you chain operators you are creating a silent understanding of how values will flow and how logic will be judged.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

When we speak of hashing in Python, we are entering a realm where identity, equality, and performance converge. In Koan 2, we learnt the difference between Identity and Equality. Hashing in Python uses equality rather than identity. It promises that two objects that are equal will share the same mark.

But the promise is not perfect.

The koan shows us two NaNs ("Not a Number"). They cannot be equal to each other, by mathematical rule. Yet their hashes are equal. How can two things be marked the same, yet not be the same?

Let us examine the seals carefully.

Part 1: What is a Hash?

A hash is a value produced by a function designed to uniquely represent an object. In Python, the built-in hash() function returns an integer, and it is used by dictionaries and sets to test membership. When you do:

Python does not search through all the keys each time. Instead, it computes a hash of "key", uses it to jump to the right location in memory, and then checks equality to confirm.

So the rule is:

1. If a == b, then hash(a) == hash(b).

2. But the reverse is not guaranteed: if hash(a) == hash(b), a and b may not be equal.

The first rule is what allows dictionaries and sets to function correctly.

Part 2: The Strange Case of NaN

Mathematically, NaN is defined so that:

float('nan') == float('nan')  # False

float accepts the strings “nan” and “inf” with an optional prefix “+” or “-” for Not a Number (NaN) and positive or negative infinity. (from Python Docs)

This is deliberate: NaN is not equal to anything, not even itself.

Yet Python ensures:

hash(float('nan')) == hash(float('nan'))

Why? Because hash-based collections require a consistent contract: objects that are equal must share the same hash, but Python does not forbid unequal objects from sharing a hash.

Part 3: Hashing and Equality in User-Defined Classes

When you define your own class, you may decide what equality and hashing mean.

Now two scrolls with the same text are considered equal, and they share the same hash:

If you override __eq__ but forget to also override __hash__, Python will often make your object unhashable:

If you define them inconsistently like returning different hashes for equal objects, your code may behave strangely or inefficiently.

These are the guidelines for implementation from the python docs:

• If a class does not define an __eq__() method it should not define a __hash__() operation either

• if it defines __eq__() but not __hash__(), its instances will not be usable as items in hashable collections.

• If a class defines mutable objects and implements an __eq__() method, it should not implement __hash__(), since the implementation of hashable collections requires that a key’s hash value is immutable (if the object’s hash value changes, it will be in the wrong hash bucket).

Part 4: Other Gotchas

4.1 Equality Across Types

Python strives for consistency across related types:

print(1 == 1.0)      # True
print(hash(1) == hash(1.0))  # True

Despite being different types, the value of 1 and 1.0 are equal, so their hashes are also equal.

d = {1: "integer"}
d[1.0]   # "integer"

At first glance this is elegant, but it means different numeric types may collapse into the same bucket, even if their meanings diverge in your domain. In scientific or financial code, this subtle equality can introduce surprising behaviour if you intended to distinguish int from float.

4.2 Immutable vs Mutable

Hashes are meant to be stable. If the contents of an object could change after being placed in a dictionary or set, the mapping would fall apart.

This is why:

• Immutable containers (like tuples) are hashable — but only if all their contents are hashable.

• Mutable containers (like lists and dicts) are not hashable.

hash((1, 2, (3, 4)))  # Works
hash((1, [2, 3]))     # TypeError

The error is not arbitrary. It protects you from placing an unstable key into a dictionary. Imagine changing the list inside a tuple after it has been used as a key — the hash would no longer reflect the object’s state.

The Final Seal

The novice saw two NaNs and said: "The seals are the same, so the things must be the same." The master corrected him: "The ink is the same, but the seal is broken."

Hashing is a symbol, not a proof. Equality and hashing may touch, but they are not bound. To see clearly, we must hold both the symbol, and the substance it points to.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the previous issue here:

The Hidden Teaching: The Nature of the Path

The student believes that a village must be a single, bounded place. But the master knows that a village can be a concept, defined by its connections and the paths that lead to it.

In Python, the import system follows a similar principle. You may think of packages as walled villages with a clear boundary. Yet the true nature of the path (sys.path) allows for the creation of unburdened villages: namespace packages, where a single logical package is assembled from components scattered across the filesystem.

Part 1: The Walled Village and the Path

The Python interpreter, when asked to import a package, follows a simple rule: it searches along a predefined path. This path, known as sys.path, is a list of directories.

A traditional package, or a "walled village," is a directory containing an __init__.py file. This file marks the directory as a package and provides its clear boundary. When you import this package, Python finds the directory on its path and all its contents are then accessible.

Consider a simple structure:

To use this, the directory containing my_village must be on sys.path. If you run Python from the directory above my_village, it is on the path by default.

The __init__.py is the village gate; it tells the interpreter, "This is the start of the journey."

Part 2: The Unseen Path and the Root of Confusion

The monk's confusion was not about the village itself, but about the assumption of a single starting point. The same confusion arises when a package is not directly on the path.

Consider the following:

If you run Python from a location that contains both project1 and project2, the my_village directories are not directly on sys.path. The path only contains the top-level directory.

The interpreter's path is not a tangled vine that seeks out all nested directories. It is a straight road, and it only knows about project1 and project2. It does not know to look inside them for other packages.

Part 3: The Unburdened Village: A Shared Destination

The solution lies in understanding the path as a shared space. A namespace package is created when multiple directories with the same name, but without an __init__.py file, are all placed on sys.path.

By adding the parent directories (project1 and project2) to the path, you instruct Python to consider them as valid starting points for imports.

The interpreter first looks on the path, finds project1, and sees a my_village directory inside. It then continues its search, finds project2, and sees another my_village directory. It then merges these two directories into a single logical package, the "unburdened village."

Part 4: The Path of the Confused Traveller

What if one part of the village has a wall and the other does not?

Consider if project1's my_village directory contains an __init__.py file, but project2's does not.

In this scenario, my_village is no longer a namespace package. Python's import system will find the first my_village directory on its search path (sys.path) that contains an __init__.py file, and it will treat that directory as the entire package.

When you run import my_village.temple, the interpreter will find the my_village directory in project1, but it will not continue to search for other my_village directories. It will only look for temple.py inside project1/my_village. The file will not be found, and you will get a ModuleNotFoundError.

The presence of the __init__.py file effectively "closes the gate" on the search for other parts of the package, turning it back into a traditional, "walled" package.

Part 5: The Power of the Unburdened Village

Why would one choose to build a village without walls?

• Extensibility: Frameworks like pytest and zope use this pattern. The core library defines a namespace, for example my_app.plugins. Developers can then create their own packages, like my_app.plugins.my_feature, and the core application will automatically find and load them without any modification to the main codebase.

• Decomposition: A large project can be broken into smaller, independent libraries that are developed and versioned separately but still function as a single logical package.

• Modularity: It allows for a more distributed and modular architecture, where a project's components can be managed and installed from different locations.

Leaving the Village

The master's reply was a lesson in perspective. The village is not the directory itself but the concept of a shared space. The path is not a single location but a collection of all the roads one can travel.

Your confusion, like the monk's, arose from seeking a single physical truth. But Python's import system, like the master’s village, shows us that meaning can be constructed from disparate parts as long as you follow the right path.

In Python, a function is more than a list of commands, as we learnt in Koan 5. When a function is defined inside another, it becomes a closure, a small piece of code that remembers its parent's environment. This memory is powerful, but it's not a picture of the past. It's more like a window. It shows you the world as it is right now, not as it was when the window was first opened. This phenomenon is known as late binding.

Let's begin by observing the simplest expression of a closure.

Here, execute_commission is the closure. It remembers the item variable from its parent function, make_commission. When we call draw_sun(), it returns "sun" as expected. The closure has a clear, singular memory of its purpose.

Part 2: The Master's Three Commands

Now, let's observe the lazy calligrapher's technique, as described in the koan. He waits until all instructions have been given before beginning his work.

One might expect the output to be "sun," "moon," and then "cloud." However, the code will print "cloud" three times.

This is the very essence of late binding. The execute_commission function doesn't capture the value of instruction at the time it's created. Instead, it holds a reference to the variable instruction itself. By the time we finally get around to executing the first function in the list, the for loop has already completed, and the variable instruction has a final value: "cloud."

All three closures refer to the same single instruction variable in the outer scope, which has finished its journey.

Part 3: The Diligent Calligrapher

To correct this behavior, we need a way for each closure to capture its own unique copy of the instruction at the moment it's created. We must teach the calligrapher to be diligent and remember each command individually.

One method to achieve this is by using a default argument. As we learnt in Koan 3, default arguments are evaluated once, at the time the function is defined. We can modify our make_commissions function to use this technique.

This time, the output is as we first intended. By setting the default argument item=instruction, we force Python to evaluate instruction and bind its current value to the item parameter for each new function. Each closure now holds its own unique copy of the instruction, rather than sharing a single reference.

Part 4: The Anonymous Calligrapher

This principle of late binding also applies to lambda functions. As we learnt in Koan 5, lambdas are simply anonymous functions. The "lazy calligrapher" effect is very common when lambdas are created in a loop.

Here is the original problem, written with a lambda:

Just like with the def-defined function, this will also return "cloud," because the lambda closes over the instruction variable, not its value.

To correct this, we use a default argument for the lambda function:

We can make this more concise and pythonic by using list comprehension to create all the commissions in a single stroke.

Part 5: The Master's Scrupulous Note

What if we could not use a default argument? Is there another way to force the calligrapher to capture the instruction immediately? The solution lies in creating a new scope for each instruction, forcing each closure to "listen" to a unique voice.

We can achieve this by creating a factory function that takes the instruction as an argument, thus creating a new, isolated environment for each closure.

This code also yields the correct output. On each iteration of the loop, we call create_executor(instruction). This function is given the current value of instruction. It then creates a new closure, execute_commission, which closes over item. Because item is a local variable within create_executor, its value is isolated from the outer loop's changes (as we learnt in Koan 4 about Python’s LEGB rule). Each call to create_executor creates a new, private memory for the closure it returns. It is as if the master wrote a small, specific note for each drawing, which the calligrapher could not forget.

Part 6: The Customized Brush

There are other, more concise or functional ways to achieve the same result. The choice of method often comes down to the clarity and style you prefer. We can use functools.partial to "pre-bind" the argument to a function. This is like preparing a special brush for each task, with the instruction already etched into its handle.

The Final Brushstroke

The Master's teaching was simple: the calligrapher did not remember what was once said, only the command that was last spoken when the drawing finally began.

In your code, your functions are like the forgetful calligrapher. The closure does not remember the values the variable held when it was created, only the last value is recalled when the function is run.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

Functions as First-Class Citizens in Python

In Python, a function is much like a harp. An object that holds potential, a set of instructions waiting to be invoked and produce a harmonious result. This design choice makes functions "first-class citizens," opening up a world of elegant and powerful programming paradigms.

Let's begin with this fundamental understanding.

Part 1: The Harp’s Existence

What does it mean for a function to be a "first-class citizen"? It means that functions can be:

• Assigned to variables.

• Passed as arguments to other functions.

• Returned as values from other functions.

• Stored in data structures (lists, dictionaries, etc.).

This is distinct from languages where functions are merely blocks of code that can be called, but not manipulated like other data.

Consider a simple instruction for making music with our harp:

def play_note(note):
    return f"The harp plays the note: {note}"

In Python, play_note isn't just a command; it's an object. You can prove this by assigning it to another variable, much like one might refer to the harp by a different name in an orchestra:

my_instrument = play_note
print(my_instrument("C major"))

Here, my_instrument now refers to the same function object as play_note. The harp exists, and we have a way to access its potential for sound, even if no note has yet been played.

Part 2: Passing the Harp to the Musician (Higher-Order Functions)

One of the most powerful implications of functions being first-class is the ability to pass them as arguments to other functions. Functions that accept other functions as arguments are known as "higher-order functions." This is akin to handing our harp to a skilled musician, trusting them to play its strings when the moment is right.

A common example is the map() function, which applies a given function to all items in an input list and returns an iterator. Let's imagine we have a sequence of musical ideas, and we want to apply a transformation to each using our harp's capabilities:

def harmonize_note(x):
    return f"{x} with harmony"

melody_notes = ["C", "G", "Am", "F"]
harmonized_melody = list(map(harmonize_note, melody_notes))
print(harmonized_melody)

Here, harmonize_note is passed as an argument to map. map doesn't know what harmonize_note does, only that it's a callable object it can apply to each element. It simply knows how to take a series of musical ideas and apply the "play and harmonize" instruction to each.

Part 3: The Unnamed Chord (Lambdas)

For simple, one-off musical expressions, Python offers "lambda expressions." These are small, anonymous functions that can be defined in a single line. They are particularly useful when you need a function object for a short period, often as an argument to a higher-order function, like a quick flourish instead of a formal composition.

Our koan's core, lambda x: x + 1, is an example of an anonymous function. It exists, it holds potential, but it doesn't have a formal name.

# Using a lambda with map to transpose notes:
base_frequencies = [440, 494, 523] # A4, B4, C5 frequencies
transposed_frequencies = list(map(lambda freq: freq + 50, base_frequencies))
print(transposed_frequencies)

Lambdas are succinct and often improve readability for simple operations, avoiding the need to define a full def function for a trivial task. They are like a quickly improvised chord on the harp.

Part 4: Plucking the String Immediately (Immediate Invocation)

Just as we can assign a lambda to a variable and then play it, we can also play it immediately after it's defined.

Consider our simple lambda from the koan:

lambda x: x + 1

This defines a function object. It is the silent harp string. To make it produce a result, we need to "pluck" it by providing an argument. We typically do this by assigning it to a variable first, then calling that variable:

increment_frequency = lambda x: x + 1
result = increment_frequency(10)
print(result)

However, because the lambda expression itself evaluates to a function object, you can immediately apply arguments to it, just like you would with any other function reference.

The koan's code demonstrates this:

(lambda x: x + 1)(10)

Here, (lambda x: x + 1) creates the anonymous function object (the harp string), and (10) immediately calls that function with the argument 10 (plucks the string). The parentheses around the lambda expression are crucial here to ensure the lambda is fully defined before the invocation operator () attempts to call it. Without them, Python would expect a function name to call.

This pattern, sometimes seen as an Immediately Invoked Function Expression (IIFE), is less common in everyday Python than in JavaScript, but it is perfectly valid and can be seen in specialized contexts, often for creating self-contained scopes or for quick, single-use computations without cluttering the namespace.

Closing the Circle

The silent harp holds all its potential for music within. Only when a string is plucked does its melody emerge. And sometimes, the note is played the very moment the string is tied.

In Python, functions are not just commands; they are entities with identity, capable of being manipulated, stored, and passed around like any other piece of data.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

Variable Scope and the Shadows They Cast

In Python, variables are not always where they appear to be.
When a function runs, it carries with it not just its own code, but the shadow of names it has seen, names that belong to outer scopes.

To understand this, we must learn to see how Python resolves names.

Let us begin simply.

Part 1: The Local Glow

The simplest form of scope is the local scope. When you define a variable inside a function, it exists only within that function's execution.

def greet():
    message = "Hello, world!"
    print(message)

greet()
print(message)  # This would cause an error: NameError

Here, message is born and dies within the greet function. It's like a lamp lit only inside a small room; its light doesn't extend beyond the doorway.

Part 2: The Enclosing Room

Now, let's open another door. Python has a concept called enclosing scope. This applies to nested functions. If a variable isn't found in the immediate local scope, Python looks outwards to any enclosing functions.

Consider this:

def outer_function():
    outer_message = "From the outer room."

    def inner_function():
        print(outer_message)
    
    inner_function()

outer_function()

When inner_function is called, it first looks for outer_message within its own scope. It doesn't find it. So, it looks in the scope of outer_function, where outer_message resides. This works. The inner function can see the variables of its enclosing function, like seeing the light of a lamp in an adjacent room through an open door.

Part 3: The Global Stage

Beyond local and enclosing scopes, there is the global scope. Variables defined at the top level of a script or module are global. They can be accessed from anywhere within that module.

global_message = "From the wide world."

def display_global():
    print(global_message)

display_global()

Here, display_global can access global_message because it's in the global scope. This is like the sun's light, visible from every room.

Part 4: The Built-in Universe

Finally, there's the built-in scope. This contains all the names that Python pre-defines, such as print, len, str, True, False, and None. These are always available.

The order in which Python searches for names is known as the LEGB rule:

1. Local (current function)

2. Enclosing function locals (from inner to outer enclosing functions)

3. Global (top-level of the module)

4. Built-in (Python's pre-defined names)

Python stops at the first place it finds the name.

Part 5: When Shadows Deceive - Variable Binding

The true complexity arises when you assign to a variable within a deeper scope. When Python encounters an assignment statement, it assumes you intend to create or modify a variable in the current scope, unless explicitly told otherwise.

Let's revisit our koan's example:

shadow = 10  # Global shadow

def outer_lamp():
    shadow = 20  # This 'shadow' is local to outer_lamp()
    def inner_lamp():
        print(shadow)  # This 'shadow' refers to outer_lamp()'s shadow
    return inner_lamp

lamp = outer_lamp()
lamp() # Output: 20

Here's the crucial part:

1. shadow = 10 establishes a global shadow.

2. Inside outer_lamp(), shadow = 20 creates a new, local variable within outer_lamp's scope. This shadow is entirely separate from the global shadow. It does not modify the global shadow.

3. Inside inner_lamp(), when print(shadow) is called, Python searches for shadow using the LEGB rule. It finds shadow = 20 in its enclosing scope (outer_lamp), and that's the shadow it uses.

The lamp() call, which executes inner_lamp(), thus prints 20. The global shadow (which remains 10) is untouched, and the shadow in outer_lamp() casts its own shadow, independent of the global lamp.

Part 6: The Illusion of Locality

Now, a subtle twist:

shadow = "global"

def inner():
    print(shadow)
    shadow = "local"

inner()

What happens here?

Python raises an error:

UnboundLocalError: cannot access local variable 'shadow' before assignment

Why?

Because Python sees the assignment shadow = "local" and assumes that shadow must be local to inner. It does not look outside anymore. So when you try to read shadow before assigning it, Python is confused. There is a local shadow, but it has no value yet.

In Python, any assignment to a variable within a function makes that variable local to that function, unless explicitly declared otherwise.

This leads us to the next teaching.

Part 7: Declaring Intent – global and nonlocal

If we wish to use or modify a variable from an outer scope, we must declare our intent.

Using global:

shadow = 5

def modify():
    global shadow
    shadow = 10

modify()
print(shadow)  # 10

Here, global shadow tells Python: “I mean the shadow from the module’s top level.”

When working with nested functions, and using nonlocal:

def outer():
    shadow = 5
    def inner():
        nonlocal shadow
        shadow = 10
    inner()
    print(shadow)

outer()  # 10

Without nonlocal, the assignment would create a new local shadow inside inner.
With it, Python understands: use the variable from the enclosing function.

Extinguishing The Lamp

The Master's second lamp illuminated the truth: each lamp casts its own shadow. Similarly, in Python, each scope can define its own variables, creating distinct 'lamps' of variables. Without explicit instruction (like nonlocal or global), an assignment always defaults to the current, innermost scope, safeguarding higher-level variables from unintended modification.

Understanding scope is not just about avoiding errors; it's about designing clear, predictable, and maintainable code. It's about knowing where your variables truly reside, and how their light extends, or does not extend, into the surrounding code.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

Mutable Default Arguments and the Echoes of the Past

In Python, a function is not just a block of code, it is an object, alive with memory. And sometimes, that memory is louder than you expect.

We expect that every time we call ring(), it will create a new list and append "clang". But instead, each call remembers the last. Why?

Part 1: When Are Defaults Evaluated?

In Python, default arguments are evaluated only once — at the time the function is defined, not each time it is called.

So in:

def ring(bell=[]):

Python creates a new empty list once and binds it to the bell parameter’s default. All future calls to ring() without a bell argument will use that same list.

Thus:

first = ring()
second = ring()
third = ring()

print(first is second is third)  # True

They are all the same object.

This behavior often surprises beginners. It feels as if Python is remembering things it should forget. But it is not magic. It is memory: persistent, and shared.

Part 2: Mutable vs Immutable Defaults: A Gentle Contrast

The issue arises only with mutable default values, like lists or dictionaries. What if we used an immutable default, such as a tuple?

Consider:

def ring(bell=()):
   bell += ("clang",)
   return bell

What do you expect?

ring() # ('clang',)
ring() # ('clang',)
ring() # ('clang',)

Each call gives us a fresh result. There is no accumulation. No echo.

Why?

Because tuples are immutable. The += operator cannot modify the existing tuple. Instead, it creates a new one. In Python, the id() function gives you the memory address of an object. You can use this to test if the objects are the same, but beware of the CPython caching (also known as interning) we learnt about last week. To avoid CPython interning, we need to assign the result to a new object:

import time

def ring(bell=()):
    bell += (str(time.time()),)
    return bell

sound1 = ring(); print(id(sound1)) # 135080430662176
sound2 = ring(); print(id(sound2)) # 135080430179360
sound3 = ring(); print(id(sound3)) # 135080430181856

Want to explore how interning works in more detail?

I’m starting a new companion series where I explore advanced Python techniques in more detail and how they are actually used in open-source projects. It’s called “Python In The Wild”, and you can subscribe here to receive the first post when it comes out:

Python In The Wild

Explore how Python is used in the wild, through real-world examples from Open-Source projects

By Vivis Dev

Each time, Python builds a new object and returns it. The original default remains untouched.

This is why only mutable default arguments pose a problem. It is not the default itself that is dangerous, it is the possibility of mutation.

When an object can change, and you reuse that object across calls, you risk unintended persistence.

Part 3: The Right Way: Use None as a Sentinel

The conventional solution is this:

def ring(bell=None):
    if bell is None:
        bell = []
    bell.append("clang")
    return bell

Now:

ring() # ['clang']
ring() # ['clang']

Each call receives a fresh list.

Why use None?

Because None is immutable and unique. It makes an excellent sentinel, a marker that tells us, “No argument was provided.”

If you use this pattern, your functions will behave predictably. The echoes will fade when they should.

Part 4: When Might You Use Mutable Defaults?

There are rare times when shared mutable state is intended. For example, a function that memoizes its own results:

def factorial(n, cache={0: 1}):
    if n not in cache:
       cache[n] = n * factorial(n - 1)
    return cache[n]

Here, the default dictionary acts as a persistent memory. But this is a deliberate choice, not an accident.

If you do this, document it clearly. Most of the time, such patterns are better handled with decorators or external caches.

Shared state is not inherently wrong, but it must be a conscious design, not an accidental side effect.

Best Practices: The Bellmaker’s Notes

• Avoid using mutable objects like lists or dicts as default arguments.
  They persist across calls and can lead to surprising behavior.

• Use None as a default when you want a fresh object each time.
  Inside the function, create a new object only when needed.

• Immutable defaults (like None, 0, '', or ()) are always safe.
  Operations like += on them return new objects, leaving the default unchanged.

• If you must use a mutable default, document the behavior clearly.
  Treat it as shared state and ensure your design requires it.

• When debugging, use id() or logging to confirm whether the same object is reused, but beware of interning which can mislead you.
  If your function “remembers” things, ask yourself: “Did I mean to ring the same bell again?”

Closing the Circle

The novice asked, “Why does the bell grow louder each time I call for it?”

The master replied,

“Because you never asked for a new bell.”

In Python, if a default argument is mutable, it will grow, echo, and persist across calls.

To write clean Python is to know which bells echo, and when to ring a fresh one.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the last issue here:

"is" vs. "==": The Subtle Art of Identity and Equality in Python

When learning Python, it's common to encounter subtle distinctions that don’t make much sense at first. One of these is the difference between is and ==. At a glance, they both seem to answer the same question: "Are these things the same?" But in truth, they answer very different questions.

Let us begin at the beginning.

Part 1: The Two Questions

Python allows you to ask:

• Do these two values have the same contents?
  You use == to ask this question.

• Do these two values refer to the same object in memory?
  You use is to ask this question.

In simple terms:

• == asks, "Do they look the same?"

• is asks, "Are they the same being?"

Let’s look at a few examples:

a = [1, 2, 3]
b = [1, 2, 3]

print(a == b) # True (same contents)
print(a is b) # False (different objects)

At first, this might feel strange. If they’re both [1, 2, 3], why wouldn’t they be the same?

Because Python has created two separate lists in memory. They hold the same values, but they are not the same object. Like two scrolls with identical text, one copied from the other. If you edit one, the other remains unchanged.

This is the heart of the difference between equality and identity.

Part 2: Why Use is At All?

Why not always use ==?

The is operator is not about equality of content, it is about identity. There are cases where identity matters deeply. One of them is checking for a sentinel object, such as None.

if user_input is None:
    print("No input provided.")

Here, you're not asking "Does the input look like None?", but "Is the input literally the None object?"

This distinction ensures clarity and precision when writing code that interacts with unique markers or singleton objects.

Part 3: The Subtle Trap of Immutable Types

Now let’s tread into trickier ground. The behavior of immutable types, such as integers, strings, and tuples.

Consider:

a = 1000
b = 1000

print(a == b) # True 
print(a is b) # False (on most systems)

But then:

x = 5
y = 5

print(x == y) # True 
print(x is y) # True

Why the inconsistency?

This is due to an implementation detail of CPython (the most widely used Python interpreter). For performance, CPython caches small integers and some strings. So small values may end up pointing to the same object. But this is not something you should rely on. It's a quirk, not a guarantee.

In other words: the student sees that 1 is 1, and assumes the same for 1000 is 1000. But the truth changes subtly with scale, like mist lifting to reveal two paths instead of one.

Part 4: Containers and Identity

Let’s go a step deeper. What about empty containers?

print([] == []) # True - same contents (both empty)
print([] is []) # False - distinct list objects

Python creates a new list each time you write []. They are separate, though they appear the same. Like the student’s scrolls, word-for-word identical, yet changes to one do not touch the other.

And tuples?

print( () == () ) # True 
print( () is () ) # True (usually)

Here again, we encounter CPython’s caching. Because () is immutable and common, Python may reuse the same object. But this behavior is not mandated by the Python language specification, it is an implementation detail.

From Python 3.8 onwards, certain comparisons like () is () even raise a SyntaxWarning, gently nudging the student to rethink their assumption.

Best Practices

1. Use == when comparing values.
   You’re asking, do these two things hold the same meaning or content?

2. Use is when checking for identity, especially for singletons like None, True, False, or a sentinel object you define.

3. Avoid using is with literals or immutable values, unless you have a specific reason to care about identity, and even then, document that reason.

Final Reflection

The master’s lesson was simple:
Two scrolls may appear the same. But when you change one, the truth of their separation is revealed.

In Python, == and is teach us to look beneath the surface. Equality is about appearance. Identity is about essence. And understanding this difference is not just a technical detail, it is a practice in seeing things as they truly are.

New here? Python Koans is a Python newsletter that helps you learn Python through short puzzles. Catch up on the first issue here: