test

test_nb

async def test_nb(
    fn, # file name of notebook to test
    skip_flags:NoneType=None, # list of flags marking cells to skip
    force_flags:NoneType=None, # list of flags marking cells to always run
    do_print:bool=False, # print completion?
    showerr:bool=True, # print errors to stderr?
    basepath:NoneType=None, # path to add to sys.path
    verbose:bool=False, # stream stdout/stderr from cells to console?
    save:bool=False, # write outputs back to notebook on success?
    profile:bool=None, # load the IPython profile, as `ipykernel` does? (default: `exec_profile` config key)
):

Execute tests in notebook in fn except those with skip_flags

test_nb is async. parallel workers are fresh loopless processes, so each enters through asyncio.run at its process boundary, which keeps cells on the worker’s main thread (where SIGALRM timeouts can arm). run_sync is reserved for sync callers already inside a running loop, like exec_show_docs under the sync processor pipeline.

test_nb can test a notebook, and skip over certain flags. A notebook whose frontmatter sets skip_exec: true (e.g. as a - skip_exec: true list item in its title cell) is skipped entirely and reported as passing; use it for notebooks that can’t run under test at all, such as those needing credentials or live services:

_nb = Path('../../tests/directives.ipynb')
success,duration = await test_nb(_nb, skip_flags=['notest'])
assert success

In that notebook the cell flagged notest raises an exception, which will be returned as a bool:

_nb = Path('../../tests/directives.ipynb')
success,duration = await test_nb(_nb, showerr=False)
assert not success

import tempfile
from fastcore.xtras import modified_env

test_nb loads the IPython profile by default, like ipykernel does, so notebooks are tested the way their author’s kernel ran them (startup files, extensions, shell config). Set exec_profile = false under [tool.nbdev], or pass profile=False, to run without it:

with tempfile.TemporaryDirectory() as td:
    td = Path(td)
    (td/'profile_default'/'startup').mkdir(parents=True)
    (td/'profile_default'/'startup'/'00.py').write_text('prof_x = 7\n')
    nbp = td/'prof.ipynb'
    write_nb(new_nb([mk_cell('assert prof_x==7')]), nbp)
    with modified_env(IPYTHONDIR=str(td)):
        assert     (await test_nb(nbp, showerr=False))[0]
        assert not (await test_nb(nbp, showerr=False, profile=False))[0]

Sometimes you may wish to override one or more of the skip_flags, in which case you can use the argument force_flags which will remove the appropriate tag(s) from skip_flags. This is useful because skip_flags are meant to be set in the tst_flags field of [tool.nbdev] in pyproject.toml, whereas force_flags are usually passed in by the user.

source

nbdev_test

def nbdev_test(
    path:str=None, # A notebook name or glob to test
    flags:str='', # Space separated list of test flags to run that are normally ignored
    n_workers:int=None, # Number of workers
    timing:bool=False, # Time each notebook to see which are slow
    do_print:bool=False, # Print start and end of each notebook
    pause:float=0.01, # Pause time (in seconds) between notebooks to avoid race conditions
    ignore_fname:str='.notest', # Filename that will result in siblings being ignored
    verbose:bool=False, # Print stdout/stderr from notebook cells?
    save:bool=False, # Write outputs back to notebooks on success?
    symlinks:bool=False, # Follow symlinks?
    file_glob:str='*.ipynb', # Only include files matching glob
    file_re:str=None, # Only include files matching regex
    folder_re:str=None, # Only enter folders matching regex
    skip_file_glob:str=None, # Skip files matching glob
    skip_file_re:str='^[_.]', # Skip files matching regex
    skip_folder_re:str='^[_.]', # Skip folders matching regex
):

Test in parallel notebooks matching path, passing along flags

nbdev_test(n_workers=0)

Success.

You can even run nbdev-test in non nbdev projects, for example, you can test an individual notebook like so:

nbdev-test --path ../../tests/minimal.ipynb --do_print

Or you can test an entire directory of notebooks filtered for only those that match a regular expression:

nbdev-test --path ../../tests --file_re '.*test.ipynb' --do_print