Data Formats and Serialization

Relevant source files

• Doc/c-api/hash.rst
• Doc/deprecations/pending-removal-in-3.17.rst
• Doc/library/archiving.rst
• Doc/library/base64.rst
• Doc/library/binascii.rst
• Doc/library/cmdline.rst
• Doc/library/compression.rst
• Doc/library/compression.zstd.rst
• Doc/library/doctest.rst
• Doc/library/email.compat32-message.rst
• Doc/library/ensurepip.rst
• Doc/library/gzip.rst
• Doc/library/json.rst
• Doc/library/pickle.rst
• Doc/library/pyexpat.rst
• Doc/library/shutil.rst
• Doc/library/tarfile.rst
• Doc/library/webbrowser.rst
• Doc/library/xml.etree.elementtree.rst
• Doc/library/zipfile.rst
• Doc/library/zlib.rst
• Doc/whatsnew/3.11.rst
• Include/internal/pycore_global_objects_fini_generated.h
• Include/internal/pycore_global_strings.h
• Include/internal/pycore_pyhash.h
• Include/internal/pycore_pystate.h
• Include/internal/pycore_pythonrun.h
• Include/internal/pycore_runtime_init_generated.h
• Include/internal/pycore_unicodeobject_generated.h
• Include/pyexpat.h
• Include/pyport.h
• InternalDocs/asyncio.md
• Lib/base64.py
• Lib/compression/zstd/__init__.py
• Lib/compression/zstd/_zstdfile.py
• Lib/gzip.py
• Lib/json/__init__.py
• Lib/json/decoder.py
• Lib/json/encoder.py
• Lib/json/scanner.py
• Lib/pickle.py
• Lib/shutil.py
• Lib/tarfile.py
• Lib/test/picklecommon.py
• Lib/test/pickletester.py
• Lib/test/test_base64.py
• Lib/test/test_binascii.py
• Lib/test/test_capi/test_hash.py
• Lib/test/test_free_threading/test_pickle.py
• Lib/test/test_gzip.py
• Lib/test/test_json/test_decode.py
• Lib/test/test_json/test_default.py
• Lib/test/test_json/test_dump.py
• Lib/test/test_json/test_encode_basestring_ascii.py
• Lib/test/test_json/test_enum.py
• Lib/test/test_json/test_fail.py
• Lib/test/test_json/test_pickleable.py
• Lib/test/test_json/test_scanstring.py
• Lib/test/test_json/test_speedups.py
• Lib/test/test_json/test_unicode.py
• Lib/test/test_pickle.py
• Lib/test/test_pyexpat.py
• Lib/test/test_shutil.py
• Lib/test/test_tarfile.py
• Lib/test/test_webbrowser.py
• Lib/test/test_xml_etree.py
• Lib/test/test_zipfile/test_core.py
• Lib/test/test_zlib.py
• Lib/test/test_zstd.py
• Lib/turtledemo/__main__.py
• Lib/webbrowser.py
• Lib/xml/etree/ElementPath.py
• Lib/xml/etree/ElementTree.py
• Lib/zipfile/__init__.py
• Misc/NEWS.d/next/Documentation/2023-09-16-23-42-27.gh-issue-109503.mZ-kdU.rst
• Misc/NEWS.d/next/Library/2021-10-18-13-46-55.bpo-45509.Upwb60.rst
• Misc/NEWS.d/next/Library/2025-05-19-20-29-35.gh-issue-133998.KmElUw.rst
• Misc/NEWS.d/next/Library/2025-05-19-21-08-25.gh-issue-134261.ravGYm.rst
• Misc/NEWS.d/next/Library/2026-03-26-09-30-00.gh-issue-146452.Y2N6qZ8J.rst
• Misc/NEWS.d/next/Library/2026-04-29-08-10-17.gh-issue-149056.jnaD4W.rst
• Misc/NEWS.d/next/Library/2026-05-07-14-18-47.gh-issue-149489.bX9iHe.rst
• Misc/NEWS.d/next/Library/2026-05-17-02-25-56.gh-issue-149571.LNyuWJ.rst
• Misc/NEWS.d/next/Library/2026-05-18-12-42-31.gh-issue-149816.F98iME.rst
• Misc/NEWS.d/next/Library/2026-05-19-18-25-00.gh-issue-150077.zstopen-close.rst
• Misc/NEWS.d/next/Library/2026-05-19-19-00-49.gh-issue-84353.ZU5zaQ.rst
• Misc/NEWS.d/next/Library/2026-06-03-19-35-32.gh-issue-150583.dedI24.rst
• Misc/NEWS.d/next/Library/2026-06-23-17-14-04.gh-issue-127802.nGSxo6.rst
• Misc/NEWS.d/next/Library/2026-06-24-12-00-00.gh-issue-152052.yBssDE.rst
• Misc/NEWS.d/next/Library/2026-06-24-23-28-42.gh-issue-151669.tPUavQ.rst
• Misc/NEWS.d/next/Library/2026-07-04-20-28-51.gh-issue-153037.XUSp_g.rst
• Misc/NEWS.d/next/Library/2026-07-26-00-00-00.gh-issue-154738.Rd5Pxe.rst
• Misc/NEWS.d/next/Library/2026-07-26-11-45-34.gh-issue-154726.hLJk5-.rst
• Misc/NEWS.d/next/Library/2026-07-29-18-30-00.gh-issue-154848.Fr4meB.rst
• Misc/NEWS.d/next/Library/2026-07-30-23-29-56.gh-issue-154936.a90443.rst
• Misc/NEWS.d/next/Security/2026-04-26-19-30-45.gh-issue-149018.a9SqWb.rst
• Misc/NEWS.d/next/Security/2026-05-18-17-46-00.gh-issue-149835.EebFlk.rst
• Misc/NEWS.d/next/Security/2026-06-10-13-08-19.gh-issue-151558.mL74i2.rst
• Misc/NEWS.d/next/Security/2026-06-23-13-28-16.gh-issue-151981.xBHEcU.rst
• Misc/NEWS.d/next/Security/2026-06-23-14-19-30.gh-issue-151987.8mNIMf.rst
• Misc/NEWS.d/next/Security/2026-06-30-13-24-13.gh-issue-152674.-2QVoL.rst
• Modules/_elementtree.c
• Modules/_json.c
• Modules/_pickle.c
• Modules/_testcapi/hash.c
• Modules/_zstd/_zstdmodule.c
• Modules/_zstd/_zstdmodule.h
• Modules/_zstd/buffer.h
• Modules/_zstd/clinic/_zstdmodule.c.h
• Modules/_zstd/clinic/compressor.c.h
• Modules/_zstd/clinic/decompressor.c.h
• Modules/_zstd/clinic/zstddict.c.h
• Modules/_zstd/compressor.c
• Modules/_zstd/decompressor.c
• Modules/_zstd/zstddict.c
• Modules/_zstd/zstddict.h
• Modules/binascii.c
• Modules/clinic/_json.c.h
• Modules/clinic/binascii.c.h
• Modules/clinic/pyexpat.c.h
• Modules/pyexpat.c

The CPython standard library provides a comprehensive suite of modules for data serialization and format handling. These range from Python-specific object serialization (pickle) to industry-standard interchange formats (json, xml, csv) and binary data manipulation (struct). Additionally, high-level utilities for archive management (zipfile, tarfile, shutil) allow for efficient data bundling and compression.

1. Python Object Serialization (pickle)

The pickle module implements binary protocols for serializing and de-serializing a Python object structure. For performance, CPython provides a highly optimized C implementation in _pickle.c.

1.1 Architecture and Implementation

The serialization process is managed by the Pickler class, while de-serialization is handled by Unpickler. The C implementation uses a stack-based virtual machine to process opcodes that represent object construction steps Modules/_pickle.c64-143

Key Components:

• Pickle VM: A stack-based engine that executes opcodes like MARK, STOP, LIST, and DICT Modules/_pickle.c67-105
• Memoization: A mechanism to handle self-referential objects by storing a reference to objects already seen in a "memo" Modules/_pickle.c37-39
• Protocols: Supports multiple versions (0 to 5). Protocol 5 is currently the default and highest supported protocol in this implementation Modules/_pickle.c47-49 It introduced PickleBuffer for out-of-band data transfers Lib/test/pickletester.py221-224

1.2 Data Flow: Serialization and Deserialization

The following diagram maps the high-level Python API to the underlying C implementation entities.

Pickle System Mapping

Sources: Modules/_pickle.c11-27 Modules/_pickle.c34-40 Modules/_pickle.c171-216

---

2. JSON Interchange Format

The json module provides an interface for encoding and decoding JSON (JavaScript Object Notation). It includes a C accelerator (_json.c) for the scanning and encoding phases.

2.1 Scanner and Encoder

• Scanner: The PyScannerObject iterates through strings to find JSON tokens. It supports custom hooks like object_hook, object_pairs_hook, and parse_float Modules/_json.c31-40
• Encoder: The PyEncoderObject handles the transformation of Python objects into JSON strings, managing indent, key_separator, and sort_keys Modules/_json.c55-67

JSON Internal Structure

Entity	Role	Code Reference
json_decoder	Global string literal for the decoder module	Include/internal/pycore_global_strings.h51
JSONDecodeError	Specialized exception for JSON parsing failures	Include/internal/pycore_global_strings.h78
scan_once_unicode	Core C function for recursive JSON scanning	Modules/_json.c91-92
encoder_listencode_obj	Recursive C function for encoding Python objects to JSON	Modules/_json.c110-111

Sources: Modules/_json.c31-81 Include/internal/pycore_global_strings.h51 Include/internal/pycore_global_strings.h78

---

3. Binary Data Structures (struct)

The struct module performs conversions between Python values and C structs represented as Python bytes objects. It uses a format string as a compact description of the layout.

3.1 Implementation Details

The module compiles format strings into internal structures. It uses specific C types to ensure cross-platform consistency of binary representations.

Key Implementation Structures:

• formatdef: Defines the translation function, size, and alignment for each format character Modules/_struct.c47-55
• formatcode: Represents a compiled segment of a format string, including offset and repeat count Modules/_struct.c57-62
• PyStructObject: The core object representing a compiled format string Modules/_struct.c66-74

Struct Code Mapping

Sources: Modules/_struct.c47-74 Modules/_struct.c11-13

---

4. Archive Formats and Compression

CPython provides modules for manipulating various archive formats and performing data compression.

4.1 zipfile

The zipfile module supports reading and writing ZIP files, implementing the PKWARE specification Lib/zipfile/__init__.py77-81

• Structures: Defines constants for headers like structEndArchive (End of Central Directory) Lib/zipfile/__init__.py84-86 and structFileHeader (Local File Header) Lib/zipfile/__init__.py156-159
• Compression: Supports multiple methods including ZIP_STORED (0), ZIP_DEFLATED (8), ZIP_BZIP2 (12), ZIP_LZMA (14), and ZIP_ZSTANDARD (93) Lib/zipfile/__init__.py61-65

4.2 tarfile

The tarfile module handles TAR archives, supporting various formats including USTAR, GNU, and PAX Lib/tarfile.py103-106

• Constants: Defines standard block sizes (512 bytes) Lib/tarfile.py76 and magic strings like GNU_MAGIC and POSIX_MAGIC Lib/tarfile.py78-79
• Format Types: Supports regular files (REGTYPE), links (LNKTYPE), directories (DIRTYPE), and extended headers (XHDTYPE) Lib/tarfile.py85-101

4.3 Compression and Binary Encoding

• zstd/gzip/zlib: CPython provides interfaces to several compression libraries. zipfile and tarfile can utilize zlib, bz2, lzma, and zstd if available Lib/zipfile/__init__.py16-36 Lib/test/test_tarfile.py26-45
• binascii: Provides low-level routines for converting between binary and various ASCII-encoded binary representations. It is used for CRC32 calculations when zlib is unavailable Lib/zipfile/__init__.py16-21
• base64: Implements data encoding and decoding as specified in RFC 4648, including Base16, Base32, and Base64 Lib/base64.py1-20

Sources: Lib/zipfile/__init__.py61-159 Lib/tarfile.py75-106 Lib/test/test_tarfile.py26-45 Lib/base64.py1-20

---

5. Encoding and Globalization

CPython manages a global set of strings and objects to optimize serialization and runtime performance.

5.1 Unicode Interning and Global Objects

To optimize memory and comparison speed for serialization keys (like JSON field names or XML tags), CPython interns many global strings during interpreter initialization Include/internal/pycore_unicodeobject_generated.h12-13

Common Interned Strings for Serialization:

• json_decoder Include/internal/pycore_global_strings.h51
• utf_8 Include/internal/pycore_global_strings.h57
• __getstate__, __setstate__, __reduce__, __reduce_ex__ (used by pickle) Include/internal/pycore_global_strings.h147-194

Small Integer Optimization: CPython maintains a cache of small integers (typically -5 to 256) which are frequently used as values in serialized data formats Include/internal/pycore_runtime_init_generated.h15-226

Sources: Include/internal/pycore_unicodeobject_generated.h12-163 Include/internal/pycore_global_strings.h51-194 Include/internal/pycore_runtime_init_generated.h15-226