This document describes TensorFlow's Python package generation pipeline, which builds platform-specific wheel files (.whl) for distribution via PyPI. The pipeline transforms compiled C++/Python source code into installable pip packages. For information about build configuration and compiler setup, see Build Configuration. For dependency version management, see Dependency Management.
The Python package generation system coordinates multiple subsystems to produce TensorFlow wheels:
@tf_export decorators to construct public Python API.The pipeline produces distinct wheel variants for various platforms (Linux x86_64/aarch64, macOS, Windows) and Python versions (3.9-3.12).
Pipeline Architecture
The build pipeline assembles wheels from four categories of artifacts: compiled C++ binaries, Python bindings, pure Python source, and generated protocol buffer code. The tf_wheel Bazel rule orchestrates collection through collect_data_files for native code and transitive_py_deps for Python dependencies tensorflow/tools/pip_package/BUILD15-27 Version metadata flows from tensorflow/tf_version.bzl19 into the generated setup.py. Dependencies are managed via requirements files for different platforms, such as tensorflow/tools/ci_build/release/requirements_common.txt4-21
Sources: tensorflow/tools/pip_package/BUILD15-27 tensorflow/tf_version.bzl19-20 tensorflow/tools/ci_build/release/requirements_common.txt1-46
TensorFlow uses semantic versioning defined in Bazel and C++ headers:
| Version Type | Definition Location | Purpose |
|---|---|---|
| Package Version | tensorflow/tf_version.bzl | Wheel package version (TF_VERSION) |
| Python Version | pip_new.sh | Targeted Python interpreter (TF_PYTHON_VERSION) |
Version Management System
The TF_VERSION constant in tensorflow/tf_version.bzl19 defines the primary version string. This is loaded into tensorflow/tools/pip_package/BUILD19 and used to configure the pip package generation. The pip_new.sh script utilizes environment variables like TF_PYTHON_VERSION to determine which Python version (e.g., python3.8) to build for tensorflow/tools/ci_build/builds/pip_new.sh28
Sources: tensorflow/tf_version.bzl19-20 tensorflow/tools/pip_package/BUILD19-20 tensorflow/tools/ci_build/builds/pip_new.sh25-44
TensorFlow maintains requirements files for different CI environments and Python versions:
Dependency Resolution
The install_pip_packages_by_version.sh script handles the complexity of Python version-specific dependencies. It installs a base set of PACKAGES tensorflow/tools/ci_build/install/install_pip_packages_by_version.sh57-89 and then applies special casing for NumPy based on PYTHON_VERSION tensorflow/tools/ci_build/install/install_pip_packages_by_version.sh122-128 CI environments use install_ubuntu_pip_deps to install requirements from the release-specific files tensorflow/tools/ci_build/release/common.sh164-176
Sources: tensorflow/tools/ci_build/release/requirements_common.txt1-46 tensorflow/tools/ci_build/install/install_pip_packages_by_version.sh57-128 tensorflow/tools/ci_build/release/common.sh164-176
The tf_wheel Bazel rule is the primary mechanism for packaging. It depends on included_headers and COMMON_PIP_DEPS.
Wheel Assembly Mechanism
Artifact collection is defined in tensorflow/tools/pip_package/BUILD included_headers uses transitive_hdrs to gather all public C++ headers required for custom op development tensorflow/tools/pip_package/BUILD44-91 COMMON_PIP_DEPS includes critical Python components like tflite_convert, autograph, and various Python ops tensorflow/tools/pip_package/BUILD147-170 Dynamic kernels, such as libtfkernel_sobol_op.so, are explicitly listed for inclusion tensorflow/tools/pip_package/BUILD143-145
Sources: tensorflow/tools/pip_package/BUILD44-170 tensorflow/tools/pip_package/utils/tf_wheel.bzl27
For Linux distributions, wheels must comply with manylinux standards to ensure portability across different distributions.
| Step | Tool | Description |
|---|---|---|
| Repair | auditwheel | Vendors external shared libraries into the wheel and updates RPATH. |
| Test | nightly_release_smoke_test.sh | Verifies the wheel can be installed and basic imports work. |
| Upload | twine | Uploads the audited and tested wheel to PyPI. |
Auditwheel Pipeline
The upload_wheel_cpu_ubuntu function in tensorflow/tools/ci_build/release/common.sh203-227 demonstrates the compliance flow. It iterates through built wheels, runs auditwheel repair --plat manylinux2010_x86_64, and then executes a smoke test tensorflow/tools/ci_build/release/common.sh214 If the smoke test passes, it uses twine to upload the package to the repository tensorflow/tools/ci_build/release/common.sh221
Sources: tensorflow/tools/ci_build/release/common.sh203-227 tensorflow/tools/pip_package/BUILD12-14
The pip_new.sh script is the entry point for the CI/CD pipeline to generate packages.
Execution Logic
The script first validates required environment variables like CONTAINER_TYPE, OS_TYPE, and TF_BUILD_FLAGS tensorflow/tools/ci_build/builds/pip_new.sh105-138 It then updates Bazel flags to include necessary action environments tensorflow/tools/ci_build/builds/pip_new.sh167-181 The actual build process is triggered via Bazel, followed by packaging logic.
Sources: tensorflow/tools/ci_build/builds/pip_new.sh105-181 tensorflow/tools/pip_package/build_pip_package.py1