Publishing a Python package used to feel like arcane knowledge passed between maintainers. Today, thanks to a handful of modern tools and a stack of well-designed standards, the path from a GitHub repo to an installable PyPI package is clearer than ever. In this blog post, we want to walk you through the steps we take to publish our dataiku/kiji-inspector library, a mechanistic-interpretability library that trains sparse autoencoders on LLM activations (and then helps us understand why AI agents pick specific tools), to the Python Package Library, PyPI.

Why package your code at all?

Packaging turns a pile of source files into something anyone can install and depend on. Once your project is a package, it's installable anywhere with a simple pip install your-package in any environment, and pip or uv will resolve transitive dependencies for you automatically. Declared version ranges make environments reproducible, and publishing to PyPI puts your work in the global directory where anyone can find and use it.

When you should not package for PyPI

Not every project belongs on PyPI, and reaching for it reflexively adds overhead for no gain. One-off scripts, Jupyter notebooks, and ad-hoc data pipelines are better off simply committed to a repo. Application code, such as Django or Flask apps, is deployed rather than pip-installed, so Docker or a deployment pipeline is a better fit. Internal or proprietary code belongs on a private index such as Artifactory, CodeArtifact, or devpi rather than on the public PyPI. And prototypes with unstable APIs that nobody else will consume aren't worth the packaging effort yet.

The rule of thumb is simple: if someone else can pip install it, package it. Otherwise, a git repo is fine.

The toolbox

The workflow in this guide leans on a small set of modern, fast tools:

• uv: an ultra-fast package manager and project tool from Astral, written in Rust, that replaces pip, pip-tools, virtualenv, and more.

• uvx: runs CLI tools in isolated environments without installing them globally, similar to pipx but powered by uv.

• twine: the standard tool for uploading packages to PyPI; it validates metadata and handles authentication.

• ruff: an extremely fast linter and formatter, also written in Rust, that replaces flake8, isort, and black.

• commitizen: enforces conventional commits and automates version bumping, tagging, and changelog generation.

• pytest: the de-facto Python testing framework, simple and extensible.

Getting started takes two commands:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install ruff commitizen
```

Why a build system exists

Source code on its own isn't installable by pip. Dependencies, metadata, and entry points need to be declared; C extensions require a compilation step; and different operating systems and Python versions may require different outputs. A build backend solves this by reading pyproject.toml and producing a wheel (.whl) plus a source distribution (.tar.gz). Build frontends like pip, uv, and build that automatically call the backend via the PEP 517 interface. You declare what to build, and the backend handles how to build it.

The standards that make it work

Four PEPs form the backbone of modern packaging, and pyproject.toml is the single entry point for all of them:

• PEP 427 (2012) defined the wheel format. The format is a pre-built binary distribution that installs fast with no compilation, replacing the old egg format.

• PEP 518 (2016) introduced the [build-system] table so pip knows what to install before building, and created pyproject.toml itself.

• PEP 517 (2017) defined the API for building frontends and backends, decoupling building from installation.

• PEP 621 (2021) standardized the [project] table, including name, version, dependencies, authors, license, and classifiers, so that a single format works across all backends.

Writing pyproject.toml

PEP 621 means every backend reads the same fields. The essential ones are the package name (hyphens are fine), a version string, a one-line description, requires-python, a license, authors, dependencies, and PyPI classifiers. A minimal skeleton looks like this:

```toml
[build-system]
requires = ["setuptools>=68.0"]
build-backend = "setuptools.build_meta"


[project]
name = "my-package"
version = "0.1.0"
description = "What it does"
requires-python = ">=3.10"
license = {text = "MIT"}
authors = [{name = "You", email = "[email protected]"}]
readme = "README.md"
dependencies = ["requests>=2.28"]

```

kiji-inspector fills in that skeleton with real-world choices: setuptools ≥ 68.0 with wheel (battle-tested for GPU and C dependencies), version 0.0.3 under Apache 2.0, a set of optional-dependency extras ([mamba], [hf], [full]) gated with platform markers, and every tool, ruff, pytest, commitizen, and uv, configured in the same file.

```toml
[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"


[project]
name = "kiji-inspector"
version = "0.0.3"
requires-python = ">=3.10"
license = {text = "Apache 2.0"}
dependencies = [
    "torch>=2.10",
    "transformers>=4.57.6",
]
```

Choosing a build backend

Several backends satisfy the PEP 517 interface, and the right choice depends on your project:

kiji-inspector uses setuptools, the most mature and widely supported option, and a sensible default for projects with optional C dependencies like torch.

The src/ layout

kiji-inspector places its package under a src/ directory rather than at the repo root:

kiji-inspector/

├── pyproject.toml

├── README.md

├── LICENSE

├── Makefile

├── tests/

│   ├── test_core_sae.py

│   └── test_runtime_api.py

└── src/

    └── kiji_inspector/

        ├── __init__.py

        ├── py.typed

        ├── core/

        ├── analysis/

        ├── training/

        ├── extraction/

        └── data/

The src/ layout prevents accidental imports from the working directory and forces pip install -e . to behave correctly. Package discovery is declared explicitly:

```toml
[tool.setuptools.packages.find]
where = ["src"]
include = ["kiji_inspector*"]
```

The empty py.typed marker tells type checkers like mypy and pyright that the package ships inline type hints, following PEP 561.

Editable installs

During development you don't want to rebuild and reinstall after every change. An editable install creates a link from site-packages back to your src/ directory, so edits to .py files take effect immediately. Modern backends implement this through PEP 660.

```bash
pip install -e .                   # classic pip
uv pip install -e .              # with uv (faster)
uv pip install -e ".[dev]"    # with dev extras
```

This is the single biggest reason to adopt the src/ layout: flat layouts can silently import from the wrong place during editable installs.

Dependencies and optional extras

Core runtime dependencies live in the dependencies list, for kiji-inspector that includes torch, numpy, transformers, pandas, scikit-learn, rich, and tqdm. Optional features are grouped into extras that users opt into:

```bash
pip install kiji-inspector[mamba]   # Mamba/NemotronH support (Linux only)
pip install kiji-inspector[hf]            # HuggingFace Hub integration
pip install kiji-inspector[full]           # everything: mamba + hf + vllm + flash-attn
```

Platform markers such as sys_platform == "linux" let you gate GPU-only dependencies to the builds where they make sense.

Build, check, and publish

With the configuration in place, shipping is three commands:

```bash
uv build                          # creates dist/kiji_inspector-0.0.3.tar.gz and .whl
uvx twine check dist/*    # validates metadata, README rendering, versions
uvx twine upload dist/*   # uploads to PyPI
```

If you prefer a backend-agnostic frontend, python -m build uses the build package and the same PEP 517 machinery.

What actually gets built

A build produces two artifacts. The source distribution (.tar.gz) contains raw source plus pyproject.toml, needs a build step on install, and serves as the reproducible fallback when no wheel matches. The wheel (.whl) is pre-built and installs without compilation; pip prefers it whenever one is available. A filename like kiji_inspector-0.0.3-py3-none-any.whl decodes as Python 3, no specific ABI, and any platform. In other words, pure Python that runs anywhere.

Version management with commitizen

commitizen ties versioning to your commit messages. You write conventional commits like feat: add SAE describe or fix: normalize activations, run a bump command, and it updates the version, creates a git tag, and regenerates the changelog:

```toml
[tool.commitizen]
name = "cz_conventional_commits"
tag_format = "kiji-inspector-v$version"
version_scheme = "semver"
version_provider = "pep621"
update_changelog_on_bump = true
```

The commit type drives the semver bump:

kiji-inspector wraps these in Makefile shortcuts (make bump-patch, make bump-minor, make bump-major, and make bump-dry-run to preview), each of which updates pyproject.toml, tags the release, and writes the changelog.

Automated publishing with GitHub Actions

The release flow can be fully automated: push a tag matching kiji-inspector-v*, build and check the artifacts, and publish to PyPI. The modern approach uses Trusted Publishing so no API tokens are involved:

```yaml
permissions:
  id-token: write   # OIDC token for PyPI
steps:
  - uses: actions/checkout@v4
  - run: uv build
  - run: uvx twine check dist/*
  - uses: pypa/gh-action-pypi-publish@release/v1
```

Trusted Publishing works through OIDC. You register the project on PyPI, add a trusted publisher (Project → Settings → Add Publisher → GitHub), configure it with your owner, repo, workflow file, and environment, and set id-token: write in the workflow. GitHub then sends a signed token that PyPI verifies. There are no tokens to rotate, no secrets to leak, and every publish is auditable through its OIDC claims.

Test on TestPyPI first

TestPyPI is a separate, independent instance of PyPI with its own accounts, packages, and index URL, basically a staging ground before you touch production. Both instances treat uploaded versions as immutable, so you bump the version before retrying a failed upload on either one. A typical dry run looks like this:

```bash
# Upload to TestPyPI
uvx twine upload --repository testpypi dist/*


# Install from TestPyPI, pulling real deps from production PyPI
pip install -i https://test.pypi.org/simple/ \
  --extra-index-url https://pypi.org/simple/ kiji-inspector
```

Testing with pytest

kiji-inspector keeps its pytest configuration in pyproject.toml and organizes tests under a tests/ directory with shared fixtures in conftest.py alongside unit tests for the SAE core, the model registry, the public API, and a training smoke test.

```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
```

Its CI matrix covers Python 3.10 (the minimum supported), 3.12 (the dev default), 3.13, and 3.14 (pre-release). A fast smoke test in CI simply verifies the package imports cleanly, catching broken imports or dependencies before they reach users:

```bash
uv run python -c "from kiji_inspector import SAE; \
from kiji_inspector.core.sae_core import JumpReLUSAE"
```

Linting and formatting with ruff

A single ruff configuration replaces flake8, isort, black, pyflakes, and flake8-bugbear:

```toml
[tool.ruff]
line-length = 100
target-version = "py310"


[tool.ruff.lint]
select = ["E", "W", "F", "I", "B", "C4"]
```

kiji-inspector exposes the common actions as Makefile targets: make format runs the formatter, make lint checks the source directories, and make fix auto-fixes issues and reformats in one pass.

Package your own project

Putting it all together, here's the end-to-end sequence for a new project:

```bash

# 1. Create the src layout
mkdir -p src/my_package && touch src/my_package/__init__.py


# 2. Write pyproject.toml -> copy kiji-inspector's as a template


# 3. Install build tools
pip install build twine


# 4. Build sdist + wheel
python -m build


# 5. Validate the package
twine check dist/*


# 6. Test on TestPyPI first
twine upload --repository testpypi dist/*


# 7. Verify it installs
pip install --index-url https://test.pypi.org/simple/ my-package


# 8. Ship to production PyPI
twine upload dist/*
```

Key takeaways

• pyproject.toml is the single source of truth, embrace PEP 621 nowadays. 

• Use the src/ layout to avoid import traps. 

• Pair commitizen with semver for painless version management.

• Lean on Trusted Publishing to eliminate API-key risk entirely. 

• Run Ruff and Pytest in CI as quality gates that never sleep

• Always test on TestPyPI before going live.

You can find the full setup working in Dataiku’s open-source projects. An example is the Kiji Inspector project mentioned, which lives at github.com/dataiku/kiji-inspector.