Update (2017-11-17)

The submission process has changed slightly since I wrote this post.

PyPI is migrating from https://pypi.python.org/pypi/ to http://pypi.org. The old site is still up, but I don’t think it’s accepting submissions anymore - something I discovered while trying to submit a new version of simplestatistics.

The two big differences are:

  • The submission url has changed. If you have a ~/pypirc file, you can probably fix the submission issue by removing a line that begins with respository=, allowing the default url to be used. See this post for more information.
  • You used to be able to specify a different url in ~/.pypirc to use testpypi, the test server you could use to test the submission. I think this no longer works, and you might need to use twine to be able to make use of that again. I think PyPI is generally encouraging everyone to use twine for all their submission needs.

I maintain and publish a math and statistics Python package called simplestatistics. The library is a port of its javascript ancestor simple-statistics. While building it up, I wanted it to be available on the Python Package Index, PyPI1, and I had to learn how to do that. The process turned out to be more complicated than I expected it to be, and so here are the steps it takes to publish your package, or its updates, to PyPI.

Note: The order of steps matters.

Note 2: Some of these steps are specific to hosting your package on GitHub.

Do you have a setup.py?

You need to create a setup script to publish your package using distutils. This is the one for simplestatistics.

from distutils.core import setup

setup(
    name = 'simplestatistics',
    packages = ['simplestatistics', 'simplestatistics.statistics'],
    version = '0.2.5',
    description = 'Simple statistical functions implemented in readable Python.',
    author = 'Sherif Soliman',
    author_email = 'sherif@ssoliman.com',
    copyright = 'Copyright (c) 2016 Sherif Soliman',
    url = 'https://github.com/sheriferson/simplestatistics',
    download_url = 'https://github.com/sheriferson/simplestatistics/tarball/0.2.5',
    keywords = ['statistics', 'math'],
    classifiers = [
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 3',
        'Topic :: Scientific/Engineering :: Mathematics',
        'Intended Audience :: Developers',
        'Intended Audience :: Education',
        'Intended Audience :: End Users/Desktop',
        'Intended Audience :: Science/Research',
        'Operating System :: MacOS',
        'Operating System :: Unix',
        'Topic :: Education',
        'Topic :: Utilities'
        ]
)

Update changelog.txt

If you maintain a changelog file, make sure you update it with the release version and date.

If you don’t maintain a changelog file, you should.

I maintain a separate HISTORY.rst file, so I make sure I update that too.

Update version number in documentation

Good documentation is important to me. It helps you understand your code and project better, and definitely helps anyone else trying to use it. simplestatistics documentation is hosted and generated automatically by Read the Docs using the very useful Sphinx package.2

When I’m publishing a new release, I make sure I update the version number in Sphinx’s conf.py.

Update version number in setup.py

I mentioned setup.py in the first step. Make sure to update the version number.

from distutils.core import setup

setup(
    ...
    version = '0.2.5',
    ...

Convert README.md to README.rst

PyPI doesn’t like Markdown. Actually, it’s not that it doesn’t like it, it just doesn’t care about it one way or another. PyPI likes reStructuredText (RST from this onwards). If you want PyPI to render the README on the package homepage like you can see on the simplestatistics PyPI page, it has to be in RST.

I’ve had a lot of trouble with RST. In my experience, it’s very fragile. It takes one extra space in a table to break rendering for the whole file.

pandoc is a great tool you could use to convert your README.md to README.rst, but PyPI may not like the default output of pandoc’s conversion. In my use case, the conversion of the Markdown tables to RST tables was the part that often angered PyPI rendering.

After some troubleshooting, I found that this command prevents the tables from wrapping around to new lines and causing README rendering on PyPI to fail.

pandoc --columns=100 --output=README.rst --to rst README.md

Here’s a bonus tip: this online reStructuredText editor, made available by Andrey Rublev, has been a huge help in debugging and fixing RST errors.

Add tarball download url to setup.py

In a later step, we will add a git tag and push it to GitHub. This will create a new release on the GitHub page, and this release will include .zip and .tar.gz files of the release (see an example here).

These compressed files are the ones that PyPI will pull from when you push your release or update. PyPI gets that download url from setup.py.

The result of this circle is that you need to anticipate the url for the release on GitHub before you push the release commit to GitHub.

You can set the new download url in setup.py based on your new version number:

setup(
    ...
    url = 'https://github.com/sheriferson/simplestatistics',
    download_url = 'https://github.com/sheriferson/simplestatistics/tarball/0.2.5',
    ...

Yes, you do set this url and commit it before it actually exists.

If there are new files that should be included, edit MANIFEST.in

The MANIFEST.in file is how you tell disutils to include files in the release file that it wouldn’t include otherwise. This is my MANIFEST.in file:

include LICENSE.txt
include README.rst
HISTORY.rst

Commit all those changes. Have a clean repo.

Commit everything we’ve done so far. Have a consistent commit comment for those changes. I usually add the message: “Prep for 0.2.5 release.”

Add/create a git tag

git tag 1.2.3 -m "Adds 1.2.3 tag for PyPI

Once you have the project in the state you want for creating the release, you add a git tag with the version number of the release. This will be reflected in the “releases” page of your GitHub repository.

Push git tag to remote

git push --tags origin master

Push those tags to GitHub.

Confirm that GitHub has generated the release file

Browse to your releases page (example) and make sure the new version has a release entry with its corresponding files.

Release testing

Update (2017-11-17)

See update at the top for notes about testpypi. The instructions below are here for posterity only. You probably want to use twine.

python setup.py register -r pypitest

You are, or aspire to be a good programmer who wants to be as cautious as possible, and so you’d like to test releasing the update on PyPI before actually doing it.

PyPI provides a test system that you can use to test the registration, upload, and installation of your package. Let’s make use of that gift.

python setup.py register -r pypitest will register the package on the pypitest server.

If you get a message indicating the all-clear, continue.

python setup.py sdist upload -r pypitest

… will upload the distribution of your package to pypitest.

If something in the package is broken, you cannot make changes and reupload the package with the same version number. This means that if you want to make a change or fix something, you will have to change the version number in setup.py (and accordingly everywhere else) and start all over again.3

pip install -i https://testpypi.python.org/pypi simplestatistics

… is the final step in the process of testing the release. This will try to install the new version of the package from pypi. If all goes well, you should be able to import the package normally in the REPL or in a Python script.

Once you’ve tested importing the package, it’s a good idea to pip uninstall your package so you can test the installation from the live PyPI servers.

Release

python setup.py register -r pypi

You are finally at the actual release stage. This will register the new version with PyPI.

python setup.py sdist upload -r pypi

… uploads the distribution to PyPI.

pip install simplestatistics

… tests the installation from PyPI. This is why we pip uninstalled the version from pypitest.

Add changelog notes to GitHub release page/tag

This is not necessary, but it’s good practice and shows care for maintaining documentation of your open source project. Edit the new GitHub release and add notes about what changed in nicely formatted Markdown.

Congratulations!

You made it. It feels good to have a package on PyPI. It helps you use your own package in the future, and it’s a good contribution to make it available to everyone else. Go celebrate.

See also

  1. Which it is! pip install simplestatistics 

  2. The thing I like most about Sphinx is that you write the code, and in the case of simplestatistics the tests, in the docstrings of each .py file. The thing I like the least is that you have to use reStructured text, which is a bouquet of sadness. 

  3. At the time of writing. Things might change and become more flexible in the future. I hope so.