Part 1, presenting a pattern for a CI/CD pipeline implementation using Gitlab CI for Continuous Integration and Continuous Deployment of Python projects distributed as wheel packages using PyPI.

In a previous post I announced the release of the download_3gpp CLI utility implemented in Python. This project also happens to be my most current iteration of a Python CI/CD pipeline using Gitlab-CI. In this first part I’m going to discuss more general Python project configuration issues relating to pipeline goals, the pipeline docker container, pipeline package dependencies and release management. In the second part I will discuss the implementation of the .gitlab-ci.yml file.

Just a reminder that “CI” is an acronym for Continuous Integration and “CD” is an acronym for Continuous Deployment or Continuous Delivery. The term “Continuous Deployment” can be a little specialized to products such as websites that get deployed live to customers in some sense; there are lots of other products don’t have the same “liveness” quality, but that still benefit from the concepts of automation and ensuring constant delivery of the end product, hence Continuous Delivery.

Often the word “pipeline” is used as a synonym for CI/CD automation and I will continue that practice here.

Release management

I’m a big fan of Semantic Versioning; three numbers separated by periods indicating three “grades” of changes to the code <major>.<minor>.<patch>. Fortunately Semantic Versioning is a subset of the Python version identifier specification PEP-440.

  • major is a breaking change; that is, changes have been made that something about the software such as an API or UI that users may consume is no longer compatible with older versions of the software.
  • minor is a non-breaking change; new features have been added that maintain compatibility with older versions of the software.
  • patch is a change made exclusively for the purpose of fixing something

By implication a patch should also be a non-breaking change, although it’s possible to see a situation where the bug being patched is so severe that it requires a breaking change. In this case I would recommend that the patch should be released as a major release, highlighting that users will require re-integration effort to accommodate the fix in their systems. Unfortunately there may be all kinds of reasons why it is considered necessary to release the breaking change patch as a patch release; it becomes a judgement call with agreement between you (as release manager) and your stakeholders (probably marketing and business management). As a regular expression in Gitlab CI syntax a semantic version is define as /^\d+\.\d+\.\d+$/.

From a git branching workflow, I’m assuming something like the GitFlow Workflow, or the related Feature Branch Workflow.

For continuous delivery we want to get as close as possible to the aspirational goal of “push button releases” 1 implying that a release is 100% automated. Note that getting to the step of actually making a release might not be very easy even though the final step of “pushing the button” is easy; reviews, verification, sign-off and all those other things relevant to your business context. So far I’ve found that manually applying a Semantic Version tag to a git repository is close enough to a push button release for most purposes and is simple to implement in Gitlab CI.

An interesting project that attempts to fully automate the release identifier management is python-semantic-release. Commit messages require a special syntax of keywords to indicate the type of content My experience suggests that there are likely to be multiple issues with using the python-semantic-release on sizable projects:

  • In my experience stakeholders frequently want to arbitrarily define release identifiers for marketing or business purposes. This could be resolved by having internal engineering releases (managed by automation) and separate marketing releases that pick up specific engineering releases, but this introduces complexity and potential errors of its own.
  • Developers have to be extremely disciplined in maintaining the commit message syntax to enable correct identification of releases. The goal of automation is to reduce complexity and errors and enable developers to focus on higher value tasks but manual maintenance of special commit message syntax increases complexity and will very likely result in an increase in errors with the significant side effect of directly impacting releases.

For these reasons I don’t intend to pursue using python-semantic-release for my own projects.

So in conclusion for release management:

  • A Semantic Version is used to indicate a release.
  • A manually applied git tag of a Semantic Version flags a formal release.
  • The Gitlab CI pipeline is conditioned to take the necessary release actions when a Semantic Version git tag is applied to the git repository.

Pipeline goals

Most CI tools offer many different ways to achieve the same objective. In fact if they did not they probably aren’t versatile enough to be useful for your development project. However those multiple paths can lead to maintenance issues in future if care is not taken with the design goals of the pipeline.

It’s really important that feature branches run the same pipeline code as releases as much as possible in order to ensure that everything will run smoothly on release. If the release jobs are too different from non-release jobs then there is a substantial risk that everything looks fine in feature branch and once merged to master and a release is tagged then the release branch fails due to a change in the feature branch pipeline code that wasn’t accounted for in the release pipeline code. Definitely “crunch time” when that happens, not pleasant, and can be more easily avoided by maintaining uniformity between feature branch and release code in the pipeline. Ideally there would be no difference at all but this can be difficult to achieve in practice for lots of good reasons.

The CI pipeline should run as fast as possible. Pipelines often necessarily run more numerous and more complex tasks than developers do in their own testing so the question becomes, what should be done to minimise pipeline execution time? In Gitlab CI terms run as many jobs in parallel as possible (“stages” in Gitlab CI represent a serial dependency, while jobs within a stage are run simultaneously). Then of course there are the computing resources for your CI infrastructure; ensure sufficient CPU, RAM and storage resources are available for your build nodes and containers.

Developers do need to push regularly to the server to check that their changes are not breaking the pipeline in any way, but when the pipeline takes a long time to run this can discourage them from pushing frequently, perhaps holding back a number of commits before pushing and now it becomes more difficult to find out which change or changes broke the pipeline.

Pipeline package dependencies

My go to Python packaging tool is flit; it’s extremely simple to use and in my experience is adequate for the vast majority (let’s call it “90%”) of the types of projects I’ve worked on. For that small number of projects where flit is inadequate then the more complex/capable tools such as setuptools or poetry can be used, but the benefit of ease of use for most of your projects is acquired by using flit.

Flit uses the relatively new pyproject.toml format specified in PEP-518 for project metadata and dependency management. Flit also makes a distinction between run time dependencies of the project and other project dependencies such as development, test and documentation. For the most part, these development related dependencies are also what is needed for the pipeline.

Recently I attended an excellent Ottawa Python Authors Group Meetup on Python CI tools by Jeff McLean of Snyk. Jeff introduced me to black and isort. Like Rusts rustfmt, black and isort format Python code to compliance with PEP-8. Do I agree with all their formatting choices? No. But do I have to give any further thought to code formatting? No again, except for variable, function and class name style conventions which are not modified by black. For this reason I intend that all my future projects will use these packages as-is without further consideration. No need for pylint or mypy.

As pointed out in this isort issue it is necessary to add some information to pyproject.toml to prevent isort from making changes that conflict with black.

So this is a sample of what my pyproject.toml looks like for basic developer and pipeline packages.

# Not a complete pyproject.toml!

[tool.flit.metadata]
# project installation and run time configuration

[tool.flit.metadata.requires-extra]
dev = [
    "black >=19.10b0",
    "isort >=4.3.21, <5.0",
    # required for `description-file` parameter in metadata (integration of README.rst)
    "pygments >=2.5.2, <3.0",
]
test = [
    "pytest >=5.3.4, <6.0",
    "pytest-cov >=2.8.1, <3.0",
    "pytest-mock >=2.0.0, <3.0",
]

# isort configuration required to ensure compatibility with black
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
line_length = 88

Until recently I used the standard unittest package for unit tests. Mostly just to avoid the additional package dependency of an external unit test framework. However I experimented a bit with pytest and realised that there really is value in the reduced boilerplate of pytest compared to unittest and have converted. When you are doing test coverage analysis using pytest it is necessary to include the .coveragerc file below to ensure that the tests themselves are not included in the coverage analysis.

# Ensure test coverage analysis doesn't run on the tests themselves.
[run]
omit = *tests/*

Pipeline docker image

There is a Python specific docker image, but I prefer to use the base Ubuntu image and add all the necessary apt and wheel packages. For Python project the Dockerfile is more-or-less the same now and looks like this example from the download_3gpp project.

FROM ubuntu:18.04

# git-core package required for flit
RUN set -x \
 && apt-get update \
 && apt-get install -y \
      git-core \
      python3 \
      python3-pip \
      python3-venv \
 && rm -rf /var/lib/apt/lists/*

COPY requirements.txt /requirements.txt

RUN set -x \
 && python3 -m venv /venv \
 && /venv/bin/pip install --upgrade pip \
 && /venv/bin/pip install -r /requirements.txt

The Python requirements for download_3gpp are defined in the requirements.txt file. Unfortunately you can see that this is mostly a copy-paste of the dependency definitions in pyproject.toml and since the docker image is maintained in a separate project I haven’t found a work around to this yet.

# download_3gpp dependencies
beautifulsoup4 >=4.8.2, <5.0
requests >=2.22.0, <3.0

# download_3gpp dev dependencies
black >=19.10b0
isort >=4.3.21, <5.0
pygments >=2.5.2, <3.0

# download_3gpp test dependencies
pytest >=5.3.4, <6.0
pytest-cov >=2.8.1, <3.0
pytest-mock >=2.0.0, <3.0

# general dependencies for build
flit >=2.2.0, <3.0

It is possible to dynamically install Python dependencies in the CI pipeline of your project but this will slow the pipeline down as it re-installs the dependencies on each job, although this might be improved by using caching within the pipeline context. In addition, installing the dependencies must be maintained for each job which can become a maintenance issue without care. Since the dependencies usually change relatively slowly, except perhaps at the beginning of the project and for the improved maintainability I consider it preferable to install dependencies in the docker image.

Another factor regarding dependency installation is version control. In the above requirements I have constrained the packages to “the same major release” with the exception of black which is in beta still. One problem with this approach is that in the pipeline I often use flit install -s to quickly install the package under test and if updated dependency packages are available they could be installed as well and this will delay the pipeline. In addition, the security best practice is to pin your dependency packages to an exact version because an adversary could introduce an issue into a new release which would be automatically acquired within the pipeline and potentially your deployment if you are doing Continuous Deployment. I had started pinning dependency package versions but one problem I experiences was that multiple packages included pinned dependencies of the same package at different versions which caused installation to fail.

eg. My project depends on packages “A” and “B” which in turn both depend on package “C”. The problem is that “A” has pinned “C == 0.1.0” and “B” has pinned “C == 0.2.0”. So now “A” can be installed with it’s version of “C”, or “B” can be installed, but not both at the same time.

For this reason I have so far reverted to more open-ended release specification in dependencies until I can figure out how to resolve this problem.

Conclusions

  • Use Semantic Version tags in the git repository for releases.
  • Aspire to having no code difference between “feature branch” pipelines and “release” pipelines.
  • Minimize pipeline execution time through design of the pipeline and deploying sufficient hardware resources for the pipeline execution.
  • Maintain the pipeline docker image in a separate repository and install all your project dependencies in that image.

  1. “Continuous Delivery”, Humble & Farley ↩