Part 1, presenting a pattern for a CI/CD pipeline implementation using Gitlab CI for Continuous Integration and Continuous Deployment of Python projects distributed as wheel packages using PyPI.
In a previous post I announced the release of the download_3gpp CLI utility
implemented in Python. This project also happens to be my most current
iteration of a Python CI/CD pipeline using Gitlab-CI. In this first part I’m
going to discuss more general Python project configuration issues relating to
pipeline goals, the pipeline docker container, pipeline package dependencies and
release management. In the second part I will discuss the implementation of the
Just a reminder that “CI” is an acronym for Continuous Integration and “CD” is an acronym for Continuous Deployment or Continuous Delivery. The term “Continuous Deployment” can be a little specialized to products such as websites that get deployed live to customers in some sense; there are lots of other products don’t have the same “liveness” quality, but that still benefit from the concepts of automation and ensuring constant delivery of the end product, hence Continuous Delivery.
Often the word “pipeline” is used as a synonym for CI/CD automation and I will continue that practice here.
I’m a big fan of Semantic Versioning; three numbers
separated by periods indicating three “grades” of changes to the code
<major>.<minor>.<patch>. Fortunately Semantic Versioning is a subset of the
Python version identifier specification
- major is a breaking change; that is, changes have been made that something about the software such as an API or UI that users may consume is no longer compatible with older versions of the software.
- minor is a non-breaking change; new features have been added that maintain compatibility with older versions of the software.
- patch is a change made exclusively for the purpose of fixing something
By implication a patch should also be a non-breaking change, although it’s
possible to see a situation where the bug being patched is so severe that it
requires a breaking change. In this case I would recommend that the patch
should be released as a major release, highlighting that users will require
re-integration effort to accommodate the fix in their systems. Unfortunately
there may be all kinds of reasons why it is considered necessary to release the
breaking change patch as a patch release; it becomes a judgement call with
agreement between you (as release manager) and your stakeholders (probably
marketing and business management). As a regular expression in Gitlab CI syntax
a semantic version is define as
For continuous delivery we want to get as close as possible to the aspirational goal of “push button releases” 1 implying that a release is 100% automated. Note that getting to the step of actually making a release might not be very easy even though the final step of “pushing the button” is easy; reviews, verification, sign-off and all those other things relevant to your business context. So far I’ve found that manually applying a Semantic Version tag to a git repository is close enough to a push button release for most purposes and is simple to implement in Gitlab CI.
An interesting project that attempts to fully automate the release identifier management is python-semantic-release. Commit messages require a special syntax of keywords to indicate the type of content My experience suggests that there are likely to be multiple issues with using the python-semantic-release on sizable projects:
- In my experience stakeholders frequently want to arbitrarily define release identifiers for marketing or business purposes. This could be resolved by having internal engineering releases (managed by automation) and separate marketing releases that pick up specific engineering releases, but this introduces complexity and potential errors of its own.
- Developers have to be extremely disciplined in maintaining the commit message syntax to enable correct identification of releases. The goal of automation is to reduce complexity and errors and enable developers to focus on higher value tasks but manual maintenance of special commit message syntax increases complexity and will very likely result in an increase in errors with the significant side effect of directly impacting releases.
For these reasons I don’t intend to pursue using python-semantic-release for my own projects.
So in conclusion for release management:
- A Semantic Version is used to indicate a release.
- A manually applied git tag of a Semantic Version flags a formal release.
- The Gitlab CI pipeline is conditioned to take the necessary release actions when a Semantic Version git tag is applied to the git repository.
Most CI tools offer many different ways to achieve the same objective. In fact if they did not they probably aren’t versatile enough to be useful for your development project. However those multiple paths can lead to maintenance issues in future if care is not taken with the design goals of the pipeline.
It’s really important that feature branches run the same pipeline code as releases as much as possible in order to ensure that everything will run smoothly on release. If the release jobs are too different from non-release jobs then there is a substantial risk that everything looks fine in feature branch and once merged to master and a release is tagged then the release branch fails due to a change in the feature branch pipeline code that wasn’t accounted for in the release pipeline code. Definitely “crunch time” when that happens, not pleasant, and can be more easily avoided by maintaining uniformity between feature branch and release code in the pipeline. Ideally there would be no difference at all but this can be difficult to achieve in practice for lots of good reasons.
The CI pipeline should run as fast as possible. Pipelines often necessarily run more numerous and more complex tasks than developers do in their own testing so the question becomes, what should be done to minimise pipeline execution time? In Gitlab CI terms run as many jobs in parallel as possible (“stages” in Gitlab CI represent a serial dependency, while jobs within a stage are run simultaneously). Then of course there are the computing resources for your CI infrastructure; ensure sufficient CPU, RAM and storage resources are available for your build nodes and containers.
Developers do need to push regularly to the server to check that their changes are not breaking the pipeline in any way, but when the pipeline takes a long time to run this can discourage them from pushing frequently, perhaps holding back a number of commits before pushing and now it becomes more difficult to find out which change or changes broke the pipeline.
Pipeline package dependencies
My go to Python packaging tool is flit; it’s extremely simple to use and in my experience is adequate for the vast majority (let’s call it “90%”) of the types of projects I’ve worked on. For that small number of projects where flit is inadequate then the more complex/capable tools such as setuptools or poetry can be used, but the benefit of ease of use for most of your projects is acquired by using flit.
Flit uses the relatively new
pyproject.toml format specified in
PEP-518 for project metadata and
dependency management. Flit also makes a distinction between run time
dependencies of the project and other project dependencies such as development,
test and documentation. For the most part, these development related
dependencies are also what is needed for the pipeline.
Recently I attended an excellent
Ottawa Python Authors Group Meetup
on Python CI tools by Jeff McLean of Snyk. Jeff introduced me to
isort. Like Rusts
rustfmt, black and isort
format Python code to compliance with PEP-8.
Do I agree with all their formatting choices? No. But do I have to give any
further thought to code formatting? No again, except for variable, function
and class name style conventions which are not modified by black. For this
reason I intend that all my future projects will use these packages as-is
without further consideration. No need for pylint or mypy.
As pointed out in
this isort issue it is
necessary to add some information to
pyproject.toml to prevent isort from
making changes that conflict with black.
So this is a sample of what my
pyproject.toml looks like for basic developer
and pipeline packages.
# Not a complete pyproject.toml! [tool.flit.metadata] # project installation and run time configuration [tool.flit.metadata.requires-extra] dev = [ "black >=19.10b0", "isort >=4.3.21, <5.0", # required for `description-file` parameter in metadata (integration of README.rst) "pygments >=2.5.2, <3.0", ] test = [ "pytest >=5.3.4, <6.0", "pytest-cov >=2.8.1, <3.0", "pytest-mock >=2.0.0, <3.0", ] # isort configuration required to ensure compatibility with black [tool.isort] multi_line_output = 3 include_trailing_comma = true force_grid_wrap = 0 use_parentheses = true line_length = 88
Until recently I used the standard unittest package for unit tests. Mostly
just to avoid the additional package dependency of an external unit test
framework. However I experimented a bit with pytest and realised that there
really is value in the reduced boilerplate of pytest compared to unittest
and have converted. When you are doing test coverage analysis using pytest it
is necessary to include the
.coveragerc file below to ensure that the tests
themselves are not included in the coverage analysis.
# Ensure test coverage analysis doesn't run on the tests themselves. [run] omit = *tests/*
Pipeline docker image
There is a Python specific docker image, but I prefer to use the base Ubuntu
image and add all the necessary apt and wheel packages. For Python project the
Dockerfile is more-or-less the same now and looks like this example from the
FROM ubuntu:18.04 # git-core package required for flit RUN set -x \ && apt-get update \ && apt-get install -y \ git-core \ python3 \ python3-pip \ python3-venv \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt /requirements.txt RUN set -x \ && python3 -m venv /venv \ && /venv/bin/pip install --upgrade pip \ && /venv/bin/pip install -r /requirements.txt
The Python requirements for download_3gpp are defined in the
requirements.txt file. Unfortunately you can see that this is mostly a
copy-paste of the dependency definitions in
pyproject.toml and since the
docker image is maintained in a separate project I haven’t found a work around
to this yet.
# download_3gpp dependencies beautifulsoup4 >=4.8.2, <5.0 requests >=2.22.0, <3.0 # download_3gpp dev dependencies black >=19.10b0 isort >=4.3.21, <5.0 pygments >=2.5.2, <3.0 # download_3gpp test dependencies pytest >=5.3.4, <6.0 pytest-cov >=2.8.1, <3.0 pytest-mock >=2.0.0, <3.0 # general dependencies for build flit >=2.2.0, <3.0
It is possible to dynamically install Python dependencies in the CI pipeline of your project but this will slow the pipeline down as it re-installs the dependencies on each job, although this might be improved by using caching within the pipeline context. In addition, installing the dependencies must be maintained for each job which can become a maintenance issue without care. Since the dependencies usually change relatively slowly, except perhaps at the beginning of the project and for the improved maintainability I consider it preferable to install dependencies in the docker image.
Another factor regarding dependency installation is version control. In the
above requirements I have constrained the packages to “the same major release”
with the exception of black which is in beta still. One problem with this
approach is that in the pipeline I often use
flit install -s to quickly
install the package under test and if updated dependency packages are available
they could be installed as well and this will delay the pipeline. In addition,
the security best practice is to pin your dependency packages to an exact
version because an adversary could introduce an issue into a new release which
would be automatically acquired within the pipeline and potentially your
deployment if you are doing Continuous Deployment. I had started pinning
dependency package versions but one problem I experiences was that multiple
packages included pinned dependencies of the same package at different versions
which caused installation to fail.
eg. My project depends on packages “A” and “B” which in turn both depend on package “C”. The problem is that “A” has pinned “C == 0.1.0” and “B” has pinned “C == 0.2.0”. So now “A” can be installed with it’s version of “C”, or “B” can be installed, but not both at the same time.
For this reason I have so far reverted to more open-ended release specification in dependencies until I can figure out how to resolve this problem.
- Use Semantic Version tags in the git repository for releases.
- Aspire to having no code difference between “feature branch” pipelines and “release” pipelines.
- Minimize pipeline execution time through design of the pipeline and deploying sufficient hardware resources for the pipeline execution.
- Maintain the pipeline docker image in a separate repository and install all your project dependencies in that image.