When I used Puppet I’ve discovered the units test and their uses. Working mainly alone, it was very useful for me to set them up with two main objectives:
Puppetlabs provides a development kit to help set up unit test with : pdk. This tool also allows full context to create your puppet moduls, we will find there the metadata.json, spec directory with helpers, a Gemfile, a hiera context…
This tool is very interesting and easy to use when you have one or two modules to manage, but it becomes much more complicated with 300. To manage and update Pdk, several solutions have been tested. From a simple bash script to a sinatra + sidekiq application, all solutions brought complications. Whether it is to manage the right pdk version, the ruby context or simply push into 300 git repositor at the same time.
After several tests I’ve finally created a solution with Gitlab CI templating. With this operation I already saw several improvements:
For the desired workflow I defined everything in a dedicated repo and there is only one ci file to include. With this one I will organize the rest, wich gives a result like:
---
default:
cache:
paths:
- vendor/bundle
# Define default variable
variables:
PDK_TAG: 2.5.0
IMAGE: pdk:2.5.0.0
GIT: /opt/puppetlabs/pdk/private/git/bin/git
PDK: /usr/local/bin/pdk
LAST_RELEASE: 0.0.1
# Include all my jobs
include:
- local: /templates/update.yml
- local: /templates/syntax.yml
- local: /templates/unittest.yml
- local: /templates/pages.yml
# Setup my stages
stages:
- update
- syntax
- unittest
- documentation
The different stages are presented in this way (we ignore the stage update for the moment)
Set up a syntax lint with rake tasks setup by Pdk, but also a little more checking of yaml files with yamllint.
---
syntax:
stage: syntax
image: ruby:2.7.2
script:
- bundle exec rake validate lint check rubocop
rules:
- when: always
yaml_lint:
stage: syntax
image: sdesbure/yamllint
script:
- >
if [[ -f 'hiera.yaml' ]]; then yamllint **/*.yaml --no-warnings; fi
rules:
- when: always
Launch of unit tests, this task will only be done during merge request or on the production branch. They run with rspec, a framework test for Puppet.
---
.unittest:
stage: unittest
image: ruby:2.7.2
script:
- bundle exec rake parallel_spec
only:
- merge_requests
- production
coverage: '/coverage:\s*\d+.\d+%/'
spec_puppet:
extends: .unittest
variables:
PUPPET_GEM_VERSION: '~> 7'
Note that this test is only for Puppet 7, but we can also use parallel-matrix to run it on Puppet 6 and 7.
spec_puppet:
extends: .unittest
parallel:
matrix:
- PUPPET_GEM_VERSION: ['~> 6', '~> 7']
And finally the last step used to generate the automatic documentation, with two different formats.
The first will be with Gitlab pages and yard/Puppet strings, but since people will rarely see the pages, we made a simplified markdown version directly in the repo. For this you need a token that has a read/write access to the repo, it will be defined as a variable in the environment variable named PROJECT_TOKEN_RW (of your group containing all your projets or to be done individually projects). The commit will only be made if there is a difference in REFERENCE.md, and in this case the -o ci.skip option will be passed on the push to avoid relaunching the CI.
---
pages:
stage: documentation
image: ruby:3.0-alpine
script:
- gem install puppet yard puppet-strings
- puppet strings generate
- mv doc public
- apk add git
- puppet strings generate --format markdown
- git add REFERENCE.md
- "[ -z $(git status --porcelain REFERENCE.md) ] && exit 0"
- "git config --global user.email 'dev@nu.ll'"
- "git config --global user.name 'Pdk Ci'"
- "git remote set-url origin https://git:${PROJECT_TOKEN_RW}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
- "git commit -m ':books: update REFERENCE.md'"
- "git push -o ci.skip origin HEAD:${CI_COMMIT_BRANCH}"
artifacts:
expire_in: 300 seconds
paths:
- public
only:
- production
All of this is relatively easy to set up and you just need an include like this for it to work:
---
include:
- project: <votre repo>
ref: main # Or your default branch name
file: templates/pdk-ci.yml
But the main goal was to put a simplified management of PDK. For that I added a check use to notify if a pdk update is required, and I automated the update (and installation) on the CI.
At first, I created a file template/release.yaml who simply contains a variable RELEASE, and in the CI I included the template with a tag and not on the main branch.
template/release.yaml:
---
variables:
RELEASE: 0.0.1
Then in the file templates/update.yml I’ve:
---
.pdk_update:
stage: update
image:
name: puppet/${IMAGE}
entrypoint:
- ''
script:
- if [[ ${PDK_RUN} -eq 0 ]]; then exit 0; fi
- ${GIT} clone --branch ${RELEASE} ${PDK_TEMPLATE} /tmp/pdk_template
- mkdir -p manifests
- cp /tmp/pdk_template/configs/sync.yml .sync.yml
- mkdir -p ~/.pdk/cache/
- cp /tmp/pdk_template/configs/answers.json ~/.pdk/cache/
- ${PDK} ${PDK_COMMAND}
- cp /tmp/pdk_template/configs/gitlab-ci.yml .gitlab-ci.yml
- touch pdk.yaml
- ${GIT} add .gitignore
- ${GIT} config advice.addIgnoredFile false
- "${GIT} add -A . || :"
- "${GIT} config --global user.email 'dev@nu.ll'"
- "${GIT} config --global user.name 'Pdk Ci'"
- "${GIT} remote set-url origin https://git:${PROJECT_TOKEN_RW}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
- "${GIT} commit -m \"${GIT_COMMIT}\""
- ${GIT} push origin HEAD:${CI_COMMIT_BRANCH}
install_pdk:
extends: .pdk_update
before_script:
- PDK_RUN=0
- >
[[ ! -f '.sync.yml' ]] && PDK_RUN=1
- export PDK_RUN=${PDK_RUN}
variables:
PDK_COMMAND: "convert --force"
GIT_COMMIT: ':construction_worker: convert to pdk module'
only:
- production
update_pdk:
extends: .pdk_update
before_script:
- PDK_RUN=0
- >
[[ -f '.sync.yml' ]]
- >
[[ -f '.gitlab-ci.yml' ]]
- >
[[ "${LAST_RELEASE}" == "${RELEASE}" ]] || PDK_RUN=1
- export PDK_RUN=${PDK_RUN}
- export RELEASE=${LAST_RELEASE}
variables:
PDK_COMMAND: "update --force --template-ref=${PDK_TAG}"
GIT_COMMIT: ':construction_worker: upadte pdk'
only:
- production
With this comparison [[ "${LAST_RELEASE}" == "${RELEASE}" ]] || PDK_RUN=1 I notify the PDK update, I also check if the file .sync.yml doesn’t exist to run a Pdk installation. To create a new release it will be necessary to go through these 4 steps:
gitlab-ci; this tag will have to be copied (and stocked on configs/gitlab-ci.yml)RELEASE variable in templates/release.ymlLAST_RELEASE variable in templates/pdk-ci.ymlAfter, your final .gitlab-ci.yaml in your puppet module looks like:
---
include:
- project: <votre repo>
ref: main # Or your default branch name
file: templates/pdk-ci.yml
- project: <votre repo>
ref: <your actual tag>
file: templates/release.yml
To finish, if you don’t want to wait for your code to be changed on the production to update Pdk, you can use pipeline schedule (for my part I put it randomly once a week when creating the repo).
WARNING: In case your repo containing the templates changes, it’s very important to keep the current repo the time to make a new release with the right references in the
gitlab-ci.yaml.