🇫🇷 🇺🇸
6 Mar 2023

Choria brokers, kubernetes and Vault Ca

Introduction

When upgrating Puppet 5 to Puppet 6, one of the biggest changes for us was the removal of mcollective and ActiveMQ. Fortunately it wasn’t a total disappearance but a ful rewrite and a migration to choria, which offers much more possibilities.

Like Mcolvective, Choria is an orchestration tool working on a principle of client <-> broker <-> servers. Unlike the usal client/server models, Choria differs in the fact that we install servers on all our nodes to be managed which will be connected to one or multiple brokers. After a client sends an action message in the broker queue to be consumed by the servers. All of the work is mainly based on the nats nats librairy, and uses a x509 authentification, making the set very powerful with a low infrastructure cost (the documentation indicates the cost of a simple broker with 4G of RAM for 50k server to manage).

Select the architecture

In our old mcollective architecture, we used an activemq cluster installed on 6 Puppetservers, then downgraded to 3 active nodes due to a useless and high connection consumption on the haproxy. For Choria we decided to start on the same architecture, and we decided to use the basic architecture, but we set up 3 brokers for a high availability.

Just before setting up the production I’ve decided to add two more targets:

  1. Implament the solution on a kubernetes onepremise context (to levelup ont the subject, and in a second step to move all the Puppet infrastructure to kubernetes)
  2. Use the Puppet Ca in vault (unlike the actual mcollective which uses it’s own CA with just a simple pair of certificates)

Set up in Kubernetes

The set up in Kubernestes is quite easy to access, a helm chart is available and contains pretty much everything that is necessary. It iss good to note, that for some reason I don’t know (probably related to the creations of the certificats), that your brokers are declared with StatefulSets. This can be usefull for your pod assignation.

The most complicated part is the Vault Ca integration. In fact the helm chart already proposes a CA management, but that’s not enough because it is a CA stored on a simple secret. In my case I’ve used cert-manager to declare mya Vault issuer with an appRole authentification:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
spec:
  vault:
    path: {{ .Values.vault.path }}
    server: {{ .Values.vault.addr }}
    auth:
      appRole:
        path: approle
        roleId: {{ .Values.vault.roleId }}
        secretRef:
          name: cert-manager-vault-approle
          key: secretId

Before it is necessary to have:

  1. Imported your Puppet Ca in vault
  2. Created a right vault policy with ["list", "read", "create", "update"] permissions
  3. Created an approle with your policy
  4. Created a secretId used for token creation, this one whould be copy on a secret named cert-manager-vault-approle with a key secretId.
  5. Added your values on the values.yaml, ex:
---
vault:
  addr: https://vault.wtf.org
  roleId: db02de05-fa39-4855-059b-67221c5c2f63
  path: pki_puppet_ca/sign/puppet
  1. Add the RBAC permissions, because the certificats are directly created by your pods with the cert-manager API. I’ve simply copied what is used by the choria ca chart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: choria:csraccess
rules:
  - apiGroups: ["cert-manager.io"]
    resources: ["certificaterequests", "certificaterequest"]
    verbs: ["get", "create", "delete", "post"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: choria:csracess
subjects:
  - kind: ServiceAccount
    name: choria-csraccess
    apiGroup: ""
roleRef:
  kind: Role
  name: choria:csraccess
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: choria-csraccess
  1. Add the issue reference for your broker configuration in values.yaml
broker:
  broker:
    certManagerIssuer: vault-issuer

Usage

For use, thes choria clients must use an x509 certificate created by your CA with a Common Name related for your policies. Rather than creating certificates by ops and changing the policies everytime, I’ve created a dedicated host for client use and filtered by AD groups on the connection. Every userbthen had sudo rights to switch to choria, was authorized to read the certificate that I generated with the puppetserverw puppetserver ca generate --certname choria.mcollective.

We have a very basic usage of choria, principaly for packages listings or enable/disable puppet agent in few ms, but there are a lot of possibilities that I couldn’t test, like creating playbooks, use the tasks puppet or have a more advanced federated architecture.

When I created this environment, I learned that the Common Name on x509 certificates has been deprecated since May 2000 via RFC2818, but this field is mandatory when creating a certificate in a vault pki.

Tags: