TLDR;

  • Kapitan (and presumably Ksonnet) is the more flexible and customizable (json and jsonnet)
  • Kustomize if the more straightforward, just released so we’ll need a bit more documentation on built-in functions (yaml only)
  • Helm combines a package approach and releases management that is powerful, with the caveats of Tiller for the release management part (additional source of truth)

UPDATE: Good article on managing complexity by Matt Farina https://codeengineered.com/blog/2018/helm-kustomize-complexity/

Overview

During the implementation of https://www.weyv.com environments in Kubernetes, we went through various stages. From plain yaml files, to Helm charts releases and finally helm charts but with helm template output. Now with the announcement of Kustomize, I take the opportunity to re-evaluate our choice of tool vs our requirements with 3 contenders: Helm, Kapitan, Kustomize. I left out Ksonnet (https://github.com/ksonnet/ksonnet) it seems very close to Kapitan.

Helm: https://github.com/kubernetes/helm

Kapitan: https://github.com/deepmind/kapitan

Kustomize: https://github.com/kubernetes-sigs/kustomize

Requirements

We need a tool for managing our Kubernetes resources that allows ut to:

  • Specify resources
  • Reuse and recombine above resources specifications across environments
  • Version control the desired state of resources

Helm

Specify resources

  • Set of resources: Chart
  • Definitions: Template files, Go template syntax
  • Variables: default values per chart, values override, values formatting

Pros: yaml templates are close to the yaml output that is sent to Kubernetes

Cons: No template reuse inside a chart , no custom class definitions

Reuse and Recombine

  • Chart can reference and use other charts
  • Sub charts variables can be overriden

Version control

  • The default usage is to do “helm release” and Tiller (server side in Kubernetes) manages the versions deployed to the cluster
  • Though the ouptut of a “helm template” can be checked in version control

Pros: Tiller can manage rollback of releases

Cons: Discrepancies between version controlled resources (desired state) and the version Tiller controls and manages. This created issues several times for us when the state of a resource is modified directly without using Helm and pushed us to switch to “helm template”.

Kapitan

Specify resources

  • Set of resources: Component
  • Definitions: jsonnet templates, json
  • Variables: hierarchical database of variables (inventory classes and targets)

Pros: jsonnet allows class, inheritance, hierarchical database of variable with 2 levels: one for reuse and one target specific.

Cons: json does not look like the generated yaml, json is a bit more verbose to type

Reuse and Recombine

  • Everything can be reused and recombined

Version control

  • The ouptut of a build can be checked in version control

Kustomize

Specify resources

  • Set of resources: Base
  • Definitions: plain yaml files
  • Variables: properties file can be used to generate configMap (configMapGenerator)

Pros: everything is yaml. input and output look like Kubernetes ready yaml

Reuse and Recombine

  • Reuse by default, patch resources in overlays folders
  • Recombine easily with declaration in kustomization.yaml

Version control

  • The ouptut of a build can be checked in version control

Conclusion

For our current situation moving from “helm template” to kustomize may make sense and require limited investment. Staying with helm, using only template functionality is viable but seems overkill since we don’t even import packages from other sources.

UPDATE: we stayed with Helm and use values file templates to generate values files and feed Helm template to generate our manifests.

Changing to Kapitan would give us more power and flexibility but would require an extensive rewrite of our helm templates to jsonnet.

What is your return of experience?

Feedback always appreciated