okay good morning
I guess still morning just for a minute
so welcome to the last day of NDC Oslo
2023 hope you've had a great conference
so far I know I have
my name is Jeff French and uh I'm going
to be talking to you today about
managing kubernetes the get Ops way with
flux
now this presentation has some nice
interactive elements to it so if you
would like to well if I can advance my
slides
if you would like to join the fun and
play along there will be some times
where you can kind of vote on some
things or you know give your input this
will also allow you to ask questions
throughout the presentation by simply
typing them in so you can scan the QR
code or you can go to mentee.com and put
in this code and you can do that on your
phone on your laptop
smart fridge whatever you brought with
you
so let's start
with the fun
which is your favorite pizza or tacos
these are the kinds of pressing
questions that we come to NDC to answer
right
let's see here oh what pizza is out into
a heavy lead
tacos trying to make a comeback
all right
let's close that voting up in about 30
seconds
man I'm really pulling for tacos I wish
I could vote because tacos are my
favorite
I need that oh here they come all right
tacos making that late showing I like it
so
okay
and these uh the website and the code
will be at the top of the presentation
most of the time so you can type it in
and join if you haven't already joined
the fun but here we go coming into the
home stretch and it looks like tacos is
going to take it
all right yay tacos
so as I said my name is Jeff French uh I
worked for a company called Moon switch
in the states we are a devops consulting
firm specializing in Cloud migration and
modernization managed cloud and
kubernetes devsecops this is what we do
all day and this is why I'm here to talk
about githubs because it's what I do all
day and I love it I'm passionate about
how much it has had a positive effect on
the way that I manage environments for
our company and for all of our customers
and so the things that I'm going to talk
to you here about about here today are
things that have been battle tested in
production with multiple clients
multiple
workload types multiple clouds even
so one more fun little interactive one
let's find out where you already are
with some of these Technologies
we'll just take about 30 seconds to vote
on that real quick
so
now to give you a little bit of my
background I've been practicing devops
since kind of before we had the word for
devops in fact I'm kind of excited to be
speaking here at NDC Oslo 2023 again
because the first conference
presentation I ever gave was at NDC Oslo
2013 where I was talking about deploying
complex apps continuously with octopus
deploy
and so for the last 10 years I have been
doing
what became known as devops over that
time with continuous delivery and
enabling application teams to deliver
their applications to
to the cloud or to their servers uh you
know efficiently
all right so it looks like we are kind
of across the Spectrum on kubernetes
experience that's cool and um
staying low on the customized helmet
flux well that makes a lot of sense
that's why you came to this talk right
you want to learn about these things
um
so
before we get into flux and get Ops
itself we've got to lay a little bit of
a foundation with some of the other
technologies that are involved in
managing a git Ops configuration with
flux and the first of those is customize
so
customize
is a standalone tool for customizing
kubernetes objects through declarative
configuration language
um and it's been built into Cube control
since version 1.14 it also has its own
Standalone CLI as well
um and this is maintained written and
maintained by the kubernetes team
and customize its whole mission in life
is to gather up your kubernetes
resources
make small modifications or to them and
then help you apply them to your cluster
so what are the benefits we've got
obviously declarative configuration
management that sounds really good if
we're trying to do get Ops right because
we're going to want to declare exactly
the state that we want our clusters to
be in reduced duplication as we'll see
customize can help you organize your
application and all of your kubernetes
manifests in a way that helps you keep
them a little bit drier and keep them
more consistent across environments and
it helps you with the separation of
concerns across your different
environments or different applications
so
basics of a customized configuration or
a customized app as I like to call them
are number one
a customization file it's basically
always called customization.yaml
and it is a definition that tells the
customize CLI
how to transform your application and
how to transform your resources before
you actually apply them to your cluster
in this file you will Define which
resources you want
and then we'll also you can organize
these into some interesting structures
called bases and overlays
okay this is where customize starts to
help you dry out your configurations a
little bit you can create a base that
has all of the common configuration for
your application that's going to go to
every single environment right so in
your base you might Define a
deployment.yaml that defines a
kubernetes deployment probably a service
and a service account maybe some config
maps all these things that it takes to
deploy your application into any
kubernetes cluster in your environments
then we've got overlays so overlays is
where we actually store environment
specific modifications to that base
your overlays will reference a base and
say Hey I want you to go get everything
that's in that base and then I want you
to apply these changes to it
so some of those changes and things that
customize can apply from those overlays
or in the base
patches patches are a really really
helpful one and customize it lets you
say hey I know we defined this this way
up in the base but for the dev
environment it needs to be just a little
bit different
or for the prod environment it needs to
be slightly changed some of those common
modifications might be let's say you
define an Ingress in the base of your
customization
well in your Ingress you probably need
to define a host name for that Ingress
and that host name is probably going to
be different in Dev and stage and prod
right so patching there will allow you
to say well I can Define most of my
Ingress at the base because the service
names and the ports and the paths and
most of the configuration is actually
the same and I don't want to have to
repeat it everywhere
but I just want to change this one
little piece so you can write a patch
and customize that allows you to
modify just that another option might be
say changing the replicas maybe we only
want to run one pod for a deployment in
Dev but we want to run a hundred in
production
uh config maps and secrets this is
another one that I really like to use
and customize because it can generate
them it can either make them up out of
nowhere based on some values that you
give it or it can you take existing
config Maps or secrets and modify them
and this is a really really powerful
tool and customize
one example of this would be if you've
ever deployed a
deployment a pod on kubernetes and
configured it to read maybe a file or
even better an environment variable from
a config map or a secret and then it was
time to change that environment variable
so you went and you updated the config
map and you put the new value in
great and then you go back and your pod
is just still got the old value
and we know why that is if you're on the
right hand side of that kubernetes
Spectrum you probably already know this
if you're on the left hand side you have
to go restart that pod that pod reads
its config map value at startup and it's
not going to change the environment
variable once the Pod is running so you
have to go roll out your deployment or
re delete your pod and let it restart so
we can pick up that new value and that's
really easy to forget to do when you're
checking this in to get and letting it
get applied automatically well one of
the cool things that customize can do
and does this by default is if you let
it generate a config map then it will
actually change the name of that config
map it will append a hash of the
contents of the config map or the secret
to the end of the name and that way
every time you run your customization if
that hash has changed because the
contents have changed it will change the
name
and that sounds great but now you've got
to reference a new config map name in
your deployment right well customizes
got you covered it knows where config
maps and secrets get used in all the
standards standard kubernetes objects
like deployments pod specs on stateful
sets Daemon sets jobs cron jobs and so
it will go and say hey every place that
you reference this config map by this
name I just gave it a new name so I'm
going to go update that for you and this
has the really nice effect of causing a
change to your pod spec on your
deployment or stateful set which will
naturally trigger your deployments
rolling upgrade whenever it gets applied
so now by simply making a change to the
config map and then applying it with
customize it will automatically make
sure that all of your pods pick that
change up correctly
it's a very very powerful tool
you can also do some light variable
substitution I wouldn't get too deep
with that but it can be helpful in a few
small scenarios but usually a patch is
probably a better approach in most cases
so
the root of our customization is a
customization file as you can see it
looks just like a kubernetes object
right we Define it in yaml just like we
would a deployment or a service or a
config map and here we're defining that
we've got resources and we're just
telling it hey pick these up now these
as you might guess deployment service
Ingress those are just yaml files that
have
standard kubernetes yaml in them they
could be applied directly to your
cluster using Cube control apply Dash F
and passing in each one of those file
names and they would be applied straight
to your cluster
customize takes those and is going to
actually combine them into a single yaml
output for you so you can either run
customize and see what it's going to do
and have it actually generate a full
output that you can inspect you could
even take that and rather than letting
Cube control apply apply it for you you
could save that as a full output and
apply it yourself
now I mentioned that
Cube control apply Cube control has
built-in support for customize and we
see this in the bottom of this slide
here where you can use Cube control
customize as a command to actually run
your customization and see what the
output is going to be test that it's
actually updating your config maps and
things like that
um
or you can also just run Cube control
apply but instead of Dash F you can say
Dash K and feed it a customization file
or a directory that contains a
customization file and Cube control is
going to automatically go and run the
customization render it into its final
form and then apply it to your cluster
now here is uh a common way that I lay
out most of my kubernetes or my
customized applications on the right
here and as you can see we've got a pod
Info app and I always like to define a
folder called base and put all of my
base resources in there that are going
to be common for every environment and
then a folder called overlays and under
that a folder for each environment that
I'm going to deploy to Dev stage prod QA
wherever
and in our base we'll Define a
customization.yaml that defines most of
what is going to be picked up and
generated and made for our app
and then down in each overlay
customization we put in just the parts
that need to be different for that
environment okay and that might be
generating or merging a config map or
secret to set a special value that might
be putting a
putting a patch in there to change some
value before it gets deployed another
nice feature of customize is that it can
automatically update labels and
annotations it can help you avoid naming
collisions by doing name prefixes right
and so
one common use is using say common
labels to say well okay I want all of
these labels on my base for every
deployment I want to know what app it is
and maybe all these other things but for
a specific environment I want to know
that its environment equals Dev right
and so I can go apply that at the dev
overlay and that label will get applied
to all of the resources that customize
processes in that configuration
okay so so that's the basics of
customize right and we can already start
to see how okay well if I can have all
of these yaml files and I can make them
special for each environment I can then
obviously check these into a repo you
can see how this starts to lay the
foundation for git Ops right because we
are going to store all of our
configuration in a git repository
customize is a great tool to help us
achieve that
Okay so
lots and lots of fun info about
customize
let's get back into some of our fun
interactive stuff
what have you enjoyed the most
at NDC so far
oh man it's hard to say but I think my
favorite was probably the continuous
food it's very nice to be at a
conference that's always always help you
know making food available
and coffee pretty good coffee too
yeah the food's making a strong oh oh
people are here for the sessions uh you
guys came to a conference for the
sessions
I'm very happy to hear that as a speaker
all right just a few more seconds
and
yeah all right I always continuous food
gonna make a late push
let's see okay oh oh continuous food
coming in at the wire all right I knew
it would get you
okay so another technology that we want
to make use of when we are
exercising get Ops with flux is hell if
you are operating kubernetes clusters
you're probably familiar with Helm but
according to our pre-workshop survey
um you know not as much so let's dive
into Helm just a little bit
so Helm is often referred to as the
package manager for kubernetes okay you
could kind of think of it like uh it's
not the same as but similar to something
like apt or yum for operating systems or
maybe npm or nougat for application
dependencies right it's a place where
people can publish Helm charts that have
a configuration that is ready to deploy
onto your cluster okay they help you
manage it kubernetes applications and
they help you define and install things
so what is a Helm chart okay it's
bundles of pre-configured kubernetes
resources
and it can be deployed multiple times
inside of a cluster and most importantly
it supports go templating for
customization so we just talked about
customize and how it could gather up all
of your
kubernetes resources and help you make
some modifications to them so why do we
need help well first off Helm is a
little more powerful mostly because of
the go templating you can do a lot more
powerful things inside of a Helm chart
to vary the output of the helm chart
based on inputs that we'll get into here
in just a little bit
um but also the helm chart is a very
good way to package what I like to
consider say off-the-shelf software
um in in kubernetes land what I mean by
that are basically things that I didn't
write okay
um I don't want to have to maintain a
whole bunch of kubernetes uh object
definitions for my Ingress controllers
or external DNS or assert manager any of
these things right I would like for the
maintainers or even vendors of those
products to maintain that for me all
right but they have to be able to
support a wide range of options so Helm
gives them the power to package this up
in a way that's easy for you to install
on your cluster with a little bit of
variance based on some inputs and be
able to manage those efficiently
so what is a Helm repository a home
repository is you guessed it it's where
we store Helm charts right and it's
really just kind of a folder with an
index.yaml in it that points to a bunch
of tarballs of all of the different
versions of a Helm chart that are
available
so some of the comment Helm commands and
these become important because we're
going to automate them uh our U.S Helm
installed to install a chart Helm
upgrade to upgrade a release they're
very straightforward but you can see how
this really works very similar to other
package managers right you simply Say
Hey I want you to go to some repository
of packages I want you to find this one
that I'm interested in and then I want
you to perform these operations on it
so
one of the most important parts about
Helm for our purposes is understanding
the values files we're not going to get
into how to create home charts here
today that's well beyond the scope of
this talk but we are going to talk about
how to deploy them and values files are
critical for that
so value files are yaml files that are
used to customize the configuration of a
chart okay every chart has a default
values.yaml file right in the root of it
that defines all of the default inputs
there those inputs are used in the go
templates inside the chart in order to
alter the actual resources that get
created based on those inputs all right
so you can then provide your own custom
configuration without needing to modify
the chart itself
so you can specify your values in a lot
of different ways during installation
my favorite and the one that we will be
kind of talking about a lot today is
using the dash F or values flag this
lets you have your own yaml file that
defines the inputs that you want to
Define for that chart
and set the appropriate values and then
pass it in on the command line and Helm
will read all those values and apply it
to the chart
and the other fun part here is that
multiple value files can be passed in so
you could say Dash F values Dash F
values and pass in multiple charts and
what will happen is that
Helm will munge all of those together
all right so anything the last file you
pass in ends up winning it's a last and
wins kind of command line option and
therefore if there is the same key in
your yaml file defined in the first
values file that you pass in and another
one defined in the second values file
that you pass in and they're the same
key then the last one is going to
overwrite that it's going to clobber the
first one and this is going to become
very important for us in the way that we
maintain bases and overlays
so as I said this is what that ends up
looking like you can run a Helm upgrade
with a dash dash install so it says Hey
upgrade this if it's there install it if
it's not and if I pass in my values and
my other values anything that is in my
other values
can clobber what is in the my values
file but anything that is doesn't have
the same key will simply get merged
together into a single yaml file
and finally when we're running these
commands we give it a release name so
this is what's going to be used to
identify the release in your cluster and
then we're telling it hey here's the
helm repository that you should look for
this in and here's the helm chart I want
you to find there's lots of other
command line options you can pass
inversions and all of that
so some tips and best practices on these
store your values file in Source control
so you have a record of all your changes
right they're just yaml now again apply
all of your normal code repository
storage uh security measures if the helm
chart takes an input for say a database
password don't check that into your
repository please
um there are other ways to handle that
and we'll talk about that a little bit
when we get to flux
um you can use a hierarchical structure
in your in the yaml file to organize
everything and
lint your yaml files
um they are yaml if they are not valid
hell won't know what to do with them and
it might fail in some really interesting
ways
all right
let's have some more fun
this one might be spicy
yeah I thought maybe I'd go with uh you
know tabs versus spaces but uh
that's that's an old fight nobody wants
to have that fight anymore
uh oh here we go AWS making a late
comeback
oh I forgot to start the timer well
we don't need to wait that long but
why not 30 more seconds
so
I I'm not surprised to see azure's out
strong I know Azure is a very popular
Cloud among NDC attendees
I uh we do work on both Azure and AWS
and uh you know they both serve their
purposes
all right well it looks like uh
although I will say this
[Music]
doesn't look like my favorite is going
to win this time though uh want
Okay so
the main event this is what you came for
what is flux what is get Ops
let's dig in
Okay so
we'll start by just defining what get
Ops is okay
that's better
so as you can see to get based approach
to managing infrastructure and
application deployments serves as a
single source of Truth for your
infrastructure and application
configuration
what are the benefits of git Ops and and
real quick let's talk about just the
word get Ops get Ops is kind of like the
word devops okay it's not a thing it's
an idea it's a collection of practices
it's a culture it's a mindset okay you
can achieve get Ops with lots and lots
of different tools okay today we are
going to talk about flux but there is
also Argo CD that's a very popular git
Ops framework in the kubernetes
community you can also achieve the the
benefits of git Ops with just your
regular CI CD system
by simply following all and you gain all
of these benefits
right we get consistency we want to make
sure that our environments are
configured consistently right just like
we want our Dev environment to be as
close to our prod environment as
possible so that we can test our
applications correctly understand how
they're going to behave right
collaboration
okay because we are storing all of these
in a git Repository
we get all the benefits of say pull
requests when someone wants to make a
change to some of our configurations
they can submit a pull request we can
have a discussion about what that change
means what its impacts are why that
change is being made make sure it's the
right change for our organization or for
our application
and then of course it provides us an
audit Trail right we can come back and
see what changes have been made by who
when and why
and it helps simplify rollbacks if
things go wrong right we could just
revert that commit go back to our last
state
so
there are a few principles of get Ops
that will kind of help guide you if you
are trying to implement get Ops with
flux or any other tool set
declarative configuration
Version Control and automatic
convergence
so
first principle declarative
configuration Define everything in
declarative configuration files much
like what we've just been looking at
with customize and Helm there are ways
where we can take our desired
configuration State express it in a file
that can be checked in to a git
Repository
that way we have and by doing so we are
stating this is our desired state of
this environment or these environments
we're saying this is how we want things
to look
Version Control
store them in aversion control system
probably get it's kind of the most
popular one these days it's right there
in the name get Ops but
spoiler alert if you are
using something else you can still
achieve the same get Ops principles with
I don't know SVN or something like that
if you're still using something like
that the idea being that as long as you
have a some sort of source control
repository where you can store your
configuration files and create that
audit Trail and that collaboration and
that consistency you can still achieve
get Ops you're not going to be able to
use flux to do it it really only works
with Git and S3 but
and finally
automatic convergence
all right this is the real key to get
Ops automatic convergence it's one thing
to say I've declared my state in a
repository and and we said that that's
what we want and maybe our CI system ran
and applied that state to our clusters
but if it's not continuously doing that
you're not really achieving get Ops
okay automatic convergence is using some
kind of tool to constantly be
reconciling the state of your cluster
with what is defined in your Repository
Okay so
flux is a tool that's really going to
help us achieve that but again you could
achieve this with a CI pipeline as long
as you schedule that CI pipeline to run
maybe every five minutes or 10 minutes
or something like that so that it's
going back and applying these states
all right so
that brings us to flux
so what is flux open source git Ops
toolkit for kubernetes developed by
weaveworks and part of the cloud native
Computing foundation
and it monitors git repositories and
automatically applies their changes to
the cluster
um
I'm not 100 sure but as far as I know I
think
the flux team kind of was one of The
Originators of the git Ops term I could
be wrong and now that I've said it at a
conference session I'm guaranteed to be
wrong and someone will correct me but I
like to learn so please do if I'm wrong
um
I think a quick note on the cncf I don't
have an internet an interactive slide
for this one but quick show of hands of
who's familiar with the cloud native
Computing Foundation
okay maybe 50
so the cloud native Computing Foundation
is a division of the Linux foundation
and it's basically a board that helps
kind of identify up and coming projects
that are good in open source uh for
cloud native Computing objectives and
sort of guide those projects through
their governance model and how to build
their Community until they get up to a
state where they are actually considered
a graduated project and what's really
cool about that is it means that they
are kind of making sure that these
projects have
good open source governance standards
they've got active committed maintainers
they've got a thriving community of
people who are both contributing to and
using the project the idea being that if
if a project is in a cncf incubator
stage it's pretty good and if it's in a
graduated stage you can be pretty
confident in taking a dependency on that
project because it's going to be well
maintained over the course of time
kubernetes is probably the most
well-known and first graduated cncf
project flux is a graduated cncf project
there are lots and lots and lots of
others on their website
so
what is flux what are the components of
it it is a set of controllers and crds
that get installed into your kubernetes
cluster okay so when you bootstrap a
cluster with the flux CLI it goes and
applies a whole bunch of crds for things
like git repositories and Helm
repositories and customizations and Helm
releases
um that it knows about and then it
deploys a set of controllers that know
how to actually interact with those
objects as they get defined and take
action based on them
okay so there's a source controller
customized controller Helm controller
notification controller and image
automation controller it's actually two
controllers
let's dig into them a little bit so
this is a little diagram of how the
source controller works so as we can see
on the left side of the slide here there
are some crds that get deployed with
flux git repository Helm repository and
bucket and they are definitions of how
to where to find and how to access each
one of those types of objects and that
lets the source controller go and clone
a git repository and constantly sync
from it or
uh load a Helm repository and constantly
sync all of the charts that are in it or
load a bunch of files from an S3 bucket
those objects get created with the
kubernetes API The Source controller
sees that and then goes and follows
those instructions and says hey I know
how to do this I'm going to go get it
and then there are consumers which are
the other flux controllers of those
artifacts that are they're going to take
action based on the idea based on the
fact that there are new things found in
these sources okay so and they will then
produce artifacts that can be deployed
throughout our cluster
the customized controller
it's aware of git repositories and
customizations
okay and so its whole job is to take
those customizations that we defined
earlier with customize and actually
continuously apply them to our cluster
right so we take all those
customizations that we wrote for our app
we check them into a git Repository
that git repository is registered with
flux The Source controller sees an
update to it notifies the customized
controller the customized controller
says oh I know what to do when there's
new files converge automatically
automatic convergence and it picks up
and runs those
uh
it picks up and runs those
customizations and applies them and
ensures that and creates all of the
normal resources you would have in your
kubernetes cluster crds namespaces are
back all of these things
the helm controller
I think you can see where this is going
now it does the same thing with Helm
charts right so we can then
Define these Helm repository and Helm
chart and Helm release crds in our
in our git Repository
and when we apply those to our cluster
the helm controller will see them and
say oh
I know what to do with this you're
telling me that you want me to go get
this Helm chart out of this Helm
Repository
apply these values to it from values
files
and
run that on and install it in the
cluster
this one also includes a die a part of
the diagram that's really interesting
and I just took these straight from the
flux website so I didn't make these
diagrams
um but they actually all do this the
notification controller which we'll get
into in a minute but that's the
interesting part is that it can give you
notifications about things actions that
are being taken by the other controllers
now what have we done
okay
the notification controller
um so the notification controller can do
two things it can receive
web hooks
and then it can dispatch notifications
so the receiving of web hooks is kind of
cool because while the source controller
if you give it a git repository it will
on an interval that you specify go and
pull that repository to find any changes
um
but you can also register a web hook
receiver with the notification
controller and then you can register
that web hook with your say GitHub
repository or gitlab repository and that
way rather than the source controller
having to pull and wait five minutes or
however long your polling interval is it
can instead just receive a web hook
and know that it can go and reconcile
those changes immediately
and then
once the other controllers take actions
they will such as a customize apply or a
Helm upgrade or install those will
surface events to the notification
controller and you can then register
dispatchers that will dispatch those
notifications to various places like
slack teams Discord rocket chat there's
a whole host of places I think you can
also just have it drop a web a web hook
to something
all right and then
the image automation controllers so by
default flux doesn't actually deploy the
image automation controllers you have to
kind of opt into that it's a little more
advanced use case
um and it's Works in a way that you know
you have to kind of determine your
organizational appetite for risk on on
what you're letting things be automated
for but the image automation controller
uh it has a reflector controller and an
automation controller the reflector
controller fetches tags from Docker
repository so you you define Docker or
oci repositories it will constantly
check them and see when there are new
tags pushed to those
you can then write an image policy
that says Hey whenever there is a new
tag in this Docker Repository
that matches this criteria I want you to
select that as the new tag
it doesn't do anything with it yet we'll
get to that in a second but
you can Define that and it can be based
on
either
semantic versioning it can be based on a
lexical sort you can tell it I always
want whatever the newest one is and
there are different reasons to do those
for different environments but once it
receives a new tag
it will then
go and read configurations that you have
in your git repository called image
update configurations or automations and
take action on those and the common one
is by placing a specially formatted
comment that you can see on the flux
website into your yaml configurations it
can go update Docker tags so you can
either put this special comment right on
your deployments or stateful sets or
jobs where you want a particular Docker
image tag to be updated or you can use a
feature of customize that will update
image tags throughout your customization
by specifying it in your customized file
or you could maybe put this comment in
values.yaml file for Helm that specifies
an image tag that you want updated
and whenever that policy that you
defined let's say it was semantic
versioning sees a new tag that is newer
than the previous one and meets your
criteria the automation controller will
go and find that file or all those files
in your repository make the change to
that tag and then commit it back to git
now we can do so on the same Branch it
read it from which is going to trigger
things to just roll out automatically
right away or you can have it commit to
a separate branch that way you could use
a pull request process in order to uh to
actually approve that change
but when it Go the important thing to
note is it does not actually make
changes in your cluster it makes a
change to your actual declared
configuration in the git repository
commits it back and then on the next
flux sync cycle flux says hey there's
been a change to this image tag I'm
going to apply it via customize I'm
going to apply it via Helm right it's
still maintaining its own get Ops
principles of hey we're not just going
to do magic stuff here
with some minor exceptions ah
but uh we're going to actually declare
this stuff the one minor exception I
would put in there is that when you are
defining semantic versioning for a Helm
chart to be released you can use
semantic versioning selectors there and
sell and tell it hey I want you to just
go ahead and install a new version of
this Helm chart whenever these
conditions are met one I'll do with that
a lot is for projects that I kind of
trust like maybe uh Ingress in Gen X or
you know cert manager that have shown a
pretty good history of making reasonable
semantic versioning choices I might go
ahead and let them automatically install
patch versions of those charts just so
I'm kind of getting updated bug fixes
and
um and security releases without having
to think about it but for minor versions
or major versions I want to control that
okay so we talked about the customized
controller
and that there's a customization crd
so
an unfortunate Choice was made in the
beginning of the flux project
you know I can see how they got here
they said hey we want to have a
controller that needs to go and deploy
customizations that are built for
customize
and so what are we going to call our crd
that tells our system how to deploy a
customization we're going to call it
customization I really wish it was
called a customization release or
something like that but it's not
so as you can see here if you're looking
closely we have two different kubernetes
objects that each have a kind called
customization
they differ by their API version
the one on our left is
customized.config.ph.io this is the
regular customization that we talked
about at the top of the presentation
okay this is the one that can go and do
all those customized things
the one on the right is flux's version
okay and it represents a set of
instructions that tells flux how to go
find the other customization and apply
it
it's a very unfortunate naming overlap
but uh
[Music]
but we will
just have to live with it probably until
flux V3 at least I don't know if there's
any
momentum on getting it changed in the
community or not but that's what it is
also I guess real quick I do see someone
asked a question about crd
so a crd in kubernetes is a custom
resource definition okay and that is
basically a way that
um
anybody but generally open source
projects or vendors can create new types
of kubernetes objects right these are
both examples of crds well
customizes is technically uh the
customization on the left is built into
kubernetes because it is built in by the
by the kubernetes team but the one on
the right is an actual crd a custom
resource definition so this is where the
flux team wrote and oh if you've ever
seen a crd
if you haven't seen a crd don't they are
literally thousands in some case tens of
thousands of lines of yaml to Define
these things
you don't want to get into that you
don't write them by hand they get
generated from go Lane code but you
define customizations and it's basically
a set of instructions to the kubernetes
API that says hey here's a new object
that I want you to be able to maintain
on our cluster all right I want to be
able to create these types of objects
and I want to have things respond to
them right so
all right so
let's dig into
one of infinite possible repository
structures when you're using flux
okay this is the one that I use all the
time this one has served me well for
several years it has been battle tested
and we'll take this through top to
bottom just a little bit
what I like to I like to refer to this
as a platform repo okay so typically
whenever especially if you are say a
platform engineer that's supporting
multiple Dev teams throughout an
organization it can be a good idea to
have your own repository that defines
all of your configurations for your
different environments for your
infrastructure in a platform repo I will
typically store all of my flux
configurations like this as well as
terraform that's beyond the scope of
this talk but here's what we've got so
I'll usually have a top level folder
called apps and under that I place you
guessed it apps here we've got an
example of this pod Info app that has
the same structure that we identified
earlier we've got a base with all of our
common deployment and service resources
and a customization this is a customized
customization
that defines how to pick up all those
resources and then we've got overlays
for Dev stage and prod whatever other
environments we might have and each one
of those has a customization another
customized customization that references
the base and says here's how you deploy
pod info for Dev or stage or prod
and then we would have other app folders
for all of our other applications back
in front end whatever all these
applications are that need to be
deployed to your cluster
we put those there now a common question
I get is like well does that mean we're
going to a mono repo no not necessarily
all I store in this is just the yaml
files for kubernetes I'm only storing
the things that tell how to deploy the
kubernetes objects the application
source code is not here you could
replicate this structure inside each one
of your application repositories if that
works better for you and you can have
flux point at each one of those
different repositories to find what it
needs to be applying okay I prefer based
on the way that I interact with my
systems and my client systems to have it
all in a single repository that is
decoupled from the application code
I then have a cluster Services top level
folder Okay so
cluster Services is something I Define
as basically all the stuff that needs to
run in your kubernetes cluster that's
not your actual applications right so
Ingress controllers cert managers
cluster Auto scaler you know maybe
Prometheus and grafana for monitoring
log collection all of these kinds of
things are what I call cluster Services
right they are sort of the things that
are helping your applications be able to
actually run in a kubernetes cluster and
I'll store those there but you'll notice
we're using the exact same structure
here
but you'll notice I've got a helm.yaml
here for cluster Services I almost
always try to deploy those with help
right those are full-on projects that
are maintained by teams outside of my
organization that
I want to just be able to install via a
Helm chart
all right and then
we'll see down here there's overlays
where I can customize each one of those
I've got a values.yaml file here in the
base I can also have a values.yaml file
down in the overlay so that I can set up
the exact type of command that we talked
about earlier where we've got our
values.yaml our base.yaml and our
overlay.yaml that are coming in to our
Helm command to modify that for a
particular environment
and then a little bit hard to see down
here but at the bottom this clusters
folder at the top level and in each in
that clusters folder I'll have a folder
for each environment each cluster that
we operate right and then typically I
map each cluster to an environment so
Dev stage prod again you'll see it's got
a flux system folder in there that has a
whole bunch of stuff in it that is
generated by flux when you bootstrap
your cluster the flux CLI will generate
all these for you put it right in your
repository maintain it there but then I
have two interesting files here
apps.yaml and clusterservices.yaml
and as you might be able to see down
here at the bottom those are called out
as flux customizations all right so my
apps.yaml file is typically one file
with several flux customizations in it
that point to each one of the
applications up in the apps folder
all right so in the dev cluster I'm
saying I want you to deploy these apps
and in the dev cluster my apps.yaml will
point to the dev overlay for pod info
in the clusterservices.yaml my flux
customization will point to the dev
overlay
for the nginx Ingress for the Ingress
engine X cluster service
this really once you kind of wrap your
head around this structure it really
kind of starts to make a whole lot of
sense and and makes it very easy to go
look and understand which things are
being deployed into a given cluster
what configurations do they have how do
those configurations vary between
environments right but you can also see
how it starts to give you a ton of
consistency across your environments
okay you now have
almost everything for each one of your
applications or cluster Services defined
in a common place with just a little bit
of variance per environment also defined
in a common place easy to reason about
easy to collaborate on easy to compare
the differences between environments on
now one more thing and we will actually
go ahead and
drop right out to
GitHub to look at this one because it's
a little bit easier but here I have a
sample repo that has
the very structure that we just looked
at and I'm going to show you what one of
my cluster Services looks like for
external DNS and how we apply some of
the features of both
customize
and Helm in order to change this
across deployments so in this helm.yaml
file I've defined a Helm repository at
the top I'm telling it I want to go look
I want to go look at the bitnami helm
chart repo to find Helm charts and then
down here I'm defining a Helm release
and this tells flux's Helm controller
that I want you to go do a release of
the external DNS Helm chart version
6.20.x like I said I let them do the
patch releases I trust them from the
bitnami helm Repository
it's got a little bit of configuration
that says okay
you know if something goes wrong I want
you to retry it at least three times on
an install or an upgrade the really
interesting part here is this values
from section okay there's values from
and I'm telling it hey I want you to go
get values from a config map called
external DNS values and I want you to
get values from a config mat called
external DNS overlay values and this one
I normally Mark the overlay values is
optional I don't in external DNS and
we'll see why in just a minute
but this says
this lets me take my values files that
I'm storing in my repository use
customize to generate a config map that
contains
that values file as a key
right and then
when I've defined this on here that
tells the helm controller go to this
config map pull that values file out and
use it when you construct your Helm
install command or Helm upgrade command
and I want and it's going to use these
in order so my base values is going to
be first and my overlay values is going
to be second that allows my overlay to
modify the base
now the other interesting thing remember
I told you that the uh
customize will append hashes onto our
config map
names you can also do this with Secrets
by the way important to note there so if
I go look in my base
we'll see that I have not defined the
file I want to show you
well let's look at a different one then
I think it's in here
I've got this customized config
because the config map generator and
secret generator they know all about the
built-in kubernetes objects so they know
where to find references to the config
maps to update them but they might not
know they absolutely do not know about
custom resource definitions right there
could be any number of custom resource
definitions in your cluster and they
can't keep up with that so here you're
writing a configuration that says Hey
whenever
the config map generator runs and
generates a new name also look for a
Helm release object and at this path in
the yaml update the name
same with the secrets right and you can
do this for any other types of custom
resource definitions by simply creating
a configuration file and referencing it
now let's look at
let's go back to our cluster services
and look at external DNS and see how it
is that we're modifying this in our base
we've got a values file that just sets
our log format to Json it's the only
common setting we really needed to set
for this nice and simple I want that in
all environments
but if I come to my overlays and look at
Dev
my values file here
I've got and this is just an example so
I don't actually have an Arn here but
I'm telling it some different things I'm
telling it oh hey this one
is in the EU North one region of AWS
with a public Zone it also needs these
other values to be changed it's a
service account that I want to apply an
annotation to obviously that annotation
is going to have to change between every
environment right so I need to define
those at the overlay level and that way
this can all get munched together and
create a way to deploy my external DNS
Helm chart into Dev or stage or prod
with the appropriate credentials
all right
so
question we answered on the crd yep that
one is
answered
um
can you point flux to Prometheus or
nginx Ingress Helm chart and at the same
time read the values file you have
created in your own repo yes
so I think
the the question there which I don't
know when that came in maybe that one
was already answered by what I just said
but that's that's what we were just
going over is that I'm storing these
value files in my repo and I'm saying go
fetch the helm chart from their Helm
repository But ultimately what flux is
doing is constructing a Helm upgrade
command really and it's going to say
Helm upgrade and it's going to take the
values that I pass it here and
and perform that install
all right is that repo public
uh yes this repo that I'm looking that
we're looking at here is in fact public
it is and I'll I will drop the link to
it in the room slack channel here but it
is at github.com moonswitch workshops
slash terraform eks flux I gave a
two-day workshop on managing clusters on
AWS with flux at the beginning of the
conference and this is sort of the
almost final state of that Workshop if
uh for those who went through it and you
can reference this and it's got most of
this information in it
all right how do you stage updates to
different environments to make sure
updates are first tested before being
released to production
uh that's a great question we use
overlays because we have an overlay for
each application and a separate
configuration for each cluster
we can say
you know if you're looking at like let's
talk about say the helm chart version
right if we had our Helm chart version
of 6.2.3 right or an image tag version
defined in our base right we want to
actually change that from the overlay
and we would do that in a number of ways
a really common one would be a patch
right so I can have a version defined in
my base and then my overlay I can write
a patch that says okay well only for Dev
do I want you to try changing the helm
chart version to 7.0 and let's see what
happens or only in Dev do I want you to
put this you know git commit Shaw based
Docker tag out there from our latest Dev
build and see how that works and if it
works then we will look at checking that
change in to the other environments in
order to actually propagate that change
all right
how do you do promotion of application
releases between environments in the
folder structure what I just talked
about right there you simply write your
uh your application write your patches
move them down through and ultimately
you know once you get to where that same
version is is in each environment then
you would just check in a change that
puts that version at the base and you're
ready to go through another cycle again
um
uh let's see here what is the main
difference between Argo CD and flux
um
well Argo CD has a GUI by default
um and they do a lot of the same things
but they take some slightly different
approaches into how they develop their
graph and which features they actually
support
um so I I and I found that I've had more
consistent reliability with flux than I
have with Argo CD I've used both I still
use both I actually have some there are
there are some features of Argo CD can
do some interesting things with reading
jsonnet files in order to generate
multiple configurations so I've used
that to set up Dynamic like QA Lane
deployments for some clients even though
I manage the rest of their cluster with
flux
uh is there any particular reason you
would choose flux over Argo or any other
same reasons I just said there I choose
flux first I only go with Argo if it can
do something that flux can't and those
things are very very limited
how does flux work when it comes to
client pull versus server Push by
default flux will pull the source
controller will pull all of the
repositories you define but you as I
said in the beginning you can define a
notification web hook receiver in order
to receive web hooks that will trigger
changes
and let's see here how would you deploy
your base cluster with flux if it
doesn't exist yet great question flux
has great documentation on their website
about their bootstrap command so you run
the bootstrap command with access to
your cluster it will
deploy everything into the cluster set
up your git repo and
we have lots and lots of questions all
right well I think we're going to have
to move off on after that so anyway
flexible bootstrap it you can also
there's a flux terraform provider that
can bootstrap your cluster for you
all right so oops
as I said uh I'm Jeff French from Moon
switch software
um
thank you for coming to learn about Helm
flux and get Ops I hope that you've
taken something away from this that you
can go and apply in your projects or at
your
company and start having nice consistent
easy deployments across all of your
environments I will stay back for to try
to answer some additional of these
questions for anybody who wants to come
talk but that is my time thank you
[Applause]