welcome all of you to this fairly warm
venue I'll try to deliver a bit of cool
content to balance it out for you
um if you read the description for my
talk I don't know if you're here to
learn how to hack your employer or if
you're here to try to make them yours
more secure but for both of you I think
you'll have some takeaways from this so
welcome
this is for the slides there will be an
additional one on the last slide if you
didn't catch this one yet
so devops I'm since you're here I guess
pretty much all of you are using Azure
devops or something similar
and so it's a no-brainer today that's
what pretty much all the successful
teams use and but if you look back at
how it all started at least it was this
way for me that this was something that
you found out about okay these other
cool guys are using this shouldn't we
try it out you do a proof of concept you
think oh this is awesome we don't have
to build we don't have to test manually
anymore and you start using it for
everything you start using it for your
production deploys and suddenly you went
very quickly from proof of concept up to
that level
did you really look at all of the
security settings inside your devops
environment when you did this step did
you evaluate the risks of all the
secrets that you're putting in there
I think in most cases this will be a
firm no so let's look through that
together
I'm Bjorn vestram I work at uppsala
Monitoring Center in Sweden as a lead
developer and I have an open source
project called Cyprus anti-lm auth
that's Cyprus is an end-to-end testing
tool so this plugin adds Windows
authentication support for it
and you can reach out to me afterwards
if you want to through these links
this is the layout we'll first talk a
bit about how devops is structured and
where the street Secrets may be stored
then we'll look at two different
techniques of how to you can expose them
I will also look at how to expose
content from repositories
when we learn all of that we might want
to learn something about how to keep
secrets away from Azure devops so there
it isn't vulnerable for these techniques
and I will close up with some hidden
gems features that I found really useful
in Azure DeVos that I think aren't
really well known
so if we look at the hierarchy you have
top level that's just the system and
then we have your organization and
underneath your organization you have
your project
underneath your project you have this
this is where you work you have your
boards you have the repositories and
everything but
are you using more than one project or
not
if you look at the recommendations in
the docs from Microsoft they say that
just use one and it can make sense
because it's less maintenance that way
and it's easier to move things around
within a project than Crossing between
project boundaries but if you have more
strict needs for separation between
different teams between between
different departments then probably you
will be setting up multiple projects
and inside your projects what are the
secrets yeah of course everyone knows
about the passwords
tweakit app certificates could be access
tokens or what have you
where are the secrets number one
this is where I used to find a lot of
them in the past
it could be hard coded into the source
code it could be in app settings files
web config file or other certificates
checked in with the code or in a Wiki
or even in pipeline definitions
I did a search on GitHub for the phrase
password inside app settings and you get
quite a lot of hits on that
yes all of them aren't real production
code most of this is just testing things
but either way even if you think this is
old news I will keep telling this story
all the time until the junior developers
that I encountered don't do this mistake
anymore because it's it's so important
you have excellent tooling inside Visual
Studio where you can use a feature
called user Secrets where you put your
secret so you don't even have to have
them for your local testing environment
and apart away from that when you go
into your CI CI environment parameterize
everything so you don't have to hard
code it
and if we don't store them there where
are they done
we have a concept called variable groups
inside azure devops
and secure files
they might also be in Azure key Vault if
you're using if you're more leaning on
to that
we have now two different types of
pipelines within Azure you have the yaml
pipelines and you have the classical
pipelines right and there could be
variables with Secrets within them
defined within classical Pipelines
or you could store it locally directly
on the server that's not touched by the
CI itself we will focus mostly on these
aspects in this talk
and if you haven't used a variable group
before this is where you find it so you
go to your pipelines you go to the
library
and there you go
let's look at what a variable groups are
for
it's just what it's called like you you
can set up a set of variables there and
they might be non-secrets or it might be
secrets and it's an excellent way to
share configuration between your build
or different environments like a
microservice URL you only have to update
it in One Singular place
and it allows you to structure your
variables for each environment at least
that's what I like to do so I have a
group for system tests I have one for
staging and I have another one for
production
and if you open a variable group it will
look something like this
uh when you create a new variable it
will by default be a non-secret but you
can make it into a secret by clicking on
this padlock and when you do that the
value becomes masked
you cannot copy it anymore you cannot
clone the variable group and try to
extract it that way and if we try to
remove the lock again even with the
alohomora spell it still won't show it
will be cleared when we do that
you can even try to use the rest API to
catch it but it doesn't work either so
from a US perspective it's pretty solid
but let's try to extract it
so in all of these scenarios we are
assuming that we have a user who has
access to your devops environment so
this either someone who's already
working at the company or someone who
has acquired them in some other way
in this case
the user knows about the variable group
variable group that has a production
secret that he's curious about
so what can you do to find it well
you create a new pipeline you include
the target variable group that we're
interested in and you try to expose the
secret from there
now maybe you've already tried this and
know what will happen so here is our
simple pipeline we connect it to this
specific group and we add a script line
where we try to Echo the content of this
connection string secret that we saw
before
so what will happen here well nothing
fancy because
as the devil's will to catch this and
mask it also in the logs so
I think they did at least a decent
effort here
but if we take now one step back
and think about how did they actually go
about to implement this masking
technology
what would you have done in this case
let's look at the pipeline again
this is something rather straightforward
here we just printing it directly in the
script I might have done something much
more elaborate storing it in a variable
and passing it around between functions
or wherever before I try to print it
do we really think that Microsoft went
the whole length and tried to parse
everything I'm doing here to know
exactly where my secret is when I try to
print it
I don't think so and I don't think any
of the other CI environments do that
either because it's just too complicated
so what would they do instead likely
hmm
I think they do
a search and replace
if that's the case how how could we get
around that
well one rather cumbersome way would be
try to logging a dictionary with like
password phrases and find which one is
masked
maybe a bit more convenient would be to
encode the secret as hex before we log
it
change the casing or even simpler
substrings
so let's try substrings
same pipeline
Small Change
I assign the secret to a local variable
and this is shorthand for extracting the
first character of that variable at a
space and then the rest of it
what do you think will happen
great success
so
I think this is a really good example
because it shows just how easy it is I
mean a lot of hacks that you come across
are fairly convoluted this is really
easy to do and I think
you probably could manage to sneak in an
additional line in a script somewhere in
your pipelines even if it's reviewed
to accomplish this right
but let's say we don't want to go that
way we don't want to Echo it out into
the log someone might be get suspicious
and see it sometime let's do it in
another way
same pipeline but here instead of
putting this in a variable I append it
to a file called Secrets text and then I
add another step to publish it and I
publish it in an artifact that no one
will probably open ever
and let's look at that
we got our artifact when we open it we
have our secrets file and there we have
it again in plain text
now maybe this is at first astonishing
but
if you think back of course the pipeline
has to have raw access to these things
otherwise it wouldn't be able to create
our app settings file or what have you
that it needs to do it wouldn't be able
to communicate with your database or
your servers so of course it has to have
this raw access but I think still maybe
we aren't aware of how easily accessible
it is even though we have a masking
feature so what can we do to kind of
prevent this well we can look at how
these variable groups are accessed we
can limit the resource access
because there are settings for which
variable groups may be accessed by a
pipeline there's also settings for which
users May authorize such access so let's
look at those
these are the different access levels
that exists for variable groups and
secure files
the read level space level where you can
just list which variables exist and
apply them in a pipeline if they're
already approved the user level means
that we're allowed to authorize the use
of the variable group or secure file in
a pipe
that's the important one then comes when
you have ACL access so you can change
who has these roles but you can note
that none of the access levels not even
the administrator has read level access
for secrets
thank you
recess authorization is a process that a
pipeline goes through when you run it
the first time or when you change which
resources it accesses
so any user that has a user plus access
level on that resource will be able to
authorize it
it looks kind of like this so I've
created here a new pipeline
it will tell you that this pipeline
needs permissions to access resources
before the Run can continue
click to click the view button we will
see which resources it is accessed it is
requiring access for and now our user
doesn't have the proper permission level
to authorize it so there's no button to
approve it they have to go to another
developer or someone in the devops team
who has that authority to approve it
okay
sounds like this might solve the problem
right
but there's another step
and this is in the variable group
settings themselves you have something
called pipeline permissions
so
you can pre-define that a variable group
should be accessible to a set of
Pipelines
and
the the sad part of the story is that
the default setting for this when devops
was young and new was that it was
unrestricted
so all very all pipelines May access the
variable group
then
sometime in May 2020 Microsoft changed
this so when you create a new variable
Group after that date then it will be
restricted by default
but all your old variable groups that
you had defined in the past will still
be unrestricted so here it doesn't
matter with this authorization they will
be automatically authorized since any
pipeline can access them
so if we click restrict permissions we
will get to this page which shows which
pipelines are allowed to access that
particular variable group and you can
add them here up front so you don't have
to go through the nuisance of approving
all of them when you run them the next
time but really recommendation go
through your variable groups and enable
this unless your variable group doesn't
contain any secrets you might have one
that's just which node version should we
use which.net version should we use just
plain text values then you're fine don't
worry
last trick with this the classical
pipelines can still use the resource
doesn't matter that it's restricted
doesn't matter with the authorization
flow all the classical pipelines will
still have full access to whichever
variable group they want
so
disable them and migrate away from them
if if you want to be a bit more safe
secure files maybe you aren't using them
but the same settings applies just as we
showed here it's just that in that case
I can't really see a use case where you
want to have them unrestricted because
the secure files are
by name something that is a secret right
so they should always be restricted
now we learned how to try to stop this
let's see how efficient that is
in this case
our user does not have the access to
approve the use of the variable group
that they're interested in okay so we
just took care of that
how would you approach it this time
well instead of creating a new pipeline
or adding the variable group to a
pipeline
the user tries to find an existing
pipeline that already has the target
variable group because then it's already
authorized right
and then we can make a branch and modify
that pipeline
we push it up and expose the variable
this is our starting point
this is a deployment pipeline that has
access to this group read access with
just a single script statement
we Branch it and modify it and add our
substring trick here at the bottom
and then we push this and we should be
able to see the secret
ah not so fast
maybe just maybe
there were some policies in place for
this branch
meaning that this won't Auto build
because the master branch is protected I
can't just push or pushing expose won't
build it and getting in it into the
master would require a reviewed PR so I
would have some additional steps and
risks if I really want to go through
with this
you have any idea what we would do would
you go to a developer and try it try to
ask them to approve this or
you can change the trigger
we have control of the pipeline we can
change it doesn't have to be Master
anymore we just change it to our own
branch
and now when we push this it will build
directly and expose the secret
all right so
if we're going to be thorough here we
have to find some way then to also
prevent pipeline modification as we just
saw
there are security settings in the
pipelines which we can use but
when I try that out it appears that they
only affect when you're trying to do
things to the pipelines from the UI
so everything that you're doing in the
yaml file directly in your local repo
won't matter whatever security settings
you set up or which uses are there so
that one's out
you could limit which users have access
to which repositors
I don't really like this approach that
much I believe in transparency that we
should be able to learn as easily as
possible from each other between teams
uh but it's a possibility
we could use the classic pipelines but
we didn't really want to use the classic
pipeline so let's just skip that and
this is getting a bit more interesting
it's possible to add checks and
constraints on a variable group meaning
that it will only be accessible on the
certain conditions it could be a manual
authorization step whenever you use a
production variable group for instance
or we could limit it to a specific
Branch like the master branch in that
case
the last scenario would not have been
possible
so that might be doable but if we're
going to limit it to a specific branch
I would say this might work for
production
but our test environments I like to be
able to release a lot of different
branches feature branches what have you
and try them out
and therefore it's much more difficult
to limit what names of branches are
allowed to push to that or to use those
those variable groups maybe you don't
think it's so critical if we expose a
few test environment Secrets but try
harder it's it's really good if you can
secure those up too I don't want to
mention how many companies I've worked
with where those are the same in those
two environments so
um and then you can use Azure key Vault
but it's pretty much the same thing
because the secrets will still pass
through our pipeline they will be linked
to a variable group and they will pass
through the pipeline so you might set
checks and constraints on your service
connection to ask your key Vault to try
to restrict that at only specific
branches is allowed
all right so there's not really a silver
bullet in there it's difficult to
prevent pipeline modification completely
so yeah what can I tell you
we will have some other aspects of this
where we try to remove secrets from
Azure devops instead later on
but let's first look at the Repository
we've only looked at the variable group
so far
in this case our user doesn't have
access to the target's Repository
but they're very curious about what's
inside okay
what can they do they create a new
pipeline
they clone the target repository
manually they don't link the pipeline to
the repository but they do a git clone
command inside the pipeline
then they can list the files and they
can expose content of a file
this is how this would look
we clone the repository Conan which we
know exists and are interested in but we
don't have access to it
and we enter the directory of Conan we
can list the contents we run it we see
which what contents exists and then on
the second path we can add statements to
Output all the contents of a file which
we are in particularly interested in
we might just as well just packed this
up into an artifact and downloaded the
whole repository if we if that's what we
wanted so I mean everything from the
triple story is available to us despite
the fact that we don't have access
why
ah okay
short demonstration of this technique
but why was this possible
because
the pipeline is not run by you
it's run by the build agent
and the build agent user is the
mastering with the default setting to
build agent users access to all the
projects in the whole organization
everything
okay I'll say that again all the
projects all the repositories in the
whole organization
we're just lucky that it doesn't include
contributor access because then we could
have done all the other tricks with the
variable groups by changing pipeline
files in whichever repository we wanted
so the bill leading agent can't by
default modify that but I've seen cases
where we've added this specific access
to specific repositories it might be
okay if you have like a release notes
repository or something like that where
you're only pushing those kind of data
into it and doesn't contain anything
Secret at all but otherwise be really
restrictive with adding this access
level to the build agent user
but there must be something we can do
about this yes there is on project and
also on organizational level there are a
few settings that we should enable
called job authorization scope settings
so the first two
are related to inter-project access
and this non-release Pipeline and
release pipeline stuff you can if you
aren't using the classical ones you can
ignore it everything is actually in this
category but doesn't matter just enable
both of them
when you do that
the build agent won't be able to access
across projects at least
it can still access any repository
within your project
okay if we enable the third one
then Azure devops will limit the access
level of your build agent to only their
repositories that are used by your
pipeline
so this means that if you have to access
multiple repositories from your build
then you have to list them up as
resources at the very beginning of your
pipeline so Azure devops is aware that
you're going to use them it can allow
you to authorize that access and then
it's then it will work
now for some reason these were all
disabled by default
um I strongly recommend you to enable it
Microsoft made a change here again May
2020 where they said made this default
enabled but only for new clients
so
yeah look through your settings and I I
recommend putting it at the
organizational level as I said
because that will override any project
setting changes
okay
now we're all scared of how easy it is
to extract stuff from anywhere but is
this something that we could do like
more long term to solve this
let's see what we can do in keeping the
secrets away
managed identity is something that I
really like
only main drawback is that you have to
be completely an Azure you cannot do it
for on-premise resources has to be
hosted and authenticated through an
Azure ID but if you have that in place
this eliminates the need completely for
us developers to handle authentication
information everything is resolved
internally in Azure we don't need to
extract it for any any means it's all
there locally and it rotates it for us
to keep it secure so there's no secrets
left to expose
if we're not that lucky
maybe we could use something like Azure
cable I should keyboard is just one
example of a Secret store which is
independent of devops
out I mean the path that we discussed
earlier was linking this to Azure devops
and then this doesn't help at all it
helps a little bit because it has
Version Control which we don't have in
the variable group so I think it's it's
still a good product but you can design
your services to integrate directly with
key Vault and fetch your secrets
this is what we do in our organization
so whenever you have a free cycle
in the startup phase it will connect to
key Vault and it will fetch the secrets
it needs and then run so there's nothing
in the apps settings for it it's
automatically populated
so I really like that feature there's
a couple of limitations to it
one is you have to have network access
to Azure keyboard from your servers
if they're tightly firewalled somewhere
or air gapped I don't know then this is
not a possibility but as long as you can
allow access at least up to the Azure
environment to access to access your key
Vault you can use it
other potential drawback that I've seen
is that
our new.net apis they have a tendency of
getting stuck in a limbo State when they
encounter an exception during the
startup
and one such exception which we've seen
now and then is that it cannot connect
to Azure keyboard
because of some temporary Network
disturbance maybe that link is down for
a minute or two but if we're unlucky
that our application pool is recycling
at exactly that time
it tries to start it tries to connect to
Azure key Vault it throws an exception
and gives up but it doesn't shut down
so it keeps running and serves this
503.20 or whatever to everyone who's
interested until it recycles again
so something to be aware of
okay
let's go another way let's talk about
some hidden gems which I think deserve a
bit more attention that are
really good to have in your toolbox for
your Pipelines
now how many times
have you dreamed that you could just
repeat a task in your build or your
deploy pipeline that single task that
just fails intermittently and it's kind
of out of your hands
this happened to me many times that I
was I was looking for this kind of
feature but it just wasn't there
until I found it by chance
just a few weeks ago it's actually been
there for about a year but I don't think
that they're I mean there's so many
changes in Azure devil so you can't
really push news about everything to
anyone I understand that but this was a
really really great addition so
you can add an additional argument retry
count on task failure maybe not the best
name but it tells you what it does
and then you set a number
so if this task fails it will retry it
up to this number of times before it
fails the complete job
you will get a warning if it failed one
or more times but that's just good I
like to see what happens but it's so
much more convenient the alternative
that we had considered was working out
some kind of Powershell script where
you're trying to do a repeat cycle
yourself if you're doing it that way you
lose all the convenience of all the
built-in tasks that are already in
available for you in Azure levels
because you can't use them that way
right so I really really like this
a couple of tasks which we have
particularly many issues with were
nougatask
that tend to fail our builds now and
then and also IIs management task which
tend to fail our deploys now and then
get shallow
repositories can get pretty large over
time
and when we do a git clone which is what
regularly happens when it's when we run
our build pipeline it will fetch the
full repository and all the history by
default right
we might have in the past committed some
large files by mistake some testing data
wherever we removed it and we're good
but it's still there it's in the history
so in most scenarios your build pipeline
don't really care about that history it
only cares about the most recent version
of your code
and there's an additional parameter that
you can add to your git clone command
dash dash depth equals one and that
limits your fetch to only the most
recent version
when we apply this in our project we cut
30 seconds on our build pipeline runtime
so I think it's significant
and but it's really well hidden away
you have to edit your pipeline you have
to click the three dots more actions go
to triggers then select yaml then get
sources so this is when you you're in
edit pipeline
you selected more actions you selected
triggers that would land you on that tab
you change to the yaml tab you select
get sources and down here
that's what we're looking for
so we enable shallow Fetch and we can
set an a custom depth there I mean there
might be cases where you want to do some
tricks tricks in your deploy where you
need additional depths but usually one
should be enough
this is also something that Microsoft
yes has enabled by default sometime in
the fall last year but
not for all your old pipelines so you
can go and enable it on all your old
pipelines and make them faster
environment resources last tip
it's a way to simplify on-prem
deployment is any of you used
environment resources
now
if you used environments
yes okay
it was the same for me so environment
that was a way for us to like add a
mutex to our all our deployments we
defined an environment called System
test and we required it to be allocated
for our deploy and this way no other
deploys could run against the same
environment at the same time I think
that's usually what you use it for
but slightly hidden inside the
environment is another tab called
resources
where you can add resources and I
started playing around with that and
found that this this was really useful
because what you do there is that you
register all your servers that you use
in your environment or at least the
service that you deploy to
and you do this by installing in the
deployment agent on each server
so when you do that they self-register
with azure
and
this way Azure devops is aware of that
those servers exist and it can deploy
directly to them
so directly from Azure devops into the
servers
this provided a lot of nice benefits we
got native Powershell we don't need to
do remote sessions anymore you just run
it as it is
this gave much us much better logging so
we see what's going on and it reduced
the runtime by 30 to 80 percent because
these remote sessions are terribly slow
and we didn't need an on-prem deployment
server anymore because everything could
go directly to the Target service
and when we don't need on-prem
deployment server we don't need a super
user account this is another way of
cutting down on your secrets
because
this
deployment server would have to have
access to specific accounts which were
authorized to do a lot of things on all
our production servers right
but this way everything goes in the
other direction it's the servers
themselves the talk to devops and ask
okay do you have any tasks for me all
right let's do that so they don't need
any specific user accounts to do that
and all our servers suddenly became
reachable we had issues with deploying
to our servers which were located in the
DM set
but now they communicate by themselves
with Ashley DeVos
you structure your deployment jobs by
using tax so when you create or you set
up a server this way you also specify
which tags it has so if it's a web
server database server or whatever kind
of distinctions you need
and then in your pipeline you reference
those tags so I say run a deployment job
on this environment with these tags and
then it will run it on all the servers
that has those tags
if you skip the tags it will run it on
all the servers
okay the only thing that I was really
hesitant about at the start was
do we really want to have a deployment
agent running on our production server
because that was like the only bigger
change from the servers perspective
it has a very reasonable footprint
it has about it uses about 50 megabytes
of ram when it's running and it's idling
when it's deploying it doesn't really
matter because it's shutting down the
services anyway but I mean it's it's not
really competing a lot with the rest of
the resources on the on the server so I
don't really see that as an issue
uh-huh
yeah okay
so we talked a lot about security
settings and that makes you kind of
sleepy but it's
the it's a classical trap that you set
up a lot of bureaucracy and
authorization flows and you make it
really hard for your developers to
actually do their job so I think it's
still really important that we trust our
developers
and let them work in an efficient manner
otherwise we just risk that they go to
talks like mine and try to get the
secrets themselves so they can work
efficiently right so
trust your developers and
minimize the need for handling the
secrets
so push as many of them away from
from both Azure devops and from your
local environment so you don't need to
handle them manually
some takeaways
in the short term we talked about the
variable groups and the secure files
restrict the pipeline access that's not
enabled by default in your old
installations and restrictive use of
user access level or above so you have
some control over who can authorize it
for new Pipelines
and disable classic pipelines when you
disable classic pipelines it doesn't
mean that they all disappear just means
that you can't create any new ones so
you still have time to migrate away from
the ones that you need to use
for the repositories
manage user access and limit the job
authorization scope so you cannot
cross-reference stuff at your at your
ease
and in the long term
get the secrets out of there put them in
Azure key Vault and use the server fetch
technique use the environmental
resources to get rid of the super user
accounts in your deploys go to manage
identity when you're finally in Azure
completely
thanks so much for your intention
it's been really nice to do this talk
live because I've done quite a few talks
during the pandemic so it's great to
have an audience again
you can find a slides there there's a
the actual Link at the Top If you cannot
manage to scan it
there were were actually some other
sessions if you're really interested in
this topic I would recommend to take a
look at Michael Kaufman's session on
GitHub Advanced security which was held
on the first day and also Kyle kotovich
session about no password both of those
are really connected I would say to this
and I will also say as first time
speaker at NDC London I thank the
organizers for letting me speak here
it's been an honor
thank you
yeah I think we have plenty of time for
questions if if you have any