welcome everyone to this talk uh I know
that lunch is coming up so I'll try not
to stay up here longer than
necessary um my name is Mark lefter I'm
the CTO for a Swedish consultancy firm
called
ement where we focus on cloud native app
development with an emphasis on message
driven
systems and in this talk I want to
explore Access Control in message driven
systems perhaps a somewhat esoteric
topic to start up with um how many of
you are message driven today say using
Pub Sub in any
form a lot of
you how many of you do any type of
Access Control permission checks when
you consume a
message fewer def
definitely okay so this is what my talk
will be about
the um action of or actions you take
when you want to
authorize whatever is going to happen
when you consume and process some type
of
message so we're going to focus on a
specific context which is zero
trust so we're going to look at what a
zero trust enironment means and why it's
so important to uh establish a
functional Zer U Access Control
mechanism in this
regard for those of you who are not
familiar with messaging we're going to
explore that a little bit see what types
of messages we are dealing
with and we're going to see how open
policy agent helps us to actually
enforce policies where we need them to
to
happen and we're going to finally look
at how do we actually
Implement zero trust for asynchronous
communication that is messaging and
hopefully we have some time left for Q&A
at the
end so I want to start
by setting up a frame of reference for
this talk what are we going
to um use as a foundation for this uh
exploration first of all access control
that is authorization so we're going to
assume that we have some sort of
verified
identity to work with meaning a user or
some system has provided
credentials we have authenticated them
and we' have received a token or rather
the user the system has received a token
which then will provide us for Access
Control so in a general sense we're
going to use this verified identity in
the form of of a token most of you are
probably familiar with the notion of
Json web tokens I'm going to call them
tokens for
short and they will help us to enforce
that this particular verified identity
can perform some type of action on some
type of
resource and we're going to use a model
called attribute based access control
because it gives us the most flexibility
in coming up with a decision can we
perform this operation or action on a
resource or not tokens usually contain
claims attributes that we can use
resources could do as well and we can
also use perhaps some environmental
attributes all of it factors into the
decision do we permit or deny this
particular
action furthermore the type of system
we'll be studying is a distributed one
of services or microservices if you will
and our aim is to take this very
verified identity and all of the claims
therein and propagated along the
synchronous flow that will set up to
perform the type of business logic that
we um desire so this identity context
needs to be available at every touch
point along this
flow so I mentioned a synchronous
communication that is the uh Cornerstone
in this talk as well so it means that
services do not interact directly with
each other rather we're decoupling them
and we're using some type of
intermediary here called a message
broker to facilitate the interaction
between
services so with this as a
baseline let's look at what zero trust
entails you might be aware of the old
adage trust but
verify this is sort of the stricter
version of that where we go with Never
Trust basically anything always
verify sounds great in theory can be
hard to implement in practice but it all
means
that we are no longer content with only
enforcing Access Control at a network
boundary we're assuming that anything
both outside the boundary and inside the
boundary is by default to be untrusted
like a real life example of this is your
Wi-Fi network at home or even in your
office a friend or colleague might stop
by and ask you for the Wi-Fi
password you of course are friendly back
so you provide them with this password
and they enter your
network in a lot of cases we're fine
with that we're trusting this particular
person to operate with our network but
do we have any idea what the application
on the particular person's machine do
sell
them so for modern use cases this is not
enough we have this divide between
application security and network
security AS application developers we
have one notion of what it means to
secure our applications and
interactions but the IT department that
de deal with network security at a lower
level they have a different notion than
we do so this gap between different
departments and different philosophies
and
policies create trouble sometimes for
example you might have a kuber cluster
administrator assuring me that yes
access to this cluster is fully fully
secured I've done my
job but turns out that for some
reason the services that are operating
within the cluster are exposed to the
outside world yes this can happen and
has happened
so yes your cluster might be secure but
it turns out you missed a couple of
things because the application developer
when writing the kubernetes manifests
didn't really agree with what the
network administrator or the cluster
administrator had in
mind and in this Modern Age we're
dealing with a number of threats inside
of our Network
boundary its breaches are not in common
we're dealing with malicious
applications both externally and
internally and it's always uh our fault
so to speak we might be using
third-party open source packages and how
many times have you seen a particular
mpm package for
example uh highlighting certain um
vulnerabilities supply chain attacks and
so forth we're not always aware of
everything that might be malicious that
we Embrace in our
applications so we desire some type of
zero trust philosophy and the
implementation of it in our applications
and our networks now this is hard to
implement sometimes and we must make
sure that implementing zero trust the
cost of it doesn't exceed the value we
derive from it in Practical terms it
means that sooner or later you will have
to work with your network security
people but we as application should make
sure that we have done our part to
ensure that along every step in our
synchronous
communication we are as sure that we can
be that we're not trusting or assuming
anything but rather making sure that our
Access Control follows the principles of
zero
trust so this talk is about messaging
that is asynchronous communication but
it can be helpful to look at what
synchronous communication entails and
how zero trust might Shape Up in in this
context so in this slide we have a API
Gateway we're assuming here that some
client has been authenticated provided
with a token which is then passed to us
uh in the Gateway now tokens they have
this nice U property of being stateless
or they can be used for stateless
authorization we don't maintain any
sessions when we're using tokens which
is nice when we try to scale out our
distributed systems so this token is
provided to the AP gate API
Gateway we validate the token and now
we're inside the network
boundary again we're trying to follow
some zero trust principles we start by
making a token exchange that is always a
good practice so we take the client
token and we exchange it for an internal
token with the client information
from our authorization that we defin
authorization server that we definitely
trust that we have to have some trust in
some things that we deploy the
authorization server is one such
entity so this token can then be trusted
by all services to operate inside the
network
boundary so this exchanged token in pink
is then passed to a first service a
backend for front end if you
will and that service in turn calls
another service using that
token so here we have two layers of
security we have the network
part which is illustrated by the blue
error labeled mtls have you heard of
mutual TLS some of you probably have it
means that each service the BFF and the
service uh that it contacts can be
mutually authenticated the client will
trust the the caller will trust the
Calli and vice versa this is often what
we deploy um or enable in a service mesh
inside of a cators cluster that's on the
network
layer but we on the application layer
also provide a the token that is to be
used inside the network
boundary so it's always good to have as
many layers of Defense as possible and
we we making sure that we have that at
two different levels in this scenario
now synchronous communication implies
shortlived interaction these tokens
shouldn't live for very long because if
this call and this call chain fails then
we're done or if succeeds we're done as
well so it's a pretty simple interaction
style so what happens if we make a
slight change if we need to have a syn
inous Communication in this
scenario for example let's say that our
back end for front
end now instead of calling a service
directly calls it indirectly through a
message broker imagine that API that we
support externally a is asynchronous as
well you can post something to the API
you get an immediate response and then
you can pull the API for a result for
example in either way we in the back for
front end we post a message or publish a
meage message onto a message
broker and sometime later our service
will consume it process
it now this is different because this
interaction is typically not short-lived
there is different definition of what
shortlived
means but imagine that this interaction
is long lived maybe the service doesn't
consume the message immediately it might
take some time or even requires manual
intervention approval from me before the
asynchronous flow can be
completed this is tricky and can cause
some problems mostly because the
identity context the JWT or token in
pink there is no intrinsic mechanism for
just including it in the message is
typically lost that's one problem
another one is that when using
messages we're implicitly embracing a
property called location transparency
when in the synchronous case when one
service calls another we know who we
call we know the name of the service
we're calling what the API is ETC here
I'm creating a message I'm putting onto
a broker I don't really know where it
will end up who will handle it and how
many consumers there will be in the case
of events for
example so
we're being more transparent about where
things will be processed and how we can
still use mtls and ACLS for example when
publishing to a Kafka broker and this
ACL can control what topics I'm allowed
to publish to again we're on the network
layer here but on the application layer
the token is typically lost or there's
no um easy way or obvious way to include
it in the communication
so this is where our Focus will be uh
for the rest of this
talk so we're talking about messaging
there is not one message there's roughly
three different kinds if you're building
a message driven system first you have
commands you can think of them as normal
requests I've term them sometimes
synchronous request
reply you send a
command to some command Handler you
don't know exactly or typically what
command Handler that will be but you can
be assured that some Handler will
process your command and return an
acknowledgement yes I successfully
process your command typically one
command Handler will handle commands for
a range of entities or
resources um but commands are pretty
simple you send them out they will be
directed towards a single command
Handler processed and you'll hopefully
will get an acknowledgement back on the
right hand side we have queries mostly
the same semantics though we have
different types of them you can send a
query to particular destination that you
know of or you can scatter the query and
gather results sometimes when you don't
know exactly how many query handlers are
out there that might have interesting
responses for you and then you have
subscription queries where you initiate
a query but it can be long
lived so you'll get a result first and
then over time you get additional
results as they
occur and then we have events which will
be the main focus here as well because
they follow the semantics of publish
subscribe you have any number of
consumers out there either present or
being added or removed in the
future and they typically don't provide
any results for you there is no
acknowledgement more than the broker
itself telling you yes I successfully
got your message and deliver it to to a
consumer so since commands and queres
are roughly the same well I'll try to
combine the semantics for them in in one
discussion and then we'll look
specifically at events and what makes
them so
particular so let's look at a simple
messaging flow where
we have one service trying to trigger
something else in a different service so
think of this as an operational workload
or flow one service calling another
service and trigging some business logic
or activity in the other service we'll
start with
command I'm using here some kind of
abstraction you can think of the command
uh broker or command bus the naming is
not really important but this command is
routed through some abstraction call a
bus
to a command Handler this will be our
first enforcement
point so imagine we have a token as
we'll shortly see in the command this is
where we'll first have to check
for uh proper
access in response to processing a
command successfully you'll often see an
event being emitted as a consequence of
processing this
command and this command again is routed
to one or more event handlers through an
abstraction we can call an event
bus and then we have the second
enforcement point the event handler once
we process this message we typically
want to enforce some policy to make sure
that some action that is to be taken on
behalf of the
user that is sort of tacked on in this
flow is permitted or
denied we're still in one service
but now we want
to propagate the identity context to
another
service so we'll emit an event this is a
subscript with an i so you can think of
as an integration event I typically
don't like that name so think of this
event as a an external event external to
the service whereas the first event
without a subscript is an internal event
and this event for example can then be
translated into another command to do
something in um in the second service
and then you have your additional
enforcement
points as well in the second service so
two Services interacting through some
kind of
messaging um system using these notion
of buses for commands and events and the
different types of enforcement points
that we uh can
encounter again this is for an
operational workload I'll be talking
about Analytical
workloads in a
bit now to provide a little bit of
abstraction here each of these Services
you can think of them as if you borrow
some DDD terminology as bounded
contexts logical groupings
of um Services one or more
services where the commands and queries
are the apis to this particular bounded
context again it can be one service or
many depending on your for example
scaling needs a command typically means
that you're updating some type of State
in this context this bounded context you
have internal events that you can use to
for
example uh update a view of some uh of
some sort that you can query for
information are you familiar with cqrs
perhaps you probably command query
responsibility segregation then this
will probably familiar to you and then
we have a bounded context emitting an
external event to another bounded
context just as we saw in the previous
slide these bounded contexts you can
make an analogy to vpcs there are
logical
groupings but the thing is inside of
this context the infrastructure we use
for example for routing commands and
internal events can be completely
different from the infrastructure we use
between context that is often the
case so here we come to the merging of
our responsibilities application
developers and network security
administrators if we can sort of confine
ourselves to bound context and reassure
ourselves that yes we're very very uh
determined to make sure that everything
is safe inside of V contexts we can sort
of relax our strict always verify never
trust um principle inside of the bound
the contexts it depends depends on how
much it will cost you to implement a
fully zero trust architecture inside of
a context but typically if we work
together application developers network
security
Personnel we'll find a good compromise
however between bounding contexts will
be less shed so that's where zero trust
principles must typically follow to a l
to a larger
extent now we've seen that there are
different number of enforcement points
where we need to check for Access for a
particular identity that is propagated
along this flow how do we actually do
this is there something we can
utilize for this
purpose some of you might be familiar
with open policy agents it's
a I would say recent standard or de
facto standard but it's an open- Source
initiative which defines certain
elements in a architecture for us to
work with Access Control policies in a
straightforward manner so imagine we
have some type of request coming into a
service or a context we'll have one
enforcement point a policy enforcement
Point how do we get hold of that actual
policy well it's not up to us in the in
that particular part we will delegate
that to a policy decision Point that's
where the actual enforcement will occur
that's where the policy resides that's
where extra data is available to make
come to a decision so we from our policy
enforcement Point provide or invoke the
PDP to make a decision for us the PDP in
turns relies on a policy Administration
point to provide it with a policy bundle
basically give it give me what I need as
a PDP to make a decision on behalf of a
p and then if we need additional data we
might might be calling out to some
external source for additional
information we can also let the policy
decision Point call out to a policy
information point a lot of lingo but
this is the basic setup again the
enforcement point is what we are in
charge of in our code flow but we
delegate the responsibility for a
decision to a PDP the decision point if
everything works out we'll be be granted
a green arrow to access to Resource do
some action on that
resource so how does this all translate
into for example handling a
command well we have a command we have a
token that is valid only inside of our
Network boundary inside of our
context and this command is being
processed by us in our code in our
enforcement point we delegate the
decision to the PDP it
takes action or comes to decision and
lets us process the command in our
Command Handler if everything turns out
okay now you see this there's a box
around the p and PDP they are typically
collocated why because we have to have
as a low latency as possible between
those two um elements so in a kubernetes
environment you'll typically have them
in the same pod two containers as a side
car so your your application container
is the pp the PDP will be the open
policy agent server so they'll be
collocated and this will be present in
every process so if your service is
actually multiple Services you'll have
this set up for every service
there okay so we've seen what
asynchronous communication means what
message messaging entails different
message types and how Opa can help us to
achieve this on a structural
level how do we actually go about
implementing this if we have all these
elements
available so first off a token typically
as in the synchronous case that we saw
earlier is very shortlived it's
expiration time is well short
now in our case that won't work why
because again our operations can be long
running so we need to extend the
expiration time for our
tokens moreover we can just do one token
exchange right at the API Gateway that I
showed you earlier we need to uh create
or generate tokens for each message that
we publish or send
out again to make sure that no one
tampers with our message we'll have to
Hash whatever data whatever
payload will'll include in our message
and we'll have that as a claim that will
be an attribute just like any other
claim for example who I am my email
address and typically we also have a
message ID that we generate so at the
top you'll have a stock change event
say we have our data product ID current
quantity and a message ID that we
generate it will help us later as I'll
show you that's our event that a Raw
event that's a data that we want to
transmit and maybe this event is a
consequence of handling another message
before a command that changed the stock
say the token that will generate
will look like this in a a simpler
format we'll have the subject that will
be me the user context or the identity
context it will include my in this case
identity my email
address will include the Mig and most
importantly the message data hash so
we've hashed the data in the top code
box that you see there and we have
included it in our token which has then
been
signed so the resulting event with the
token will be pretty much like we had
before but we've added the token as well
now this token and this message this
event in this case are
bound if for some reason this token were
to be um
maliciously retrieved to be used in
other messages that won't work because
we're bounded to this particular event
and this payload so it's useless in any
other
circumstance so if we send us this stock
change event with the
token how do we consume this token so
let's say that we have published this on
a message bus or broker and then we have
one or more consumers consuming it and
processing it well first of all we need
to verify the token just like we always
do we get the claims the attributes that
we'll need use for Access Control we'll
first see have we processed this
message
before so for example let's say that
this event was somehow
intercepted and some attacker is
replaying sent us the same event again
we can use the message ID as sort of a
check if we have seen this message
before it's no different really from how
we approach at least once delivery so
you can think of it that way
so we can reject that event if we've
seen it before it's m maliciously
replayed then a super important thing
we'll see if the hash of the data in
this
event has been tampered
with if this check it doesn't turn out
okay we must reject the event we can't
consume it or process it because it
might have been changed by some
attacker and finally the whole reason
why we're doing this is to make sure
that we can then contact our policy
decision Point our open policy agent
server with the resource that we want to
perform some action on and the claims
that we have retrieved from the
token so again the token which we
generate for every message that we want
to publish command event or query will
be bound to that message so it doesn't
matter if this token would be hijacked
somehow because it would be useless in
any other
case so this is basically the mechanism
that we need for zero trust we're
enforcing zero trust everywhere we have
enforcement points possible enforcement
points in our async communication flow
and this works again for commands events
and
queries
so there are some things to consider if
you actually want to implement this and
employ it on a broad basis in your
distributed
system for
one we're exchanging tokens or
generating tokens for every type of
message that we publish now for low to
mid
volume uh um messaging that could be
fine but when you start to reach a huge
volume of messages that you need
to publish that can be tricky because
you need to contact this authorization
server to make sure that your new tokens
are exchanged and generated
properly so there are some Alternatives
you can use what's called self-sign
tokens for this um this does require
some certificate schemes that can
compliment complicate your
implementation but it's an option or
this authorization server this one
instance that I've shown earlier that
everybody talks to could also be placed
as a side car as a side container in
your pod say so you'll both have an open
policy agent server which enforces
policy and access control and this
authorization server to
quickly uh create new tokens for this
new event or command that you want to
publish
also we've looked at operational
workflows meaning one service needs to
indirectly communicate with some other
service to trigger some business
activity in the in the latter one not
all processes or flows are of that
nature some are more
analytical so for example when I took
the CF cab from the airport to my
hotel I'm or rather the system the bolt
system that I implicitly use gener Ates
a price for me how important is the user
context there could be useful or maybe
not depends on what the analytical
workload
entails
so yes you want to enforce zero trust
but it might be less important in
certain
circumstances but you should probably
not avoid it entirely but be aware that
not everything uh is as enamored of this
identity context that you propagate as
you might think some scenarios are less
demanding of having that identity
context
present I'll you see a bit more of this
on the next slide event sourcing how
many you heard about event
sourcing some of you it means that we're
storing events long term we're basing
all of our state in one bounded context
in one or more services in that context
on storing events and replaying them
when we need to inside of that cont
context now typically when you process
events without event sourcing you don't
store events for a particularly long
time you have a window of events that
you process and after that if some
consumer enters the picture wants to
consume events they're not available
they have to wait for any new events to
be uh sent and then consumed but with
event sourcing you're storing them
potentially
forever what does that mean for your
expiration well
you have to set it to a really long time
we mentioned that we need to extend it
in this time what appropriately long
time means is up to the actual
application but it needs to be set to
such a time that if you replay this
event stream a year from now say you can
still do access control so the token
hasn't
expired normally this is not such a big
deal
because um
when you process an event inside of a
context you might for example send a
notification via email but you only do
that once if you replay the event stream
a second time or a third time you're not
sending additional emails you're maybe
using the Replay for other purposes say
building a view that you can
query so we're typically avoiding side
effects when we replay these events so
again the identity context will not be
as important as the first time when you
actually wanted to check can this user
send out an email to this target of
emails email addresses but once you've
done that you don't maybe need to check
for that second or third time because
you're not sending out those
emails um after that first
time so again if you have these
scenarios where the identic context is
not that important again for example for
analytical workload
you might um not disregard but downgrade
the importance of me the actual user and
exchange me for a system user or system
role how many of you are familiar with I
am in the cloud you have users that
invoke an action for example
to uh deploy a Lambda in AWS but that
Lambda when it needs to talk to a
database it will not use my credentials
it will assume a role to contact that
database on my behalf so to speak we can
have the
same um thinking here we will elevate a
system role instead of my identity as
the primary thing to check
for and this comes into play for example
when we want to handle user removal so
let's say that I've been authenticated
and authorized in some async
flow and
we are
replaying an event
stream but I have been removed from the
system I'm not I don't have an account
anymore again events are immutable and
if they're stored longterm like with the
vent sourcing my token the information
about me will still be there even though
I'm not a user anymore in the system my
account is
gone so this is where a system role can
probably be better so it's an assumed
role and we'll check and do zero trust
enforcement with the system role instead
of
me so again this can come up when you
have analytical scenarios like
determining a price surging price um or
with event sourcing can also happen if
the background processing is not fast
enough so an event is published onto a
broker a consumer is expected to consume
it right away but for some reason it
doesn't happen so the
token um is still there but I might be
removed as user then we need to handle
it properly in this manner as well
through a system user system
role and um if I'm removed we can also
check this in our Access Control policy
so if you use this Opa Rego language you
can see that if I as a user am no
longer part of it the system if I'm not
if I'm present in the user's removed
array well okay then access control
cannot be enforced for
me this is a very special case though so
again I just wanted
to uh provide the notion of sometimes
zero trust and enforcing Access Control
will occur for a system level role
rather than me which is the sort of
original uh user that we need to
verify and with
that that's that's all I had to say
about the topic of access control and
message driven systems again this is can
be tricky to implement but it's mostly a
conversation with you between you as
application developers and your your
network administrators who work with
security on their level but there are
some simple mechanisms you can Implement
to enforce zero trust hopefully youve
got an idea of how to do it in this
session thank you very much and if you
have any questions please go
[Applause]
ahead