all right yep we're good okay
so thank you everybody for coming
um we have a very very big room very
full room so thank you very much for
coming and choosing me over all the
others so
um thank you very much so today we're
going to be talking about operable
software and more specifically about how
we build that using tdd
and it's not necessarily the way you
think it might be it might not but um
for a lot of people and in my experience
it's not so who am I so my name is
Martin Thwaites I go by martin.net on
the twitters follow me for um stuff
about observability with a hint of
British snark in there so you've been
warned
um I'm first and foremost an
observability advocate so I
have for many many years been pushing
the ideas around observability and
observing systems that kind of stuff
um I'm also a developer advocate for a
company called honeycomb we have a booth
concealers blah blah blah you all know
the thing okay
so
um before we get into the agenda I want
to talk a little bit about the
motivations for doing this talk so
before I work for honeycomb I spent many
many years Consulting and for the last
two years I've spent um the time
building a
um a credit lending Bank platform it's
not really relevant other than the fact
that it's
um very it needs to be very very
consistent it needs to be up all the
time so completeness and things like
that are really really important and
what we found was the ways that people
were writing tests the way that people
were building the software wasn't really
building it in a way that you could run
it in a bank and we want it to go full
CD and all of that kind of stuff which
requires a few more things so we built
over the last two years a way of doing
things that we believe is really really
good
um and that's what I'm here to talk
about today which is the reason why I
love talking about it because I'm
basically talking about what I've done
for the last two years so
cool
um so what we're going to do today we're
going to go through
um what I consider operable software
um we're going to talk a little bit
about how as an industry we toss we test
software right now we're also going to
talk about something called outside in
testing
and then we're going to talk about how
does observability fit into this whole
thing
um now some of these terms will be
things that you know some of them they
won't some of them they'll be new but
hopefully what I'm doing here is I'm
going to bring together a load of
Concepts that exist in books and blogs
and all that kind of stuff and give you
a practical way that you can use them to
make your software better Okay so
let's start by talking about operable
software what is it we mean by operable
software or what is it I mean by
operable software more specifically
now for me it's building for production
too many times people build for their
local development environment they are
fine you know we all know the term it
runs on my machine that kind of stuff we
build for that environment of our local
machine building operable software is
software that's built to operate in
production so it's built to run on
production servers it's built to run in
a production environment and that's what
we're building for that's what we're
catering for we're not catering for a
localized Docker container we're
building for that production environment
and that's our primary concern because
anything else isn't really what users
want
it's also built with live operations in
mind so I'm talking about the the Ops
Team whether that's you or somebody else
but you're building it so that that team
knows what's happening that team is
actually running your software the fact
that it runs on your local machine is
irrelevant
the fact that it runs on production is
the most important thing and that that
team whether it's you or somebody else
are the people that are actually running
it
it's also built for fast recovery you
may have heard mttr I know it's falling
out favor now but the idea is we need to
be able to recover our software fast and
that is what we should be doing when
we're building it we should be building
it so that we think that this thing will
be able to fail over this thing will be
able to recover quickly
and all of that leads to the idea that
our application should be emitting some
robust Telemetry signals to tell us
what's going on
and all of this is stuff that you should
be doing day in Day Out and these are
the things that I miss and when I see
Engineers working with software is they
don't see these things they're quite
content with hitting the big play button
in Visual Studio putting the debugger on
yes it works great ship it
um which is only part of building
software
you build it for the production systems
you build it for users
now what do we when we say software
what is it we mean
because well we write code we write
applications
but that's only part of it
the software that you build is
everything
it's your application
it's the test that you write to prove
that your application works
it's your CI CD pipelines that allow you
to do that in a centralized environment
that allow you to deploy this code to
production
it's the infrastructure that hosts your
platform
but it's also the tooling from your
observability tooling
that allows you to know whether that
system is running
all of that is the stuff that you build
that is your software that is the
boundaries of the thing that you're
building it's not the little application
bit in the middle
because if you build an application and
you don't know that it's running it is
not running
so
let's talk a bit about testing
and more specifically about CDD
because it is the new hotness well old
hotness medium hotness some hotness in
there somewhere it's been around for
many many years and
it's often done very very wrong in my
opinion
um I'm going to prefix the entire talk
with this is my opinion okay
um
so
what do I mean by tdd
contract first now when we talk about
doing things tdd
we're talking about working out how
something interacts with our system
first we're defining how somebody thinks
going to interact we're defining a
behavior
Within doing red green V Factor
most important bit being the refactor
which everybody misses but you know hey
ho
um but we're also talking about test
first and implementation that's the most
important thing when we talk about tdd
now the thing is you'll notice there
I've not mentioned a class
I've not mentioned a method
I've not mentioned moq
that's tdd
now I'm not saying I knew up a class
I'm not saying that I inject all my
dependencies you're in moq using moq
that's tdd however
tdd is often misunderstood as mock all
the things
um
I don't know I think my my biggest class
that I've seen somebody mock had 42
injections into the Constructor
um
and they were writing tests around that
and it's
but they were writing tdd because they
write the the test and then they do the
implementation but they inject up
sorry trauma
um
but it's also misconstrued as well as
the the idea of single method tests I
have a single method I test that method
I inject some things and I make sure
that my mock has been hit because that's
what we do with tdd
and it's really really not so
if we talk about tdd and we talk about
why is it good why do we want to do tdd
what's the reason why we should do tdd
because it's not got any benefits then
we should all stop doing it but it does
have benefits
so the biggest thing for me about tdd is
it improves the design of our system
because we're writing a client that uses
these methods and if we can't use our
methods if they don't make sense to us
when we're writing them then sorry
they're wrong if we have to pass in 15
parameters instead of passing in a
complex object probably not the right
thing to do because we're struggling to
use it and if you're struggling to use
it then other people are struggling to
use it so using tdd as a method to be
able to write our code helps us enhance
those contracts
it also improves our own understanding
of our code base
because we're using it we're using these
classes we haven't just written wrote
that class and then used a service to
call that class we're actually using it
in various different ways so we
understand how that code should be
interacting what the outputs are
so our own understanding of our code
base
will increase
we also are documenting
how our code works
because that's what your tests are doing
your tests are using it and if you've
wrote them properly
then your documentation is actually in
your code in your tests
so you don't need to write comments you
don't need to write
God forbid Confluence documents about
all of your code
um
do write a readme just to be clear put a
read menu repo
um it will help
so
why is tdd bad
um
so the problem I see with tdd and these
aren't the the reasons why tdd is bad
these are the reasons why the ways that
a lot of people do tdd is bad
and it's not that it's bad
it's just a misappropriation of effort
they do
potentially prove some things out but
that effort can be used in much much
better ways
now the thing is tdd is normally done in
the absence of consumers so you're doing
tdd at your class down here
and you're the only user of it up here
and then your users are actually all the
way up here and they're using your apis
but your class is solid right you know
you've got four thousand unit tests on
this class
and there are you know 12 000 classes in
your application but yours is solid
you've got 4 000 tests
with the absence of these consumers here
what you're doing is things in isolation
and you're potentially not doing the
right thing you're not building the
right things because you're
concentrating on that class
the other thing is they can be brittle
which is one of the annoying things
um the idea that you write lots of unit
tests and then you do some refactoring
and then you have to refactor all of
your tests
that means you've not really got tests
because
I was talking to somebody recently who
their entire way of doing this was if
they had to do some refactoring they'd
delete all the tests and rewrite the
tests
trauma
um so you know that idea of making these
tests so close to those classes means
that you can't refactor
I would Hazard a guess that a large
proportion of the room shies away from
doing refactoring on a large scale
because of the amount of tests it would
mean that they need to refactor and they
don't really know what those tests do
and if you are in the here going that
isn't me you are lying
okay
so let's talk a little bit about
component test now everybody's I'm
assuming seen this
um so component testing is a subset of
um integration testing integration
testing being
a really
abstract term that doesn't really mean a
lot of things anymore
um it can mean 20 different things to 20
different people
um so what do we mean by component test
or what do I mean by command test now
when I talk about component tests
I'm referring to the idea of outside in
these are tests that we write from the
outside of our application
not a class we're writing them from the
outside of our application
because that's what's important nobody
cares that your class has 4 000 tests
and runs really really fast what they
care about is whether the users are able
to hit the apis
that's what they care about
nobody is going to praise you for doing
four thousand unit tests on a single
class
so
what we're talking about is working from
the HTTP level now I'm using this in the
context of a service obviously if you're
writing libraries the outside isn't your
HTTP service area
if you're writing an internal library
and in a source pattern something like
that your external API service is going
to be the API
so that's the surface not your internal
Factory classes and not your factories
of factories and your factories that
generate factories of factories those
aren't the things that we're talking
about here we're talking about that
external API surface oh sorry we're at a
net conference not a Javon okay
um
cool um so it's also about testing
inputs and outputs what we're doing is
we're testing our API from a perspective
of the user
we're testing that we can get data in
we're testing that we can get data out
we can we're testing that we get data in
and our response comes back right those
are the things that are important those
are the things that are important to
users
so we're testing the inputs and outputs
we're not testing the bowels of our
application because nobody cares nobody
cares that you're hosting on
versus SQL your user really doesn't care
unless they're doing a PCI audit which
they probably do
um
but we're also talking about testing
from a user or consumer perspective
we're talking about the idea that a
an end user being maybe it's your you're
writing an API platform it's the people
who are consuming your API it might be
that you're writing a library they might
be writing an internal service the user
is the person who's going to be hitting
the thing that you build not you as a
developer
obviously you might be doing a
multi-service architecture where you are
your own consumer
um and don't check your own homework
just get somebody else to do it
um but the other thing is we're testing
business requirements
because we're testing at that consumer
level
we're testing at that edge
what we're actually doing is we're
testing from a
um a business user saying
let's use the bank example
um we're testing that we can do an
authorization and that authorization
succeeds if we've got a good balance
that's a business requirement
I don't care that the balance class gets
hit I don't care that the transaction
class gets a transaction added to it I
care that when I do my transaction up at
the top it succeeds that's a business
requirement
no business is going to tell you make
sure that your balance class returns
true
if they do then you've probably not got
a business analyst
um you've probably got a developer
um so what do we get from doing testing
in this way in an outside in pattern
well like I say we're testing the
business requirements first we're not
testing arbitrary things inside of our
code base
we're testing what users actually use
we're testing what users actually want
that's the thing that we're testing
now the other thing is that if you're
writing them from this way you get the
ability to write a lot more readability
into your tests so your tests become
more understandable
you can use terminology that is
terminology that you use
as a business
not terminology that the the transaction
service class has been hit
well you don't really care you're using
business terminology which makes things
a lot more sense
now the other thing is each of your
tests is holistically testing your
application
you're not testing your class down here
with your 4000 tests that you're really
proud of
we're testing the application we're
testing the thing that is going to be
deployed and this is where I come down
to the building for production we're
building operable software now we're
building software that users care about
and that operations can use
because they don't deploy your class
with 4000 tests
the other thing is we don't do any stubs
or mocks
and I'm sorry if the creative remark is
here but I'm trying to get rid of you
from every single organization I've ever
been in
foreign
now the the number one benefit for me
is that we get 100 code coverage
does does everybody strive for that in
their application
do you have it in the build pipeline do
co-coverage tests doesn't get 100 fail
how many of you have um been writing
tests and then realized that you can't
test the dispose method
so that you you add something in the
dispose method so that you can test the
dispose method so that you can get 100
code coverage
um
what you get with outside in testing is
the ability to say well
if that code isn't here
one of two things has happened
I've either
got a missing use case a missing test
that says well this is how the system
works and that's why that code exists
or you delete the code
because it's never been hit
one of those two things is true which
means you can get to 100 code coverage
not that 100 co-co coverage is a
positive metric it's more of a negative
metric the lack of code coverage will
tell you that you haven't got enough
tests 100 code coverage does not tell
you that you should stop testing do not
stop testing at 100
okay
now what do we mean by readable tests
now when I talk about writing readable
code
I talk about writing readable tests as
well we should put as much effort into
our tests as we put into our actual code
because the tests are more important
than the code the code can be replaced
your tests will tell you whether your
system works whether it's functionally
accurate
those should live longer than your coach
would
should also be refactoring our tests as
well
again something that people miss when
they're doing tdd
so the first thing is test names that
descriptive make your test name tell me
what you're testing
what's the precondition what is the
thing you're testing the amount of times
where I see a test that says
does the thing
it's like I call the class and it
Returns the correct response
okay but what's the correct response
tell me what you're looking for are you
looking for the right balance are you
looking for the right name on the
account are you looking for the right
surname are you looking for the right
first name you've got a response on your
API that's this big
you're not testing everything every
single thing on there
what we're doing is we're giving
ourselves signals so that when that
thing failed when that particular test
fails it tells us what it was testing so
we'd have to go delving into it and
looking at cert messages and stuff like
that to try and find out when it is and
God forbid don't use equivalent to
please
okay reusable actions so when we're
writing re um when we're writing
readable tests
if you were writing your code
and you decided that you were going to
reuse a bit of code you would extract
that into a class right you would give
it a nice name some context
and then you'd reuse it over your things
the amount of people who don't do that
in their tests
is immense
we can do a lot of things with tests
because they're code if we're writing
things in X unit if we're writing things
in C sharp we can use reuse vaccines we
can do extractions and we can give them
context which makes our tests make more
sense we can use readable names for
these things
we can also work on abstracting the
implementation details
now if you're writing a test
and the first thing first line in your
test is newing up the object that you're
going to test you're probably doing it
wrong
because those things aren't important to
the test that you're writing the fact
that it news up that object is noise
abstract away any implementation details
just make it make sense and we'll go
through some examples these are not all
abstracts I've got an example that we're
going to walk through that will show you
some of these techniques about how we do
it
now the other thing is we're using
domain language do not use the technical
language use the language that your
consumers are going to use when you're
writing your tests
if it's to do with an a authorization so
in the banking World you've got
authorizations and transactions they're
kind of one and the same so
if you've got one endpoint that says
transaction and sometimes it does
authorizations and sometimes it does
transactions well in your tests use
authorization and transaction as names
give that context use that domain
language rather than using the technical
language well I'm hitting the
transaction endpoint so I should just
call it transaction because I understand
what that means
now the other thing that I think a lot
of people think is controversial is
don't add 40 asserts in your test
break that out into individual tests and
it comes back to that first point
which is make your test names
descriptive
yes you can use the equivalent to
and it can massive object like this
but then what you have is a test that
says returns correct response which
doesn't really give you that much
information about why it went wrong
have more tests don't shy away from
having more tests
if you write reusable reusable actions
if you write your tests in the right way
writing another test is copying three
lines into a new test and then doing
these search and giving it a name it's
not onerous
these things run in nanoseconds most of
the time
so adding more tests isn't going to make
your test framework slower
but it does make it easier for you to
understand and it makes it easier for
other people to understand
because at the end of the day we don't
write code for ourselves we write code
for other people
how many people have come back to a code
base and gone oh my God who wrote this
this is oh oh right okay
so sometimes the other person is
yourself
so be kind to yourself
um so the other thing that comes up is
well is what we're doing here not just
bdd
well we're defining behaviors and we're
testing behaviors well guess what tdd
was about testing the behavior not the
implementation
that's what tdd was about
what we're not doing is we're not
writing gherkin tests there is no given
when lens we don't need that kind of
complexity we're writing the code
ourselves we're writing the test for
ourselves
so we can write them with proper c-sharp
code we can write them with reusable
test names we can write them with
reusable actions we don't need to write
it in a way that has given one then but
yes we are testing the behavior
so yes the original definition of what
bdd was about yes this is what we're
talking about
cool so
let's have a look at some code
um as a an example now I'm going to be
using here something called the web
application Factory which is a
um a class in.net that allows you to new
up your entire API
with all of the dependencies
in memory
so there's no HTTP call going out over
the network
it runs it all in memory
and I've wrote a little to-do app
there is a code sample there'll be a
link at the end that you can have a look
at and purposefully don't put it up so
people don't want to look at it now
because then you'll see the answers
um but what we're looking at is a to-do
app
and it's not bad test name that we've
got there so what we're going to do is
we're going to add a to-do to the API
and just make sure it returns okay
now people are familiar with tdd right
the most simple test first
well this is the most simple test we'll
not even test that it saved it anywhere
but we've made sure that it actually
returns okay it might have gone into The
Ether I mean obviously the easiest
implementation of that is returns okay
in the method great let's do it but the
idea is we will write this first
it will fail
because we don't have a to-do endpoint
but we can write tests like this in a
tdd fashion write the test first
describe what it is that your API should
be doing
and then we'll write the test
now
that test there will probably make sense
to the majority of the room if you're
all Developers
and it will make sense because it's like
what five lines five real lines
but
we can do better
so we're going to use this really small
example
to show what we can do to make this
small example really really be usable
and understandable in a wider context so
first thing we're going to do remove the
implementation
now if you look before we had a web
application Factory it does not matter
for this test that I'm using using a web
application Factory that's an
implementation detail I don't care
when you look at that it feels like
that's important to my test that I'm
using a web application in fact I'm not
it's not important what's important is
the other bits
so what we do is we're going to move
that out so in in essence what I've done
there is I've just moved that into the
Constructor or into a base class so it's
not there it's not important for my test
and already this test is making a little
bit more sense because I've extracted
that out and I've called it API
so now what I've got is api.post
and I've got my ad response
so API to post to the to do great
makes a little bit more sense now
because it's more succinct
but we can go a bit further
we can now work on a triple a pattern
I'm presuming majority of people have
heard of AAA arrange access
so well let's define what object we're
trying to do because the object is
important to us
or is it we'll sign we'll find out
and then we do our action and then we do
our assert
okay so now we've got a pattern that
makes more sense in people's head
but we can go even further now let's add
some context
because the values
from that object aren't important
what's important for this test is that
what we've got is a valid to do
we don't care that it's got those two
properties those aren't really important
to us
and this is called evident testing
make sure your test only includes the
things that are important to your test
because when you come back to it and
look at it and you go
is the title important
is it important that it says new to do
is that the only thing that's acceptable
does it have to be that length is that
important no
what's important for this particular
test which is our with valid request is
that there is a value to do so I've
given it some context and called it get
valid to do and called it a to-do item a
valid to do item more specifically
now the other thing is well
we've got that test leave that test
that's good it's a perfectly valid test
that's going to say it'll return okay
I'm going to add another set now if you
look at those first two lines those
first two lines are the same as the last
test
so to create this new test which this
new test is going to do two assets
notice how I didn't say only write one
assert per test I said don't be afraid
um to write more tests I'm not putting a
number on it it's not five it's not ten
it's not a specific number of asserts
that's okay
it's about writing things for yourself
later that allow you to be able to see
what's happening
so
what we're doing now is we're writing
another test
and this one is going to specifically
test that we've got a valid ID that's
come back
now I'll return okay that we had before
it's not going to work anymore so this
will Now fail we're now in a test driven
development cycle we're adding more and
more tests that allow us to write the
implementation detail but at no point
here have I reached into a database
at no point here have I injected some
data
I don't care whether this is hosted on
or SQL or Cosmos or any other
system I don't care the user doesn't
care so the test shouldn't care those
are implementation details
but we do want to check that things are
persisted
so we want to check that when we've done
our to-do item because I can make that
test pass pretty easy
by just return one on an okay response
great test passed I love adversarial tdd
testing it's amazing
um when somebody writes a test and then
you write the smallest implementation
and you control them really bad it's
really funny
um so
what we're going to do now is we're
going to
work on
our next test and our next test is
well we've created our to-do item we
need to make sure that we can get our
tattoo item back out
well we need an API call for that
one of my favorite things to do with b8s
when they start to get into technical
detail when they go must save to the
database
as a acceptance criteria just say Yes it
saves to the database and don't do it
because until they've given you criteria
that says I need to get it back out
again
saving to the database doesn't matter
because you can say yeah it's in the
database and they can't check
until they say to you I need to be able
to get that thing back out of the
database on the get request
you shouldn't even be thinking about how
you store it because these things where
we say need to go and get it these are
what are going to define those
requirements too many times when people
write tdd they jump to that the
conclusion that they need a repository
class and they need to test their
repository class
you don't until you do until somebody
gives you that requirement so this test
here
what you can see now is I've actually
abstracted all that stuff from the
previous test
so those first two lines there I've
abstracted again
into create value to do item
because what do I care about in this
test
well once I've added a to-do item my
range
I should be able to get that to do item
back
and it should have the right IDs
and again what I've not done here is
I've not gone into the database and
injected and bleeded my schema out and
the database implementation into my
tests
what I've actually done is I've used my
API
to post that to do item in and then go
and gut it
because too many times you have that one
class with 4 000 tests on it that works
really really good but doesn't integrate
with anything else
so
let's just have a look at what we've
tested there in that particular scenario
now this is a big one we've tested our
dependency injection startup
so how many people in the room have done
a feature wrote a really nice class four
thousand tests
and then forgot to put it in your
dependency Junction container
yeah the people who are not saying yes
you are lying
go away and think about your life
it happens
but you get it to production
and all of a sudden your application
fails
because well it's the dependency
injection it's not it's not getting that
class that he expects
with the approach of doing outside in
testing
it's already testing it the other thing
you're testing is routing
so you're testing the idea of does my to
do item route work does it work with a
slash or without a slash
do they both work is that important
and guess what I'd have two tests for
that
one for weather slash and one for
without
so we're also testing a really important
thing which is serialization and
deserialization
now I always Advocate when you're
writing these tests
to write your own models in your test
you know everybody seems to love the
dot-common class inside of their
solution that stores all of their API
models
and then they reference that in their
class in their tests and then well when
you refactor your your common thing well
your tests have been refactored as well
oh oh right yeah just broken all my apis
sorry
um what you're testing is serialization
and deserialization which means you
actually get Contract testing out the
box
because if you've got to change your
models inside of your test class
well you know that your consumers have
got to change their code
so now you've got a breaking change
so you can actually alert on things like
that somebody's changed stuffing in the
uh in the API models so don't deploy
let's have an extra approval step
because somebody's changed the API
models and we don't want to break our
API
we're testing that serialization and
deserialization
anybody been caught out by the switch
between new and soft and system text
Json with serialization
yeah yeah
um it works subtly different and you
don't notice it until you notice it and
that's normally too late this tests all
of those things
because it doesn't care that you're
using newtonsoft he doesn't care that
you're using system text Json it cares
that an object came in in Jason
and then it just deserialize persisted
and then when it came back out again you
got the right value out
we're also testing model validation so
we're not testing our little service
class at the bottom here we're not
testing our little static helpers class
what we're testing is that whole thing
so if you put something on there that
says must be a number
and it shouldn't have been a number
well yes we're going to test that
and guess what there should be a test
for it that shows those things
the other thing with testing as well is
database interaction
now in the example that if you go and
look at the the code for it we'll use an
Entity Framework
so all I did was I switched out Entity
framework for an in-memory SQL light
database
so I'm actually testing all the database
integration as well I'm testing that my
where Clauses work
because your service class won't test
that
it'll test that you hit a mock and
obviously that's really good yeah
but we're testing everything from
selecting stuff to primary key
constraints foreign key constraints all
of those things are being tested
holistically from the output
but like I say we're also testing
business requirements we're testing what
the business has asked for this API to
do
and that is invaluable
because the amount of times where the
business comes up and said oh I didn't
mean it to that you can go back and say
look starting the business requirements
because I tested the business
requirements I didn't write an API on
the abstract
I wrote it based on what you actually
sent me because here's all my tests that
test all the business requirements
so you're covering your own behind
but without the other thing we're
testing because we're creating our own
example client of our API
well we're actually testing what the
customer sees
so we're a consultant we're a customer
of our API we're a consumer immediately
of what we're doing
which means that you're getting valuable
feedback about how to use your API when
you're using that API and you're going
oh this is awful oh I wrote it okay
um maybe I shouldn't make it awful
because what you're doing is you're
imposing on users something that you
wouldn't use yourself
so
I've told you all the awesome things and
if you look at that list
and I didn't show you anything else
everybody in the room is going right
okay this is something we should
absolutely be doing there's zero
downsides okay but there are there are
certain things that you can't test when
you do this
so there are things like code path
changes
caching being a really big example of
that
well I can test my API if I need to test
that it came from a cache or it came
from
um the database lookup I can't test that
because from the outside it looks the
same because it should look the same
from the user that's a new item that I
go and get from the database
well it looks the same whether it came
from the database or whether it came
from an intermediary cache or
um something like that
so from the outside I shouldn't know
we also can't do things like auditing
so we can't test things like an audit
Trail was created
or can we
we can't test performance
so we can't test the idea of how well
does this thing perform in production
because on a nanosecond scale that these
sort of tests run it's really hard to
notice that things at scale will run and
at the end of the day performance should
only really be tested in production
because that's your only real
environment that you've got anyway
everything else is artificial
we also can't test cloud services in
this model because they're generally
slower
so we can't test that it uses Cosmos we
can't test that dynamodb is being hit
we count as the configuration
to make sure it's got the right config
that points to the right Cosmos DB the
right dynamodb the right SQL database
we can't test that it's got security
access either
however
some of those things we can
and this is where
observability comes in
so let's talk a little bit about
observability
so I said at the start
the first and foremost I'm an
observability advocate
and the reason why I am is because I've
built and run production systems for the
past decade at varying levels of scale
and the idea of observability is really
important to that being able to know
that your system is running in
production and not relying on hopes and
dreams because
they don't work
so let's define what observability is so
charity Majors our co-founder at
honeycomb
um coined the term 2016 in the context
of software which is the idea that it's
about understanding and debugging the
unknown unknowns and it's about
understanding the internal um the inner
System state by asking questions from
the outside
now if you look into that you can
actually see why I think this is
important to what we're talking about
with outside in testing because it's the
ability to understand our inner state
well that's what we were just talking
about the things that we can't tell the
internal bits of our application that we
can't know about
well actually yes we can because
observability is about that idea of
knowing those internal States
so let's talk a little bit about
um observable of observability driven
development observability during
development whatever you want to call it
um there is no accepted term and I
really don't want to coin a term but
what we're talking about is using
observability during our development
life cycle
whether that's local development when
you're doing it and running up locally
or whether it's inside your test it's
the idea that observability isn't just
for production
you can also use that locally
now first thing is observability is an
output of your application
we talked about testing inputs and
outputs
of our application
well observability is an output it is
something that our system tells us about
our system
that we can use
there are certain things that
observability can do it can detect
parallelization you can test that two
things run at once
anybody
swear blind in their application that
two things run in parallel and until you
actually saw a waterfall chat and then
realized oh no no no they run in series
don't they yeah that'd be why the
system's slow
um
I've done that numerous times and the
amount of times where we go into
organizations put tracing into their
application
show them that system and they go yeah
that all runs in parallel and then you
go show them the waterfall chart and it
doesn't it starts when this one finishes
and they're like no no it's wrong
it's not wrong
the system's telling you that it's doing
that but observability can detect those
things and that's really important
it can also detect which code paths
because if you write tracing into your
application if you do manual
instrumentation you can detect which
kinds of code paths have been taken
through your application
now the other thing is that what's
important in
your local environment is likely
important in production
now if we take the caching example
where we care that a second request hits
the cash
that's important to us well that's
important in production as well
I should care in production that things
are hitting the cash when they should
hit the cash
so all of a sudden this is an output
that we need to have in our application
so let's look at a little bit of code
now what I wanted to prefix this with is
doing this approach is not a way around
you injecting things into the database
and reaching back around and just add
some things in just so that you can test
them please don't do that
but what it is is the ability for us to
know our application a little bit better
so this is a bit of caching code for any
future employers out there that might be
looking at me this is not how I would
write a caching code this is an example
and it illustrates what I'm trying to
talk about
so
for those who don't read code fast what
we're doing is we're creating a
um a to-do list we're putting that in a
memory cache and then we're going to go
and get the
um the to-do's from the database put
them in the memory cache next time
around we're going to go and get them
from the memory cache
so pretty simple implementation the idea
is first request goes and gets from the
database second request will always get
it from the cache
forever
don't write caching code like this
if anybody comes up and says Martin said
I wrote this and then I got fired
disclaimer okay
so
if we want to write a test
that says did we call the database or
did we call the Cache because we want to
assert that well the first call hits the
database no it's the test name
testing is really specific there about
what exactly we're trying to do
tells you exactly what we're trying to
achieve
so what we're going to do is we're going
to use something called open telemetry
open Telemetry is the number two project
on the cncf
um
Foundation it is the de facto standard
around tracing and becoming the dev
facto standard around metrics and logs
as well
and what it provides us with in this
particular context is the ability to
have a tracing provider there's a lot of
code there
which we'll walk through a little bit
but what we're doing is inside of our
test framework
we're going to set up a tracing provider
we're going to monitor some specific
Trace traces or tracers
in.net that's called activity or
activity sources
we're going to monitor some of those
we're going to add something called a
test spam processor which is going to
allow us to say which test run this was
part of
and then we're going to add some Auto
instrumentations we're going to add in
the asp.net core instrumentation which
will automatically do some tracing for
us but we're also going to add in
something called the in-memory span
processor or in-memory exporter
now what that's going to do
is it's going to say that every span
that's created in our application it's
going to stick it in a dictionary for us
it's actually a list
but it's going to stick it in there for
us
so that we can interrogate it
so we can know for an individual test
what spans were created
we're also going to add honeycomb in
because I work for honeycomb so I'm
obviously
okay
so
the other thing you need to do is once
you've got your tracer
you then need to use configure services
so if you're using the web application
Factory you have access to inject more
services so yes the stuff that you
define in your program and startup.cs
whatever you're using those are still
there
but you also have a hook to add more
services in so we're going to add in the
Tracer provider now
there's a lot of people who are
reluctant to bring in something like
open Telemetry into their application
because well that's an adoption thing
that's everybody needs to use it what
we're using here though
is open Telemetry in the context of
tests
there's not a wider adoption here we're
not talking about adopting open
Telemetry as a standard across all your
applications All 4 000 applications that
you've got all your services using
distribute tracing we're not talking
about all those yes you should do those
come see us at the booth afterwards and
we'll show you how but what we're
talking about here is using it for tests
we're using it for observability inside
of our tests so that we know how our
application is working so that we can
stay on the outside and build our tests
from the outside
so we're going to add in
the Tracer provider clearing the login
providers because on Linux it doesn't
like it
um a lot of the time but hey
so now what we can do
that caching code that we talked about
before
what I've done is I've wrapped it in a
using statement I've created an activity
called get to-do list from DB
and I've wrapped the bit where it goes
to the database
in that span
now the reason I've done that is because
that's something I would probably want
to report on in production in production
I'd want to know how many times did it
go to the database versus how many times
did it not go to the database
so
we've got a span created now
for every one of our calls that comes in
only when it hits the database
so now we can write a test
so now we can write a test that comes in
and says right go and do the get request
so API get to-do list great also don't
call your endpoints to do hyphen list it
was just easier
so we're doing a get for our to-do list
and then what we're going to do is we're
going to check our spans
on the back of that and say did one of
them contain this
yes okay well we're going to make sure
that that wasn't null now what I've done
is I've wrote a test that tells me if I
make one get request it's always going
to go to the database
great I don't need to German check a
mock I don't need to go and inject into
my application something that I can use
a mock verify to see whether it hit it I
don't need to do any of that I've stayed
on the outside and I've got really
succinct test now I've kept that and I'm
not going to refactor away that bottom
bit where it does the cache activity but
I would obviously go and then refactor
that to put some reasonable action and
give that context to say spam exists or
something like that that makes it
readable for you to use
but now what we can do
is we can write the next test
so first one same as we did before
get the to-do list
clear out any spans from that call
because we don't care about those
anymore
make the second one and just make sure
we don't get it
make sure there isn't a call there isn't
a span created
that has the to-do list from the DB
so I've been able to stand on the
outside of my application
completely
and now I've been able to test some of
those internals but what I've also done
is I've made sure now when I go to
production
I've got some observability around when
cash hits happen because if it's
important to me locally to know that
that code path was taken at some point
it's going to be important in production
so if you get those in your test and you
keep writing the test in this way
you are going to be able to get better
observability in production and this is
how why I talk about building operable
software because you're considering what
it would take to get to production
you're considering what production would
look like
you're not writing it for local because
let's be honest if you wanted to do that
locally yes you could write some mocks
in your controller classes inject all of
your stuff in make sure it goes to the
different different places or just hit
play
and then debug it through yes it works
because I debugged it
now the other benefit that you get from
this is you can then go and look at
these results in your observability back
end
obviously on this one we're showing
honeycomb
so each one of our test runs now becomes
a Trace
that we can then look at
not that impressive when you look at it
in a graph like that
but if it's a trace waterfall
I can actually look for each one of my
tests that's run each one of the traces
that's been created I can follow through
how that hit my code
I can work out well was that slower or
faster than the last time and yes we are
talking millisecond nanosecond scale
but it will show you
and you'll be able to say actually no
this went through a different code path
this time the test failed okay well it
went through a different code path than
it went through last time
it kept it in the database oh maybe the
caching's not working so it gives you
more things to look at
and you can look at things like how long
did my test run go which ones were the
slow tests
with the Advent of stuff like GitHub
actions we're losing a lot of the things
that we used to have that were built
into CI CD runners
but this allows you to do that but it
also allows you to say this is what it
looks like or should look like in
development in my tests
when I go to production does it look
like that because that request should
look identical because the trace should
look identical
if you've got a test in place that tray
should look identical
that is all I've got to show you I
wanted to give you a few different
jumping off points from here so there is
a repository that you can go and look at
there's a few things that I also think
would be really useful for you to go and
watch after this so
um Ian Cooper who is on in the other
room thanks for coming to see me not him
by the way
um
a lot of the things that I talk about
here come from
um the tdd Revisited talk that he did at
NDC London a few years ago
um and it's a really really good talk
that breaks down what tdd is in a much
much more depth way than what I've
talked about today
the other one is a guy called David
Whitney who wrote A a talk called
writing tests that don't suck
um and he did that at an UC London as
well
um go and view that that is a really
good talk it goes a lot more into depth
about readable tests and writing tests
that are better not specifically about
outside in more about just writing
readable tests that are really really
good
um obviously I work for honeycomb and
wouldn't be here without them so you can
have a free account forever for with us
and come see us on the booth if you want
to chat about it and we also wrote the
book on observability engineering quite
literally
um and it is free for download um if you
go and get it on our website
and that's me thank you very much
I have a little bit of time for
questions if people want to onto our
stuff but
um if not just come up and chat