hi good afternoon everyone uh I'm gonna
get started straight away I'm going to
try and pack a lot into 60 minutes which
is always my problem
um my name's Steve I'm a Microsoft MVP
pluralsight author and engineer at
elastic and before I get into how you
can contact me there's a bitly link on
this slide if you grab that that is a
link to the slide so if you want
anything to review afterwards or the
resource links and things just grab that
one link and you have everything you
need
so if you do want to reach out to me I'm
at Steve J Gordon on Twitter same
username over on Macedon you know just
in case and stevejagorn.com UK is where
I blog pretty infrequently at the moment
but hopefully more so in the future so
I'm gonna I'm gonna talk to you today
about writing code with code and
specifically using uh something called
the Roslyn apis
so to set expectations uh we'll do an
intro to Roslyn so don't worry if you've
never heard of it or you've never sort
of worked with it before we'll talk
about what it is and why you might use
it well specifically then look at some
demos around visualizing syntax trees
and generating code from those syntax
trees don't worry if you don't know what
a syntax tree is we touch on that too
and specifically where this is kind of
Applied practically is in co-generating
the next version of the
elasticsearch.net client so I'm going to
talk a little bit about like how we
determined creating a spec transforming
that spec into sort of models that we
can sort of represent with c-sharp types
and then actually building those syntax
trees emitting the C sharp files and
some of the lessons learned and things I
would probably do differently with the
knowledge that I've learned as I've gone
along
so let's start with Roslyn Roslyn was
originally the code name for a project
that kicked off what we're about to talk
about Microsoft are pretty good at good
code names often they then ditch them at
release and come up with a boring brand
product name they kind of have the.net
compiler platform SDK is kind of the
other name for this thing but Roslyn
fortunately is stuck so you will still
find stuff in the documentation and the
repos and you can search Rosalind
c-sharp and you'll find probably what
you're looking for
so as I say it was sort of initially
kicked off in 2010
um and then I think it was a sort of
first released
um after some previews in 2015 and
ultimately what Rosalind is is a set of
Open Source uh compilers for c-sharp and
vb.net open source is pretty
self-evident it's on GitHub you can view
the code you can play with the code you
can learn from the code and create
issues all of that kind of stuff
compilers for c-sharp and vb.net so the
interesting thing there is that the
compiler for c-sharp is now written in C
sharp so to compiler the compile the
compiler you use the compiler that
you've just written and same for vb.net
so the interesting part of this
particular statement is open box
traditionally combined compilers have
been quite a closed box you put source
code in one end and out the other you
get some kind of artifact like an
executable or a dll and what happens
inside is kind of opaque to you and you
can't really access any of that
information and you might not think
that's important but the Microsoft team
did and ultimately what they've set out
to achieve is producing a compiler
platform which is basically opening the
box and giving us access to the apis
that Roslyn can give us to see what the
data structures are that the compiler
can see and hook into various points
along that compiler pipeline if we want
to
so this is used heavily inside like
Visual Studio to provide all of the
features you're probably using day to
day without really paying much attention
to them ultimately I summarize this as
the maker of squiggles and the finder of
things it's what understands your source
code and is able to give you hints and
intellisense but also to highlight when
you've made mistakes if you forget a
semicolon on a statement it can give you
a syntax error squiggle that tells you
you've done that and also we have other
sort of more advanced squiggles that
give us like code refactoring so ways to
maybe make code more concise or move
some more modern c-sharp language
features
finder of things like go to all final
references go to definition searching
for particular symbols within your
source code they're all enabled through
things like the rosin apis
so one of the concrete examples of where
these can be useful is like building
analyzers and code fixes so a lot of
these are already available to you like
just when you use the framework but you
can also bring in analyzers and code
fixes in nougat packages so people can
write these and ship them on nougat for
you you can write your own internally
within your business so ultimately what
an analyzer is is it's something that
contains code that recognizes some
violation of a rule
and that rule can be pretty much
anything related to the code that's been
written uh the structure of it but style
and naming conventions you could if you
want to write an analyzer that says you
can never have a class name that has the
word builder in it and then a force that
across your organization
now once you've got sort of an analyzer
to detect a problem which will appear as
a squiggle in your in your code base and
you will configure that as maybe
informational or warning or error it's
nice if you could also have something
that fixes it for you rather than just
telling you you've done something wrong
here's how you actually change your code
to apply with the rule and so a code fix
is essentially that it's something
that's actually going to fix that
violation in the code for you because we
have all of the information through
Roslyn about what the syntax and
semantic meaning is of the code I'll
show a quick demo of of sort of
analyzers in action
once the resolution fixes itself and so
inside this is actually something we're
using inside the elasticsearch
um code base for the client and so as a
library we're recommended that we should
apply configure away Force to the end of
anything asynchronous don't worry about
why specifically I don't have time to go
into that but we're doing it and we can
easily forget to do this so if I
actually delete this here
and just give Visual Studio a second or
two to realize I've done that
we get a squiggle and if I hover over
this we can see telling me I'm possibly
missing a configurable weight Force here
so we've installed through a nuget
package an analyzer called configure
awake Checker and then this picks up the
fact that we haven't applied the
configure away on the end of this method
cool
and so if I go in here and I can say
well I want to correct that then to put
it back on again and that's the code fix
putting that back for me similarly in
box we get something like if we forget
to await an asynchronous method again
Visual Studio will moan at me here and
stay because this call is not awaited
execution of the current method
continues before the call is completed
and then says to consider applying the
await operator and again we have a code
fix to add that back
so that's kind of in a nutshell some of
the uses that you probably use daily
inside Rosalind it's something more
exciting more recent is Source
generators this isn't a full talk on
Source generators there was one this
morning on incremental Source generators
but just to summarize it if you haven't
heard much about these These Are A
c-sharp compiler feature that lets
developers inspect user code while it's
being compiled you might think well do I
care about that why would I want to know
about that well you can develop
components that run during that
compilation of your code to access the
rich metadata about the code the
syntactical structure of the code and
some of the semantic meaning again you
may still be thinking okay I don't care
you can create new c-sharp source files
at this point so you can create them on
the Fly and add them to the compilation
and if you're still searching for
reasons for it let me give you a couple
of examples so where these are typically
useful is reducing boilerplate code in
applications so an example in the
Microsoft docs I think they actually now
ship the source generators to fix this
or to help with this is when you need to
do like I notify property changed and
you need to implement that and there's
lots of boilerplate code you'd have to
write today you can just plug in a
source generator by putting an attribute
on a few things and then the source
generator compile time can put all that
boilerplate into your compilation for
you and that saves you a lot of code
that you would normally have to write by
hand and avoid you making mistakes and
missing things
another more interesting example I think
is where we can actually improve the
execution time of code potentially so
inside system text Json when you're
serializing a type to or from Json the
first time that happens typically a
serializer is going to have to start
doing some inspection usually using
something like reflection to understand
okay well this is the class they want to
serialize to or from here's the
properties it has here's the attributes
some of them have Json ignore some of
them don't here's the Json property name
Etc so it needs to build up that model
and that meta information to perform
serialization and that happens normally
at runtime and it gets cached but it's
still overhead with Source generators we
can actually do that up front because we
have the finished code that you want to
produce and you're compiling and so the
source generator can see the same
information about what properties are on
the types and what attributes are
applied and so it can spit out a c-sharp
file for you that contains all of that
information pre-calculated and optimize
source code and serialization
so that's pretty cool
uh we're not going to spend time on
compiler Theory
um but just very very briefly there is a
compiler pipeline so ultimately we write
C sharp source files they go into the
compiler it has to parse them so
understand where the tokens are in that
text file that we've given it we have to
identify any symbols that have been
declared in our source code so methods
and classes that we've defined also
methods and classes that have been
defined in stuff that we're importing so
you know new get packages we've pulled
in and then there's a binding phase to
understand so what are the results of
expressions and variables in our code
what do they actually bind to and what
symbol do they bind to and then finally
emitting IL
and so this is the phase where we
actually produce IL code that actually
is the result of the compilation so C
sharp VB dot f sharp compiled to Il IL
is an Intermediate Language so C sharp
VB F sharp high level languages machine
code low level language the Intermediate
Language sits in the middle and the nice
thing is the compiler is all just enough
to know how to produce IL code and that
can run on any architecture on any
platform where.net can run and then the
runtime is responsible for taking that
IL and just in time compiling to machine
code so as methods are needing to be
invoked if they haven't previously been
compiled then they can be compiled at
that point
Rosalind includes a very rich compact
compiler API that Maps very nicely to
that compiler pipeline so you can see
there the parsing phase has a
corresponding set of apis within the
compiler API that to do with managing
syntax trees there's an API for symbols
there's an API for emitting IL code
and so it's these apis that we're going
to be sort of talking about specifically
we're going to focus in on this level
the parsing level and the syntax tree
level
so that's introducing tax trees
syntax trees are the result basically of
the analysis phase of the compiler so as
that parsing phase has gone through the
code and it's worked out where all the
tokens are it understands the syntax
that we've written
and so from there we can build up a tree
representation of that syntactic
structure of the code and this includes
three main things nodes which are
generally going to correspond to kind of
language constructs so like a class
declaration or a method declaration
would be defined as a node in this tree
nodes will always have a parent except
for the root node and nodes may have
child nodes below them so a class can
have methods and properties and those
are nodes below it
nodes also are made up of tokens and
tokens are really the individual
language keywords that correspond to
that declaration to that node so for
example we can put a public keyword on a
declaration for a class and then we've
made it public but that token has to be
attached to that node so that we
understand that it's part of that
declaration tokens are kind of
Terminators within the tree although
they do have trivia attached to them and
trivia is anything within that source
code file that isn't something that's
going to be involved in the actual final
compilation of idle code so this is
white space new lines and also things
like comments single line comments
multi-line comments XML documentation
comments that's all structural things
that we can include in a c-sharp source
file that aren't necessary for
compilation but we do want to represent
them in the tree so that we understand
where those things belong because for
example with an analyzer or maybe even a
code refactoring if it's going to move
some code around in that tree it needs
to make sure that any comments that
belong to that line follow it around as
well now
ultimately what a syntax tree gives us
is the ability to interact with source
code in a more meaningful way so we're
no longer just looking at a text file
that needs parsing but we actually have
a data structure now that represents
everything that we've defined for our
programming that source file
very important points about syntax trees
so they're immutable as are the nodes
and everything below so this is
important if you're making changes to a
syntax tree you're not changing anything
what you're doing is saying I want a
copy of this node with these changes
applied now internally this is all done
as sort of a relatively highly optimized
way to avoid sort of too much memory
allocations things like that but
ultimately it's important because
multiple different things might be
looking at that syntax tree at the same
time and we don't really want to have
locking we want these things to be
thread safe so if you're going to change
something you get a copy of that thing
with the changes applied
the other important point is that
they're round tripable so you can parse
a c-sharp source file through the the
compiler and you'll get a syntax tree
and because that includes everything
that's in that file including all of the
detail about the trivia you can return
back to a c-sharp file or just a text
file ultimately that has the exact same
representation of that source code in it
so entirely around triple
so there is some tooling that we can use
to kind of get an understanding of what
a syntax tree kind of looks like this
isn't installed by default but if you
want to this is where you need to know
the the product name on the right there
you can see the.net compiler platform
SDK is the component I've installed you
can find that under Visual Studio
extension development if you want to
install that piece
so with that let's have a look at a
syntax tree
yeah I don't want that one
so in this uh solution I'll give a link
to this at the end but it's not very
exciting we have a single
um
program class uh in this single project
here
um so it has a program main method has a
very simple data class and we just
basically create an instance of the data
class and then write something out the
code is not exciting nor is it important
but what we'd like to do is actually
understand what the syntax tree looks
like so if I go with that tooling
installed if I go here to other windows
I can bring up the syntax visualizer pin
over there hopefully this will be big
enough
if it isn't the space to come forward
um
so if I click in this code file
somewhere it populates the syntax tree
so you can see the roots of the tree
here is this compilation unit which is
basically just AC sharp source file and
if I click on this you can see it
highlights all the text that belongs to
that node and below
so if I click here I can see I've got a
file scope namespace declaration so
using the file scope namespace syntax
here so although this isn't curly braced
and then indented like a traditional old
style namespace declaration
um falscope namespace is still applied
to the entire file so everything in this
file belongs to that namespace just by
the nature of how this feature is
defined so everything appears under it
in the tree by being indented that is
defined with a keyword so this is a
token which is a language keyword that
we have in C sharp in the grammar and
then you can see here the trailing white
space so we can identify where that
trivia is appearing too we then have an
identify name which is made up of an
identifier token so this isn't a
language keyword it's something we've
decided to name the namespace and then
we can see we even have the punctuation
represented as a token the semicolon
here and then that's followed by a new
line we have two class declarations if I
expand this first one and we dive down a
little bit into the method and we go
down to the block so this is the the
method body you can see again wrapped in
open brace close brace tokens we have
our two statements and the local
declaration statement defines the
variable and then we have our expression
and if I dive down here and click on the
semicolon token at the end of that we
have multiple bits of trivia applied
because we have white space
then we have our single line comment on
that same line and then the end of line
trivia so this is all the trivia that's
attached to that token so that we know
what's actually in this file so that's
one way you can kind of get started with
getting a view of the syntax tree
another thing I've installed is
something called the dgml editor which
you can find if you search other
components and if I click right click on
this class declaration
desperately try and fit this on the
screen so one two five percent
we get basically a kind of graph view of
that syntax tree so I've chosen to start
at the class declaration and you can see
below that we have one of its children
is the method declaration
one of that has a child of a block node
and then we see all right I don't want
to move stuff around we see the
statements statement there statement
there so obviously you can see there's
there's lots of syntactical information
that has been identified here we have
all the tokens in green
we have the trivia so leading trivia
that comes before a token is in white
trailing is in Gray
um and that can be kind of useful if you
want to kind of get a feel for what um
what a syntax tree looks like
I'm going to close that so I'll get some
more memory back
um
cool
um so with that knowledge sort of set
let's talk a little bit more about how
we're going about co-generating a
language client
so elasticsearch has grown over the
years and elasticsearch is entirely
managed through its HTTP API so as
features get introduced the team add
more and more endpoints there's well
over 400 endpoints there there today and
for representing all of the data that
can flow in and out of those there's
over 2000 data structures use is
requests and responses for most of those
endpoints most of those have tens of
properties then there's sub-objects
below those that have to be represented
with Concepts like different queries and
aggregations and filters you can apply
so it's a complex API
and our job on the clients team is to
build the language clients that allow
you to interact with that
programmatically through various
different languages
so with the version 7 language client
for net one of the major problems we had
was it's nearly entirely hand written
and if you can imagine keeping up with a
server team of hundreds of people who
are producing new endpoints new features
or modifying existing requests and
adding new properties
um that's a lot of Maintenance work
because the client's team is less than
10. and so ultimately we're going to not
be consistent with the API you know in
the original days it was easy to kind of
hand roll this stuff and keep up at a
reasonable Pace but you know we have to
When someone tells us they've added a
new endpoint we have to see if they've
written documentation yet and if they
have we can see the docs and we can get
a view for what's available there if not
we have to go to the elasticsearch
server code base which is Java we have
to basically reverse engineer all of
their modeling there to figure out okay
well we want these properties on the
c-sharp model then and so you know we're
getting it out consistency we're never
going to release things quite at the
same Cadence that their server team can
and that's a problem
and so for version 8 we started thinking
about how we could get more intelligent
about this you know it was the time to
make the move to optimize this so for
net and Java in particular we've kind of
basically decided that we'd create the
next generation of those clients and
specifically to make things easier we
would look to try and code generate a
vast majority of them based on a formal
spec
so ignoring the spec for a second we'll
come to that
um if you're told you need to generate
some c-sharp files what are the first
possible options you might think about
well strings and string Builder is a
valid way of doing code generation you
can have strings that represent you know
a structure of a c-sharp file that you
want to produce with placeholders that
you populate for each type or you could
break them down into you know
representation of a class and a method
and then chain these things together
with string Builders that's a valid way
of defining ultimately a text file but
it's a bit brittle you don't get any
kind of checking that you're doing the
right thing if you misspell namespace
you will maybe only find out about it
when you generate your client and then
you try and compile the compliant you
find the error and then you have to find
the string that that code was defined in
another option that's maybe a step up
that's still really working with with
text ultimately is templates so
something like T4 Razer we were actually
using the Razer engine in the V7 climate
client for the small amount of Cogen we
did do we basically created Razer views
that represented the structure we wanted
to generate and that can include
constructs like loops and things and
then you have the view model which is
something that you can populate for each
type and then use the Razer engine and
model binding to actually produce the
final file and that worked quite nicely
but it's still kind of templates which
are text
ultimately what I landed on was I wanted
to try and play with the Roslyn apis
because my thought was well we can
construct a syntax tree with the rosin
apis and generate C sharp from it
because just as we can parse a c-sharp
source file and we can get to the syntax
tree as you'll remember it's entirely
around triple but we don't have to start
with parsing a file we can actually
construct programmatically that syntax
tree and then just emit the file from it
and so that's a look at what that looks
like in um in.net
so I have another I do have another
project Factory in this solution and
that has a file in here called spec.json
this is like a really super simplified
version of the elasticsearch spec it
doesn't Define enough to be a rich
specification for defining classes but
it's enough for getting started so you
could generate any spec you want you can
have any format you want as long as it
contains the information you need to
understand what classes and things you
want to produce you could create one but
we've chosen a basic Json structure so
we have an array of types the type has a
type name and it might have an array of
properties and the property is just
going to have a type and a name so a
very simple spec and we have two classes
to find inside this project
I have a reference to the package
microsoft.codeanalysis.c sharp which
gives me the compiler apis and
specifically stuff for working with the
c-sharp syntax tree
um that we're going to need so this is
where the Rosalind apis come in and
they've obviously used different names
for everywhere else that we've just
talked about but as long as you know
where to find it you know what you're
looking for
the program for this is only 55 lines I
think
most of that is actually parsing the
file and using statements so the first
part of this program is basically look I
want to deserialize from that Json spec
file that we've got into some types
in.net that represent it so I've got a
schema which has a collection schema
type the schema type as the type name
and the collection of schema property
and then we have a type and a name and
those are strings in this model so that
allows us to get some some objects from
in C sharp that represent our structure
that we want to generate
and then we're going to create this this
class declaration syntax array so this
is basically one of those syntax nodes
or a set of syntax nodes relating to
classes and so we do that by accessing
all the types on the schema we're going
to just do a link select and we're going
to use this helper class create class
down here to actually create the class
declaration syntax
so that takes in the class name and any
properties that are assigned on that
spec
so now what we're doing is actually
creating a class declaration and rather
than newing up the class declaration
syntax object itself we're using this
method on the syntax Factory and the
syntax Factory is I've got a using
static here because otherwise your code
gets littered with syntax Factory dot
all over the place and that gets hard to
read so I'm just going to call the
factory method here to create a class
declaration and it enforces the fact I
have to provide an a token as the
identifier so this is already giving us
some level of sanity checking that we're
generating something vaguely valid as a
syntax tree if we were just doing text
technically you could miss the
identifier you probably wouldn't but
here we're getting enforced by the
syntax tree understanding there's
definitely going to be an identifier on
the class please give it to me and
that's just the class name that we
passed in and then by default we're not
going to have any modifiers any keywords
applied to that declaration so we want
to add a modifier here and the modifier
is just we're going to attach a token of
syntax coin public keyword so a language
keyword so this syntax find here
contains basically an enumeration of all
the keywords we might need from the
language
so we now have a class declaration and
then for every property we're going to
add that property as a member so as a
node below that class declaration and
add members in this case we're creating
properties now the important point is
this reassignment to the variable so if
I take that off for a second
that looks valid probably I can't scroll
up for some reason that looks valid but
the problem is what we're doing here
because this is an immutable structure
is we're not actually updating
what this variable is referencing we're
calling ad members and that's returning
to us a class declaration syntax which
is the new copy the modified version the
the With The Changes we want so we have
to make sure we reassign that back to
our variable otherwise when we split
this thing out we won't have any
properties
to declare a property we need the
identify name for the at the type and
the identifier for the actual property
again we're going to make it public and
then we're going to need some accesses
so we can get and Set uh the value of
this property so in this case I'm using
this add accessor list accessors method
a bit of a mouthful and I like this one
because it takes a params attribute so I
can just pass into access the
Declarations comma separated here and it
will just take those in for me so we're
having a get and a set and we have to
put in the the grammar and the
punctuation so that the semicolon token
here of syntax kind semicolon token
there are other methods that you've got
so there's various with methods so you
can create like with access the list but
this requires an accessor list syntax
which you create through the Factory and
then you have to pass it a list so it's
a bit more of a bow so I prefer the kind
of add methods for kind of tersa code
and that's it for declaring the class
declaration we're ultimately going to
want these in a namespace so I'm now
going to use another Factory method to
create file scope namespace declaration
again I need the identify for that and
then to that as child children I can add
it the class decorations and now I've
defined my syntax tree so I'd like to
spit this out so I'm going to create a
stream to a file and I'm just going to
call right you can call right to on any
of the syntax nodes and it will spit out
what's below it
so if I
run this code
which is done and we go to the generated
code ah
not quite what we wanted so we're
missing spacing spaces and new lines
here we've just got we have got what we
asked for we have a public class first
class but this is not something the
compiler is going to be able to read in
and compile once we actually try and
compile it later on so why do we have
that problem
well we didn't actually specify any
trivia this code
just specifies the nodes and just
specifies the tokens we want now it's
possible to add trivia like we can do
with leading trivia with trailing trivia
and then we can attach the trivia that
we want that would get quite painful
quite quickly and ultimately I don't
care too much about the formatting so
I'm just going to cheat and I'm going to
call this normalize once white space
method and so what this does is just
apply standard white space rules for
where new line should appear after
semicolons and braces and you know
you're going to have a space after a
token a keyword before the identifier
that kind of thing and so then if we
rerun the code
we now get what we're hoping for so we
now get our generated c-sharp file it's
something a c-sharp compiler can
actually compile
um now personally I'd love an extra line
there because I just like that
um but that's not the rules so that's
what's generated try not to get too
precious over your generated code files
um most of the time you're not modifying
these things by hand you might debug
through them occasionally you might
check them when you start code
generating to make sure they're giving
you what you're expecting but the
formatting is not important there are
ways to pass like formatters in you can
even I think pass in your editor config
rules and it can apply those in the
formatter that stuff we're not worried
about
But ultimately we've generated the file
so actually one other thing you might be
wondering how would you know what the
rosin API calls are to generate your
c-sharp code and often you don't so I
start with this really handy website
which is basically where I've lived for
like a year Rosalind created
azurewebsites.net nice open source
project and if I paste in some c-sharp
code up here
and I click
um get the Russian API calls the output
here is close to what we wrote by hand
you can see by default you know this
project just tends to use the with
method so you can see with accessor list
then we create an accessories then we
give it a list which has got an array to
populate the list which is weird but
ultimately it's giving us an example of
all the nodes we need in the structure
we need them and the tokens that we're
applying to those the modifiers and so
you can kind of take this and then slim
it down if you if you care to slightly
terce a code you can even click to
preserve original white space and now if
I generate this you get all of the the
tokens have a leading and trailing
trivia sometimes empty and up here we
can see we've got a space after the
namespace keyword
um and you can see that that was
obviously a lot much longer and that's
why I don't bother messing around with
that stuff myself because I don't care
that much about the formatting of my
generated files
so moving on them um let's think about
generating the.net client so we're going
to need a spec and so how do you define
a specification for something that's
quite complex like elasticsearch so I
wasn't there when the team started this
sort of piece of work so this started
I've been elastic nearly two well just
over two years actually and this kind of
kicked off just before I joined and the
team first assessed something like open
API this is this is a specification for
defining HTTP apis and specifying them
ultimately at the time at least open API
was a bit too limited elasticsearch is
very complex and there would be needing
we'd need custom extensions added in
there to handle some of the edge case
things that we have and the weird
polymorphism that some of the requests
and response types have ultimately our
problems about representing data
structures endpoints are quite easy to
represent and specify and document the
data structures are harder as I say
elasticsearch apis sometimes they have
unions of the types they might return so
a property might come back as a Boolean
in some scenarios but sometimes as an
integer we need to know that so that the
C sharp code can be built to try and
handle that problem similarly you know
you can send any number of queries when
you're doing a search and all of those
have different structures that need to
represent those but ultimately the
server will handle whatever data it gets
but we need to model that somehow
and the other thing is managing an open
API spec for 400 endpoints and over 2000
data structures by hand in yaml or Json
um is not going to be that fun the
output today is about 180 000 lines so
managing that sort of size of
specification by hand is probably not a
great idea
so what the team decided to do is
actually let's let's deal with the data
structure issue and model
our data structures using typescript
because it's a static type checking is
in there so we can actually kind of make
sure that our types make some kind of
sense it's a type system built to
represent Json JavaScript we have strong
ID support for working with it across
all the developers no one was sort of
against using it in different languages
and it has its own compiler apis that
you can hook into with libraries like TS
morph to do some clever stuff that we'll
talk about later
so this is ultimately an example of kind
of what we've ended up with in terms of
defining our types so here we're
defining a request this isn't the
complete request and you can see we have
inheritance so we can say well look
there's a bunch of stuff that's on all
requests we'll put that in a base class
and we can say that this extends that
the fact it's just called request it's
actually a search request there's a name
spacing that we're using to figure out
where these requests belong to but we
also have meta information that are
basically specific tags in the comments
so in this case the spec name when that
was introduced in the server API is it a
stable endpoint and then we have to
define basically three main things about
the request the things that could appear
in the URL path the things that can
appear on the query string and the
things that might appear in the body and
we have separate objects to represent
these so here for the path we have an
optional index that can be passed in so
when you do a search you can either
search everything and just going
underscore search or you can provide the
index name or a set of indices that you
want to search against and limit your
search results and so that's optional on
the path and so we Define that with an
optional property here with the the
question mark and then we Define a type
for that so this is indices which is
actually a union of index name or index
name array the reason we have to be very
specific there you might think well we
could just use the array can't we but
the problem is on some of the responses
we might get back either a string or an
array containing strings to represent uh
the result of some index on that
response body and so we Define it very
explicitly to say what exact data types
we might expect in all scenarios and
then we also have like index name here
is basically an alias over a string that
gives us kind of a strong name for that
string type that's used across our code
in our typescript model But ultimately
it's just representing a string
on the body some of the things that are
interesting here so we have a property
opt to find on the body for aggregations
don't worry about the typing of that for
now but the interesting thing is that we
have this Alias so when you actually
send a search request for elasticsearch
you can send it aggregations optionally
and you can use aggregations as the
property name in the Json you can also
call it ads and the server will deal
with that on the way out that's not a
problem for us because we can choose
which we use but if the server returns
as a query in any of the data structures
it might choose one or the other of
those names so we have to again in the
c-sharp code base we have to have a
custom serializer that says okay I'm
looking for either ads or aggregations
and that binds to this property
ultimately
um so we have to declare that
information in the spec we also put like
documentation information into the spec
here and then that can be picked up by
the language clients to include their
own documentation where it's relevant
um a response is interesting because we
have user provided types
a response is going to take contain
generally search results which will
include the documents that you've
matched and those documents are the
things that the user indexed so we don't
know what the the structure of those is
we know what our request sorry our
response body is going to have in terms
of fixed properties but we need the
generic system in typescript to
represent there might be this other sort
of user provided thing and so ultimately
the hits on the response body here is
this type hits metadata of T
another interesting thing in here is
total is pain because again depending on
how you've configured your request you
might get back an object that contains
some properties the total hits object or
you might get it back along so this is
yet another case where we need to know
that as a a language client to be able
to deserialize either option depending
on what we're given and there's no way
of knowing up front in all cases because
some of these things are controlled by
server-side configs as well hits here
hits of T
um you can see there where this applies
is in the source so if you get a hit a
result back we have some fixed
information the server is always going
to include and then if you've asked for
it the source which will be this T
document and we can convert that into C
sharp quite nicely to generics as well
and then when you do a search request as
a generic argument you tell us what the
type is You're Expecting for your
results
and then you just um we can do serialize
that for you using our source serializer
uh this is another interesting concept
we had to come up with was how to model
those polymorphic scenarios where
something can be sent in multiple forms
or maybe received back for queries what
we've done is model this as a kind of
container and basically the container
lists all of the possible strongly typed
queries that elasticsearch supports and
then we Mark that with this attribute
here that says this is a container and
it's going to contain all the variants
that we could possibly send up and then
the language clients are free to do with
this information what they will they
have to come up with a language
structure that makes sense in the
language to represent that and how you
work with It ultimately in C sharp we
still end up with this container object
and it takes one of these queries that
it's going to contain and we just
constrain them by a base class to say
all of these things are valid in that
inside that container
now the other important thing we've got
to do is validate this spec we've
generated this massive typescript
specification and we've modeled it and
we've got all the strong typing in there
to make sure you know all of that
logically makes sense as a model but how
do we make sure it works with
elasticsearch and so for that we hooked
into the integration tests so ultimately
we have this thing called a flight
recorder that we run and we run the
integration tests and then we just
capture all the requests and respond to
Json for every test to every endpoint
and then ultimately what we can do is
just check that those serialize and
deserialize to and from our typescript
types so if if we get a response that
has a property on it that we haven't
defined in our model then we get an
alert and we can go and fix it and so we
get a lot of tests for free and we don't
have to think too hard about the problem
because the elasticsearch team will
create the integration tests as they add
features
and this runs as part of PR so if you do
a PR to the specification repo here cess
modified the user profile and it detects
all of the endpoints where that might be
used somewhere in the request and
response below and so you can see for
example security is your authenticate
has 15 recorded requests and responses
for different test scenarios that will
give us different examples of valid
requests and response bodies and so it's
checked that that type is still can
serialized to and from those
then we need to generate the spec we've
got the typescript model but how do we
create the spec now in theory we could
generate the types the the c-sharp code
from the typescript by using the
typescript apis to access their abstract
syntax tree and then generate our code
ultimately that's quite low level
and the main problem is it's not fully
constrained different language clients
and different authors of those
generators might infer different things
from the syntax Tree in a different way
and that's not ideal if you're trying to
generate consistent clients so we
decided what we would do is transform
that typescript to a simpler
specification tailor made for our needs
and unambiguous
so this is kind of our Cogen pipeline we
have all the typescript files in our
sort of project for the typescript and
then we have this thing called the spec
compiler so traditionally typescript
would compile to JavaScript what we do
is we hook in through that TS morph
library that I touched on earlier and we
accessed the syntax tree and then we
basically inspect that and go okay well
this is the Json structure we want to
produce for our schema and so that's
ultimately what the output is for our
spec compiler it's our single
schema.json file that lists all of the
possible endpoints and then also all of
those data structures and the
annotations that go with them
so from there we can create our code
generators so I'm created a.net code
generator that spits out most of the.net
client the other language clients are
doing the same for theirs the dynamic
people have got a great time because
they don't have to worry about the
typing so much the Java go and c-sharp
people we're having to think a bit
harder about how we model these
structures and some of these Concepts in
in our languages now I have some history
with the previous clients and I've tried
to apply that most of the time but we've
also had the opportunity to rethink some
of these things and how we model them
and also we've had to make decisions
around standardization in terms of the
code generator is going to have to do
this thing once for for this concept so
making sure it applies and works in all
cases
the other interesting thing we could do
and we're starting to play around with
is we have an hour and open API
generator that takes our schema.json
format and spits out an open API spec
and then teams within elasticsearch that
want to work with open API can build
like API documentation off of that spec
the exciting idea is that we could even
generate some of the server-side code
because if we get entirely schema driven
and we agree up from what the
specification for endpoints is going to
look like and how we're going to type
them in our typescript model we build
the schema.json and that's our single
source of Truth For What elasticsearches
API should look like so we generate the
clients but we can also generate the
endpoints on the server and their models
and that would get us to a really nice
place where consistency wise we should
always be consistent and if we're wrong
we're wrong everywhere at least
um quick demo of schema.json we won't
die as I say it's like 180 000 lines of
code we probably don't have time to look
at all of it
um
I've grabbed two examples
so if we go to line
9840 of the schema
we'll see the definition for our search
endpoint so this is that same endpoint
that we've been talking about so far so
you can see the name here we get all
that meta information so if we've
attached the documentation URL the
description we have the types for the
request and response so you can see all
of the types come out into the spec with
an actual namespace so we don't have any
clashes so this is why we don't name
this thing search request in the spec in
our model
uh we can see you know does it require a
body no it takes application Json it
returns application Json it's been there
since the day one it's stable and it has
two URL paths that could potentially
match for it because we have that
optional index on our skip specification
if you don't provide that when you you
build up your request then obviously
you're searching everything here with
that URL but otherwise we need to apply
it so the language clients have to know
all the URLs and they have to know if
properties are set on their whatever
however they've modeled a request if the
properties are set for that that Define
the the indices that you want to search
against we have to use this other URL
so that's that part of the spec if we
jump down even deeper we'll get to the
um
search request and again this is itself
is is quite big so I'm going to look at
the whole thing but ultimately the
object starts about here and and the top
part of this defines its body properties
so to align with what we saw in the
example code earlier on there's that
aggregations
property that is available we have a
list of any potential aliases that the
clients can then decide how they handle
aliasing if they need to this property
isn't required on a search and then we
have the type information so in this
case this type is a dictionary of and
the key for this is an instance of
string and the result sorry the value is
an instance of that aggregation
container so we looked at query
container aggregation contain a very
similar concept we have many possible
aggregation types that you can send in
and the container basically is a way of
constraining that to all of the possible
built-in types that we could represent
an aggregation with
and that ultimately is what that entire
file contains just for all of those 400
endpoints and types that we have
eventually
so in terms of net code generation
um our first job is deserialize Jason
pretty easy
then I do a phase where I'm essentially
doing what I call building the context
so the the Json file contains everything
we should need to know about the
elasticsearch API and its types but it
doesn't have enough information for me
just to generate blindly I have to do
some sort of
modifications and understanding of what
that means to see what's valid for C
sharp and we'll talk specifically about
what this I call the marketing Rich
phase because it sounds fun and I'll
talk through what that actually does to
that is sort of applying stuff to those
contacts But ultimately once we've got
all of these contacts for every type
we're going to generate we can do what
we've just seen before just on a
slightly more complex level and build up
a syntax tree using those rules within
apis we just build a syntax tree for
each type that we're going to generate
and we emit a file or set of files
potentially out to the project that
we're generating into
so some of the things we have to think
about so one of the important things is
naming a name spaces it's possible the
typescript model includes names for
types that are not valid in C sharp
because they can flick with a keyword or
whatever else there's often the cases
where some of those types have
underscores and different casing
potentially as well
um and so we have to make basically
sanitize the naming of all the types
um sometimes we can do that on a
rule-based basis sometimes we just have
like configuration as code that says
rename this thing on this type to that
because it makes more sense
um also we have to think about
namespacing we don't want a namespace
that says undersall Global in C sharp I
think it would be unhappy we want it to
be in our root project namespace for our
assembly
um in that scenario and then we do
introduce namespacing in the c-sharp
library to avoid type name conflicts but
we've tried to limit the depth of that
so in the typescript model the depth can
get quite quite far down I think you
could have like dot five times for
different parts of the namespace we try
and bring that to a more sane level of
less namespaces so if you're using the
different parts of elasticsearch apis
you'd have to bring in lots of using
declarations for them we also walk the
type type hierarchy so looking at the
base types to find in the spec the base
types in this in the typescript model
are there as a convenience for us not to
have to retype all the properties on
every request they don't represent an
actual construct on the server
necessarily because the server May model
that differently so what we've decided
to do in most of the strongly typed
languages is actually just flatten all
the properties from all those base types
onto the search request in this case for
example so that request Base Class just
contains some properties we hoist up
logically inside our kind of context we
have to identify the relationships so we
resolve all those schema namespace names
we have to identify okay well this
property has this type and that's this
c-sharp object over here that we've
pulled from the from the specification
and deserialized and how they
interrelate so we can navigate those
relationships in any direction
we Mark all the request types so
anything that's used on the request
you're going to need to be able to
modify it and set its properties because
your you know consumers are configuring
the request they want to send response
types we don't expect you to modify
those so we make them basically read
only and we deserialize them and give
them to you we also Mark all those
special Concepts those containers the
variants we also have something called
internally tagged unions which is
another way of representing the
polymorphic Json we might get back
and ultimately we also want to simplify
some of those type aliases so where we
had like index name is an alias for
string that one we actually keep because
we have some special logic around the
index name but that's used a lot in the
typescript model and many of those we
don't want our users to have to
basically we'd have to create a type a
class or a struct that wraps a string in
order to model that and then you'd have
to construct that through like a
Constructor or something or we give you
implicit operators that's a pain to work
with and it's hard to reason about so we
just simplify those to whatever they
aliased to the string or the integer in
the properties that we're generating to
make working with the API a bit easier
and then we also Mark some some types
that need special serialization so the
bulk API takes new lines are limited
Json not just a standard Json structure
and so we have to know that that that's
going to need custom serialization logic
that we're going to create and so we
basically stub that out and then we can
manually manage that code ourselves in
the in the client
so what does our generator code look
like pretty much what we saw in the demo
earlier on just slightly bigger amounts
of code scattered throughout my
generator so when you see a request on
our type
or the type in RM schema that's one type
but in C sharp we're going to generate
probably at least four classes from for
that we generate a class that represents
all of the query string parameters we
call that the request parameter class
and that's what the code you're seeing
here we also generate the request class
itself that models the body of a request
and then because we have another Syntax
for fluently defining requests in
elasticsearch we generate what we call
descriptors which is how our fluent
syntax is put together and that will
usually have a non-generic and a generic
version for reasons for when you're
working with those in different ways in
the in the API so here we're doing a
class declaration for the request
parameter class from the context we get
the name we add the modifiers we can add
a base type and then ultimately we'll
create properties for each of its query
string parameters to find and we add
that that adds class there just adds it
to our root namespace declaration
um we also then need to figure out what
Constructors we're going to need on the
request so we saw a request had an
optional path parameter sometimes
requests have a required one and if you
have a required one what we do is
generate a Constructor that enforces
that so that consumers can only create a
request with the required path
parameters so we do all of that sort of
analysis up front and then ultimately
we're generating again another class
declaration for the request class we add
all of our various members the
constructors some common properties a
property that maps to the generic
argument it's query between parameters
and then the body properties and so body
properties here is using essentially my
own syntax Factory which
helps me create this serializable
property syntax so I take in a property
here which is basically what represents
a property in my contacts I do some type
name resolution to find out what the
fully qualified name of the type is so
rather than figuring out what using
declarations each file is going to need
I generally just use the fully qualified
name of the type where we're declaring
properties and things like that so once
I have that information I can create the
property property declaration we can add
any attributes that are needed for
system text Json so the Json include the
Json property name and we get all of the
property names off of the spec
if it's used in the request we add a
getter and a Setter otherwise we don't
and that's that's pretty much it
for our code generator
so let's now think about what I've
learned through this process and what I
would do differently so we have a
working code generator that's generating
not all of the endpoints yet mostly
through for the fact that we haven't
spent the time testing everything looks
good but we have a working generator but
it was put together rather rapidly
through evolving knowledge because we
were defining the spec as we went we
were fixing problems we were coming up
with other constructs in the the model
for elastic searches data structures
that we needed to somehow specify and so
we often found ourselves kind of having
to adjust based on new things we were
adding to the spec or even new kind of
things like those variants
um and so as I've gone it's got a bit
spaghetti code
um you know we all try and write clean
code but this is a bit of a mess so it
needs refactory
um and also I've understood where the
various things are that you need to
customize in this because ultimately
what I'd like to do is productize this
so we can get maybe an open API spec or
another schema.json file for a different
product and then spit out of a working
client but right now it has a little too
much knowledge built into the code about
elasticsearch and some special edge
cases
so what I'd like to do is add like a
plugable transform pipeline to this
thing that basically takes you know a
set of defined Json files that could
represent how we would take the spec and
then transform it so this would be
things like renaming those properties
that conflict we'd have a you know a
rename transform that you'd read in and
it would tell us what to rename and so
at this level we really just tweaking
the spec that we've just read from Jason
it could include things like skipping
certain Properties or adding other
properties and so I'm thinking you
probably have an eye Transformer and
then you plug those in through Di and
most of them will read out really in
Json input
similarly for working out what we're
actually going to generate right now
it's just kind of a an allow and deny
list in code that kind of says yeah
these are the endpoints I want to print
out we can make that more intelligent
with a similar concept some kind of
filter pipeline so we can say well have
a filter that says we only want to
generate endpoints that have been
available up and or since this version
or before this version we only want to
generate stable endpoints or we want to
generate you know this specific allow
list of endpoints
and ultimately I want to make this thing
it's something that we can kind of hand
off to other teams that makes it easier
for non-developers in those teams to
configure it through Json files rather
than having to kind of figure out what's
what's going on I want to kind of hide
all the complexity of Rosalind for them
and the really cool thing that I I
haven't had a chance to play around with
that I think would be really good and
useful is to actually look at getting
more intelligent with this so part of
the reason I chose the rosin apis is
there's a lot more you can do with it
including working with something called
the workspace apis
so the workspace apis
basically give us a way to hook into
like Solutions and projects so there's
an MS build workspace that you can load
up and you pass in a solution location
and then you can access the solution and
then you can navigate through to see
okay that solution has these projects
and it has these source files or these
documents and then you find all of the
c-sharp documents and then you could
basically build up the syntax tree and
the semantic analysis for everything
that's in there essentially within that
compilation and then we can take the
knowledge about what we're about to
generate and you and kind of dip it
basically so today what the co-donator
does is it obliterates the generated
folder in the project and it rewrites
every every file for every part of the
spec that it's generating which is
pretty low Tech but what we could do is
say well okay I'm about to generate this
type I know I'm going to have these
properties can you find this type is
that a symbol that's already in this
project okay it is but it's missing
these properties so we're going to be
just adding those and that's where we
can basically modify
the syntax tree so create a new copy of
the syntax tree from that node With The
Changes applied reapply that into the
workspace and then have it emit out the
changes to c-sharp files so we're not
obliterating everything every time
because right now I have to basically
get diff on the elasticsearch client
after I've run the generator and go okay
it's added this stuff
the important thing is we'd want to
detect breaking changes if someone
removes a property from the spec what do
we want to do in C sharp do we want to
remove the property and introduce a
breaking change or do we want to maybe
preserve it and decide that we put a
sort of deprecated uh obsolete attribute
on it or something
the other thing we could potentially do
once we have access to this level of the
code because we understand the full
project we could actually then in memory
compile the whole project and we could
check that what we've generated is
actually compilable because right now
again I run the code generator I switch
to my other Visual Studio instance that
has the client I do the git DF check
what's changed and then I say compile it
and say okay yeah great it's worked
you know that's manual it's not ideal we
could automate all of this through the
workspace apis potentially and then we
can just have basically a GitHub action
that says generate this version of the
spec please and then give me a PR that
introduces those changes that's kind of
where I want to get to this thing
um
resources so I'm pretty much done
um the link to the slides again uh
writing Cobra code on the bitly there so
if you grab that you get all the slides
you get all the links I'm about to show
you uh the demos I've showed you on my
GitHub repo they're not particularly
exciting but if you want to have a look
at them you can
this is the the Microsoft page for the
rules and stuff so you can see here it's
referred to as Rosalind which I think is
better Roslyn quota definitely your
friend if you're doing this kind of
stuff for just taking what you've
already got or you know you prototype
what the c-sharp is that you're going to
generate and then you go back to the
Ross and Coast to get the rosin API
calls you need simplify them down a bit
and ultimately package that up as your
code generator if you want to look at
the result of this kind of stuff if you
go to our elasticsearch client repo and
go to the dot generated folder in there
um underscore generated folder in there
you can find all of the generated files
if you're curious what the ones we're
generating look like so far and I'll
spec is public as well so if you want to
view that specification typescripts and
and if you want to contribute or just
kind of see what we're doing in there
you can go and view that and that
includes all of our custom compiler
stuff I think in there as well
the co-generative donor is not public
I'm afraid so you're just going to have
to go on what I've shown you so far
um once it gets more uh refactored and
productized for other teams internally
you know maybe we'll be able to release
that as well and make that open but
right now it's kind of intrinsically
tied to what we're doing and it's a mess
so you're not going to see it
um with that thank you very much for
coming I hope this has been useful maybe
interesting or just a good place to nap
after lunch otherwise uh you know find
me online if you have any questions that
pop up I'm going to clear the way for
the next speaker but if you want to hang
around for questions afterwards uh yeah
come and find me thank you